Gompertz Linear Units: Leveraging Asymmetry for Enhanced Learning Dynamics
Authors: Indrashis Das
Mahmoud Safari
Steven Adriaensen
Frank Hutter
Venue: NeurIPS 2025 poster ()
Keywords: Deep Learning
Activation Functions
Primary area: deep_learning
TL;DR: We introduce the Gompertz Linear Unit (GoLU), a novel self-gated activation function with superior performance on a diverse range of tasks.

Abstract: Activation functions are fundamental elements of deep learning architectures as they significantly influence training dynamics. ReLU, while widely used, is prone to the dying neuron problem, which has been mitigated by variants such as LeakyReLU, PReLU, and ELU that better handle negative neuron outputs. Recently, self-gated activations like GELU and Swish have emerged as state-of-the-art alternatives, leveraging their smoothness to ensure stable gradient flow and prevent neuron inactivity. In this work, we introduce the Gompertz Linear Unit (GoLU), a novel self-gated activation function defined as $\mathrm{GoLU}(x) = x \\, \mathrm{Gompertz}(x)$, where $\mathrm{Gompertz}(x) = e^{-e^{-x}}$. The GoLU activation leverages the right-skewed asymmetry in the Gompertz function to reduce variance in the latent space more effectively compared to GELU and Swish, while preserving robust gradient flow. Extensive experiments across diverse tasks, including Image Classification, Language Modeling, Semantic Segmentation, Object Detection, Instance Segmentation, and Diffusion, highlight GoLU's superior performance relative to state-of-the-art activation functions, establishing GoLU as a robust alternative to existing activation functions.

Gompertz Linear Units: Leveraging Asymmetry for Enhanced Learning Dynamics
Indrashis Das University of Freiburg dasi@cs. uni ceil-freibugg.cmm de
Mahmoud Safari
University of Freiburg
safarimecs. uni www.effrruurrooo
Steven Adriaensen
University of Freiburg
adriaens@ee uni muilfreiburg.de
Frank Hutter
Prior Labs & ELLIS Institute Tübingen & University of Freiburg fhêcs. uni-freiburg.de
Abstract
Activation functions are fundamental elements of deep learning architectures as they significantly influence training dynamics. ReLU, while widely used, is prone to the dying neuron problem, which has been mitigated by variants such LeakyReLUU, PReI.U, and ELU that better handle negative neuron outputs. Recently, self-gated activations like GELU and Swish have emerged leveraging their smoothness to ensure stable gradient flow and prevent neuron inactivity. In this work, we introduce the Gompertz Linear Unit (GoLU), novel self-gated activation function defined as GoLU(x) = æ Gompertz(z)), where Gompertz(x)== e e- The GoLU activation leverages the right-skewed asym- metry in the Gompertz function reduce variance in the latent space more fectively compared to GELU and Swish, while preserving robust gradient flow. Extensive experiments across diverse tasks, including Image Classification, Lan- guage Modeling. Semantic Segmentation, Object Detection, Instance Segmentation, and Diffusion, highlight GoLU's superior performance relative to state-of-the-art activation functions, establishing GoLU as robust alternative to existing activation functions.
-tate-of-the-att alternatives,
Introduction
Developing effective activation functions has been a longstanding area of research deep learning. In the early days, the Sigmoid [Verhulst] 1838 Rumelhart al. [986] and Tanh [LeCun al. 2002 functions were popular choices. However, these activations can suffer from the vanishing gradient problem due their tendency saturate. The introduction of ReLU [Nair and Hinton 2010) marked turning point, as it allowed for more efficient training by alleviating the vanishing gradient problem and inducing intensity equivariance [Nair and Hinton 2010]. However, ReLU comes with its own challenges, notably the dying-ReLU problem. To address these challenges, several ReLU variants have been developed, including LeakyRel.U [Maas al., 2013), , PReLU (He al. 2015) and ELU (Clevert et al.) 2015). Despite the emergence of these alternatives, ReLJU remains one of the most widely used activation functions today, owing its simplicity as a piecewise linear function and its computational efficiency.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

In the deep learning community, the landscape of activation functions has gradually shifted towards self-gated activations such as Gaussian Error Linear Units (GELU) [Hendrycks and Gimpel, 2016]. Swish [Ramachandran et al. 2017), and Mish MMisra 2019]. These activations provide probabilistic interpretations while enhancing robustness when combined with normalization techniques [loffe and Szegedy 2015 2015 Ba et al.) 2016 Ulyanov et al., 2016 Wu and He] [018 Zhang and (ennnichh[2099) Unlike ReLU, which strictly enforces gradient preservation due to its piecewise-liness nature, Swish, Mish and GELU, as smooth activation functions, relax these constraints. Their smoothness allows for improved gradient flow without strictly adhering to intensity equivariance.
In this work we introduce Gompertz Linear Units (GoLU), a new activation function of the self-gated gated family based on the Gompertz function [Gompertz] [825] as its gating mechanism. The Gompertz function was initially developed to model human mortality rates, and has since been widely applied in biology. Notably, also possesses probabilistic interpretation, as it represents the cumulative distribution function (CDF) of the standard Gumbel distribution. While both the Sigmoid function and the Gaussian CDF exhibit reflection symmetry around the point (0, 0.5), the Gompertz function manifests subtle rightward asymmetry, leading to distinct qualitative behavior.
Our experiments indicate that GoLU, compared to existing self-gated activations, effectively reduces variance in the latent representation. Moreover, it contributes to smoother loss landscape, making it less sensitive to small perturbations the model parameters. Additionally, an analysis of the learned weights in our trained models reveals that GoLU induces more spread weight distribution compared to commonly used activations (see Section 2.2 for details).
more spread weight distribution may indicate the network's ability to capture a diverse range of features from the data. On the other hand, variance reduction in activation outputs can help eliminate irrelevant information, allowing the network to focus on distinguishing patterns and potentially mitigate overfitting. However, overly broad weight distributions may introduce instability, while excessive variance reduction could result in underfitting and the loss of essential features, ultimately degrading performance.
Extensive, task sesk-specffic evaluations, suggest that GoLU effectively addresses this trade-off by achiev- ing a balanced level of both weight distribution and variance reduction, leading to improved perfor- mance over baseline activations (see Section 33. To facilitate reproducibility, we have made our code available at htttps://githiii com aattmo//oollllo
2
Gompertz Linear Unit
2.1 Definition and Properties
In this section, introduce the GoLU activation function and discuss its properties. GoLU is defined through Equations [ and visualized in Figure (Left) as the red curve, alongside other activation functions for comparison.
GoLU(x) Gompertz(x)
Gompertz(x) -
The gate function Gompertz(x) refers the Gompertz function introduced in [Gompertz] 1825 and is plotted in red in Figure (Right), alongside the gate functions of other gated activations. The Gompertz function can also be interpreted probabilistically, as it corresponds to the CDF of the standard Gumbel distribution, Gumbel(0, 1), with probability density function
Gumbel(x) = (x+e=")
From Equations ! 2 and Figure I understand that, contrary to ReLU and its variants which are monotonic and non-smooth at zero, GoLU is a smooth and non-monotonic self-gated activation, similar Swish and GELU. In fact the formulation of GoLU using exponentials makes it infinitely differentiable. However, in contrast to Sigmoid and the Gaussian CDF (i.e. the gate functions of Swish and GELU), the Gompertz function is asymmetric, it does not mirror evenly around central poin
Formally, we refer to a scalar function f(z) symmetric if there exists point x' such that for any input we have f(x" +2) f(x') = f(x") f(x" 2).

---

30
GTLU
Swnin
Sgmood bnttsoffplessxi)
0.8
Output Neuron Value
2.0
EL
3.5
3.0
Output
0.5
0.0
0.0
Input Neuron Value
Input Neuron Value
Figure 1: Activation functions (Left) and their corresponding gates (Right). GoLU and its gate, the Gompertz function, are highlighted in red. Note the slight rightward shift of the Gompertz gate.
0.40
Go Gumbel
GF111
0.35
Gaussion
eb/113 ote1)
1.5
1921
Swithh
troftlusteilll3
Probability Dersity
0.30
0.
0.5
Grodient
0.5
010
005
0.0
0.0
Input Neuran Figure 2: Comparison of the distributions underlying the gate functions (Left) and the gradients (Right) of various gated activations. The Gumbel distribution exhibits a slight rightward skew.
Inaut Neuron Value
This asymmetry, which has a bias towards the right, arises from the inherent asymmetry of the Gumbel distribution, which favors positive input values, illustrated in Figure Sauree](e (Left). In fact, the right-leaning asymmetry of the Gumbel distribution leads to smaller gate values across the entire input range, inducing compression effect the output distribution. This behavior extends to GoLU, yielding output values closer to zero, both for positive and negative inputs, when compared to other gated activation functions, effectively reducing the magnitude of the activation output. We note that, while Mish also exhibits an asymmetric distribution, it is skewed to the left, producing the opposite effect relative to G
From a more localized perspective, the Gompertz gate exhibits a reduced value in particular at the origin. This leads to decreased steepness of GoLU near this point, as indicated by GoLU' (0) Gompertz(0) from Equation This property of reduced slope magnitude is not confined to the origin but extends domain, as shown Figure 2 (Right). Additional details are provided in Appendix A
neighborhood around it and spans substantial portion of the negative input
In the large negative region, the Gompertz gate, and consequently the GoLU activation, decays extremely rapidly as a double exponential, suppressing unimportant features like ReLU, while maintaining smoothness, unlike ReLU.
Compared to the Gaussian CDF and the Sigmoid function, the Gompertz gate initially exhibits flat plateau, followed by steeper growth rate that aligns more closely with the Gaussian CDF. As the input values become large and positive, the growth rate flattens and resembles the Sigmoid function, with the difference falling off as O(e 22r) (see Appendix A
See Appendix B for interesting of flipped Mish distribution with night-leaniee asymmetry.

---

2.2 Effects on Training Dynamies
The distinctive properties of GoLU influence the training dynamies, as we will outline here.
Variance reduction As illustrated in Figure (Left, GoLU exhibits profile that remains closest to the x-axis across the entire input range. Moreover, its slope, particularly near the origin and over a substantial portion of the negative input domain, is smaller in magnitude compared to other gated activations, as pointed out in Section 2.1] These characteristics suggest a reduced sensitivity of the activation output to variations in the input. In fact, for scalar activation function the variance of its output can be shown to be approximately proportional to the square of its slope
Var[f(x)] = f (2))
where μ and o² denote the mean and variance of the input, respectively (see Appendix A for the derivation). This analytic relation explains more directly how the smaller slope of GoLU contributes to a lower variance in its output. As a result, GoLU effectively reduces variance in the latent representations, and promotes smoother activation outputs, enhancing the model's ability to distinguish between strong and weak features.
To visually illustrate this phenomenon, process Figure 3 (Left) through a 3 x 3 2D Convolution followed by 2D Batch Normalization. The resulting pre-activation is then passed through various activation functions, and the pixel distributions of the normalized pre-activation and activation maps are plotted for GoLU, GELU, and Swish in Figure 3 (Right). As observed, GoLU exhibits a distinctive "squeezing effect", compressing the same distribution into smaller output range, and reducing variance most, compared to GELU and Swish.
Feect far 0.966 (naw)
Siith teggn1 353
Gold
Figure 3: Image created by Dall-E 3 (Left) and kernel density estimation curves for distributions of activation outputs for the image (Right). GoLU reduces variance most compared to baseline activations.
To further substantiate this observation, we randomly sample four images from the CIFAR-10 dataset, apply the same preprocessing pipeline, and pass the results through different activation functions. The variances of the activated signals, summarized in Table highlight GoLU's ability to achieve notable reduction in variance compared to widely-used activations, enabling smoother data representation. This effect is not limited to CIFAR-I0 10, as we also observe a similar trend on ImageNet. Finally, to illustrate this effect in fully trained model, we randomly sample three images from the ImageNet- 1k dataset and pass them through a ResNet-50 model trained on ImageNet- 1k. As shown in Figure 4 the output distributions of the final activations demonstrate that GoLU produces more peaked distribution compared to other activation functions, highlighting this distinctive effect on latent representations. We note that this effect is not limited to the final activation, as shown in Figure but is consistently observed across intermediate layers of ResNet-50 as well. This lower activation variance can be seen form of implicit regularization as the network's representation of the input becomes smoother, focusing on the core patterns rather than fine-grained details or noise.
Smooth loss landscape Reduced activation variance results in less noisy and more consistent gradients. This typically means that the loss function changes more smoothly with respect to model parameters. As a result, the optimizer more likely to converge to flatter regions of the loss landscape with smaller curvature. This is expected to result in better robustness to small perturbations of the

---

Table 1: Variances of randomly sampled images from CIFAR-10 10 after applying a 3x3 Convolution followed by Batch Normalization and further passing the feature maps through different activations.
Activation
ReLU
0.3024
0.3063
0.3627
0.3594
LeakyReLU
0.3055
0.3100
0.3639
0,3626
ELU
0.5677
0.6227
0.4699
0,6418
GELU
0.2995
0.3102
0.3583
0,3701
Swish
0.2685
0.2872
0.3332
0,3399
0.3448
0.3700
0.3677
0,4200
GoLU
0.2133
0.2150
0.3213
0.2783
GoUu
Actwated Meracs
Figure 4: Distributions of final activation outputs of ResNet-50 trained on ImageNet- 1k for three randomly sampled images from ImageNet-Ik. GoLU leads to a more peaked distribution for the final activation output.
model parameters. We explore this by adding two different Standard Normal noise terms, scaled independently by a, B, to the weights of ResNet-20 trained on CIFAR-IO. We compute the test loss across a grid of scaling factors a, 33 for the two terms, while keeping the noises constant (refer Appendix C for more details). ResNet-20 with GoLU shows relatively smoother, less-spiked loss landscapes compared to other activations (Figure 5) which is consistent with GoLU's better generalization gains reported in the next section. Additionally, GoLU exhibits lower average loss, as shown in Figure 3 We note that Figure D offers qualitative insight into the smoothness of the loss landscape along 2D random subspace and is intended as a rigorous or quantitative analysis. Figure 9 in Appendix presents a comparison of loss value distributions across the loss landscape, indicating that GoLUU yields lower loss variance compared other activation functions.
Spread weight distribution In contrast the reduced variance in the latent space, observe a wider distribution in the learned weights of our models trained with GoLU, at least in the region where most weights concentrated. Figure 6 compares nonn-nomalizatiorff weight distributions of ResNet-50 and ViT-B/32 trained on ImageNet- k and GPT2-S (124M) trained on OpenWeb'ler Web'eex,, with different activation functions. The broader weight distribution for GoLU around the peak suggests that the network has learned more diverse transformations, enhancing its capacity to distinguish between features in the data.
This may reflect the network's response to reduced activation variance, counterbalancing it by spreading the weights around the peak to maintain representational diversity. Specifically, reduced output variance naturally leads to more uniform gradients, which turn encourages a broader spread of weights.
Notice that a wider weight distribution around the peak does not necessarily translate to a larger overall variance. However, focusing on the bulk of the distributior we find that GoLU consistently achieves the highest variance. This behavior suggests that networks trained with GoLU effectively
As learned transformations in the model are mainly encoded in the weights of fully connected, convolutional attention layers, it more meaningful to exclude parameters of Batch Normalization and Layer Normalization layers, although including these parameters we obtain qualitatively similar distributions.
'Specifically, we take the intersection of the middle 98% intervals of the parameter distributions of an architecture trained with each activation.
5

---

ReLU
GELU
Swish
GoLU
Figure 5: The loss landscape on the of ResNet-20 trained on CIFAR- 10 with ReLU, GELU, Swish and GoLU after adding random, scaled perturbations to the learned weights (refer to Appendix for more details).
Resliet00
VIT-8/32- ImageNer-1K
GPT2-5 Open Web Text
CEUU
Swith
Hith
GKUl
Denalay
Demalt
Paramoter Values
Parameter Valuee
Parameter
Figure 6: Learned-weight distribution of ResNet-50 and ViT- B/16 trained ImageNet-Il k and GPT2-S trained on OWT. GoLU leads to a more spread weight distribution. The range of parameters clipped for better visualization.
suppress density in extreme values while expanding the distribution around the peak. Such a pattern implies that the model captures a broader range of meaningful transformations without over-reliance on extreme parameter values certain features.
We emphasize that the effects attributed to GoLU, as described above, are not guaranteed hold universally across all scenarios but rather represent general trends observed in our empirical findings.
Moreover, while asymmetry has been highlighted note that its high performance, detailed the next section, cannot be solely attributed to asymmetry, but arises from an intricate interplay of properties, described in Section 2.1
distinctive feature of GoLU, it is important to
3 Experiments and Results
3.1 Overview of Experiments
We conducted experiments across various architectures and datasets, spanning a diverse range of tasks in both vision and language modeling. We begin with image classification, training ResNet-18, 34,
50 (He al. 2016), WideResNet-500 ZZaoruykoo and Komodakis 2016). DenseNet- 121 (Huang
al. [017]] EffcceenNee-BBO [Tan and Le 2019]. Tiny Vin Wu et al. 2022 ViT-B/32 and ViT-B/16
Dosovitskiy et al. 2020 on ImageNet- Deng et al. [209)]
We then extend our experiments to language modeling. We train babyGPT on the TinyStories (TS) [Eldan and Li 2023] dataset and GPT2-S [Radford al. 2019 on the WebTet (OWT)
Gokaslan et al. 2019] dataset, leveraging the nanoGPT repository Karpathy [2023).
Additionally, we assess GoLU's performance Semantic Segmentation (DeepLab V3 Chen et al. (017)), Object Detection (Faster R-CNN-FI Ren et al. [2015), RetinaNet-FPN Lin (017]) ) and Instance Segmentation (Mask R- R-CNN-FPNI He et al. [2017] MS-COCO Lin et al. [2014].
leveraging our pre-trained ResNet-50 backbone on ImageNet- Further, we test GoLU on Denoising Diffusion Probabilistic Models Ho et al. 2020 the CelebA Liu et al. (015) dataset.
We closely follow established baselines for all model architectures and tasks, ensuring that the integration of GoLU is the primary change. Hyperparameters, optimizers, learning rate schedules,

---

Table 2: Top-1 test accuracy ResNett 18, 34 and 50, B0, ViT. ViT-B/32 and ViT-B/16 on ImageNet-
WideResNett50-22 DenseNet- 121, EfficientNet-
Architecture
ReLU
LeakyRel.U
ELU
GELU
Swish
Mish
GoLU
RN-I8
69.7420.07
69.78=0.04
67.10+0.07
70.6640.05
70.60:0.06
70.53=0.06
70.76:0.06
RN-34
73.26+0.01
73.25=0.03
69.27+0.09
73.4440.04
72.74+0.05
72.73=0.07
73.71=0.04
RN-50
75.4420.07
75.67=0.08
71.87+0.09
76.07:0.06
75.17=0.14
75.53+0.09
76.63=0.03
WRN-50-2
76.96=0.07
77.17=0.1 12
71.90+0.01
76.72+0.01
75.41=0.03
75.75+0.19
77.37=0.03
DN-121
74.95=0.09
75.03=0.06
68.95+0.04
74.64+0.11
72.81+0.06
72.97+0.10
75.25=0.03
EN-BO
76.52=0.07
76.65=0.04
76.21
76.90=0.01
76.84=0.02
76.76=0.06
76.86=#0.04
Tiny VIT
82.91+0.02
82.83=0.03
80.2940.07
83.0540.03
82.9240.06
83.014 0.02
83.21+0.02
VIT-B/32
74.31=0.04
74.5320.03
65,8240.07
75.4840.05
72.3142.15
75.1620.07
75.74=0.09
ViT-B/16
80.06+0.05
79.93=0.02
13.3640.16
79.3940.99
79. 19.1940.94
77.97+1.95
80.72=0.04
and other training settings are aligned with the standard practices for each task. All our experiments are conducted on three seeds and the results are averaged out and reported with the standard error.
In Appendix E we further present a Critical Difference analysis to systematically compare the overall performance of activation functions. In Appendix | assess the performance of GoLU machine translation task, using the WMT14 English- German benchmark. Finally, in Appendix [ the application of GoLU to the task of learning curve extrapolation.
we explore
3.2 Image Classification
We evaluate GoLU's performance in image classification tasks on ImageNet-Ik, comparing it against
six state-of-the-art activation functions, ReLU, LeakyReLUU, ELU, GELU, Swish and Mish. Table presents the top-1 test accuracies with standard errors for ResNets 18, 34 and 50, WideResNet-
50-2, 121 EfficientNet- BO, ViT-B/32, VÝT-B/16 and Tiny ViT Wu et al. [2022]. The
training settings, detailed in Appendix G.1 are adopted from Torchvision Torch Vision [0166] for
all experiments except EfficientNet-BDD B0 which taken from the timm library Wightman 2019 and
ViT which is taken from Wu et al. [2022] As highlighted, GoLU consistently outperforms the standard activation functions across all architectures, with exception of EfficientNet-BKe where the performance difference is minimal. Notice that EfficientNet-Bl -O is exception because its nonlincarity arises not only from activation functions (which are replaced) but also from a squeeze- and-excitation block, which remains unchanged in experiments. For ResNet-50 and ViT-B/32, test loss and test accuracy curves are shown in Figures 17 and 18 respectively, where GoLU consistently delivers lower test loss and higher top-1 accuracy over the epochs. GELU is generally the second-best performer, while ELU performs worst across most architectures.
We further evaluate GoLU CIFAR-10, comparing it against top baseline activations. We report Table the results of image classification on CIFAR-10, with ResNets 20, 32, 44, 56, and 1 110 WideRessNer2-22 DenseNer40 and ViT-T//16-224. GoLUU consistently outperforms the standard baselines across all tested architectures. We have further underlined the second-bes activations for each model. No single activation consistently ranks second.
Table 3: Top-1 test accuracy on CIFAR-10. GoLU consistently outperforms baselines. Second best activations are underlined.
Arch.
ReLU
Leaky
GELU
Swish
GoLU
RN:20
91.41=011
91. 61.600000
91.64+0.1
91.77=0.1
RN-32
92.21x0.1
92.40.0.0
92.54x0.1
92.1640.0
92.6910.1
RN-44
92.58=0.0
92.78.0.0
02.78+0.3
02.51+0.0
92.85=0.0
RN.56
80:00
92.7540.1
92.8644.1
92.9340.1
93.15.0.1
RN-110
93.21+0.0
18.10.0.1
92.75+0.1
23+0.
93.25=0.0
WRN-25
94.96:0.0
94.81+0.0
94.55+0.1
33.58+0.1
95.03+0.0
DN-40
93.13.0.1
93.1340.1
43.4140.0
93. 33.3010..1
93.4420.1
VIT-TI
91,7480.06
91.61+0.18
91.37=0.11
80.61
92.60=0.05

---

Table Test perplexity score and test token accuracy of babyGPT and GPT2-S trained on TS and OWT respectively.
Activation
babyGITT
TinyStories Token Accuracy
GPT2-S
OpenWebTesst
Perplexity
Perplexity
Token Accuracy
ReLU
4.51920.006
61.24240.030
17.84510.078
44.099000099
LeakyReLU
4.516+0.005
61.237+0.032
17.778+0. 125
44. 14110300.77 074
ELU
4.872+0.005
59,859#0.022
18.375+0.035
43.721=0.040
GELU
4.46240.00 005
61.46550.00 034
17.525+0.015
44.262+0.044 0.222
Swish
17.785+0:026
44.155=0.025
Mish
4.53940.007
61.135a0.036
17.7974:0.06
44, 10.
GoLUU
4,444=0.005
61.545#0.029
17.297+0.023
44.413=0.023
Table 5: Test loss and test mloU of
Table 6: Test Box mAP of Faster R-CNN- FPN ResNet-50 and RetinaNet-FPN ResNet- 50 trained on MS-COCO.
DeepLabV3 ResNet-50 trained on MS- COCO.
Activation
LRR0000
LR=0.01
Activation
Test Loss
moot
Test Loss
Test Test
Faster R-CNN
RetinaNet
ReLUU
0.34420.003
64.99:0.173
0.35040.004
65.11=0.326
ReL.!
37.44+0. 146
39.90=0.063
LeakyReLU
0.34240.003
64.79s0.122
0,350a0.002
65.5520.131
LeakyReLU
37.4140. 140
39.8740.041
ELU
0.36720.001
59.3120.065
0.358+0.001
60.70+0.089
ELU
35.36+0.041
37.43=0.00 041
GELU
0.341+0.002
64.53+0.136
0.241
65.59+0.. 162
GELU
38. 18.10000.44
40.68+0.090
Swish
0. 0.34840.003
02.52:0.034
0.345+0.002
64. 14#0. 135
Swish
37.2840.078
40.27+0.087
0.34440.001
62.9740.022
0.34240.002
64.4040.
37.7140.087
40.45=0.091
GoL.U
0.33950.000
64.98s0.129
0.341+0.001
65.98+0.124
GoLU
38.31+0.058
40.77=00.65
3.3
Language Modeling
We train GPT TS and GPT2-S (124M) OWT, both sourced from the nanoGPT repository Karpathy (2023]. As shown in Table GoLUU demonstrates superior performance, achieving lower perplexity and higher token accuracy on both babyGPT and GPT2-S. GoLU's superiority is also evident in the test loss curves in [9 and 20 The general trend of GELU being the second-best activation function holds language modeling well. Appendix G.3 outlines the architectural details and provides additional information on the datasets and training settings.
3.4 Semantic Segmentation
For Semantic Segmentation, train DeepLabV3 the MS-COCO dataset with PASCAL-VOC labels, from the Torchvision benchmark (see Appendix G.4). We employ our ResNet-50 backbone, pre-trained on ImageNet-Ik.
Table 5 (Left) presents the test loss and test mloU using the original learning rate of 0.02. GoLU achieves the lowest test loss, whereas ReLU attains the highest mloU, with GoLU ranking second. However, the difference in mloU between ReLU and GoLU is statistically insignificant.
We conduct small ablation study on the learning rate and find that 1r=0.02 is suboptimal for training the model. Instead, Ir=0.01 yields the best performance across all activation functions (see heatmap 10 in Appendix D for full results). Table 5 (Right) reports the results with lr=0.01, where GoLU consistently outperforms other activation functions terms of mloU. Additionally, the inference loss and test mloU curves over epochs, shown in Figures 21 and 22 further emphasize GoL.''s strong performance in semantic segmentation.
3.5 Object Detection
For Object Detection, we train Faster R- R-CNN-FPN and RetinaNet-FPN on the MS-COCO dataset. As shown in Table Figure 23 GoLU outperforms all activation functions on object detection as well, with higher Box mAP (AP @ loU=0.50:0.95, area=all, max across both Faster R-CNN-FPN and RetinaNet-FPN architectures, while GELU ranks second. Appendix G.5 outlines
experimental details.
8

---

Table 7: Test Box mAP and Mask mAP of
Table 8: Test Loss at LR=0.0003 and LR=0.001 of Denoising Diffusion Probabilis- Model trained on CelebA.
Mask R R-CNN-FPN 50 trained on MS- COCO.
Activation
LK=0.02
LR=0.03
Activation
Test Loss
MAP
Mask
MAP
Mask
LR-000003
LR=0,001
ReLUU
13.3340.13
34.1940.1 129
38.2540.14 144
34.28+0. 116
ReL.U
6.020225=0000
0.0192820400
LeakyReLU
$8.31a0.230
14.1920.160
38.42a0.105 105
34.2610.U65
LeakyReLU
0.020030720.0
0.019281244 12
ELU
36.41+0.087
32.81+0.082
35, 35.85+0.069
32.38+0.043
ELU
0.0200398:0.0
0.0193941#0
GELU
39,00+0.033
34.73+0.034
39. 1240.028
34.93+0.061
GELU
0.0196956=0.0
0.0190221
Swish
38. 28.1940.224
33.9940. 168
38. 184
33.95a0.112
Swish
0.0196364:0.0
0.0190055+00
Mish
38.7640.036
34.0040.026
38.5640.
34 34.64+0.078
Mish 0.0196865=0.0
0.019065740
38,9640.
34.54+0.083
39.36+0.192
34.9750.146
0.0196419=0.0
0.0189506+0
3.6 Instance Segmentation
For Instance Segmentation, we train Mask R-CNN-FPN with ResNet-50 backbone from the Torchvision benchmark on the MS-COCO dataset (see Appendix training settings). As shown in Table ( GELU achieves the best performance in this setting (with the default 1r=0.02), with GoLU ranking second in Box mAP and third in Mask mAP (both implying AP IoU=0.50:0.95. area=all, maxDets= 100). However, Figure 24] which depicts test Box mAP and Mask mAP over epochs, reveals that GoLU generally outperforms GELU and ReLU throughout the training process. We perform learning rate ablation study and observe that, similar to the Semantic Segmentation task, a learning rate of 0.02 is suboptimal for this specific architecture-datt dataset combination. contrast, increasing the learning rate 0.03 leads to improved performance across all activation functions (see heatmaps [2] and 13). Surprisingly, at the optimal learning rate of 0.03, GoLU outperforms all baseline activations, shown in Table 7 (right).
3.7 Denoising Diffusion Probabilistic Models
We train Denoising Diffusion Probabilistic Model on the CelebA dataset (see Appendix G.7). As shown in T for the default 1r=0.0033, gated activations perform comparably to the baseline activation, Swish, which achieves the best performance, with GoLU ranking close second. Figure 25 (Left) further illustrates the test loss over epochs. Similar to our findings in semantic segmentation and instance segmentation, conduct learning rate ablation study. Results, summarized in heatmap
I1 in Appendix D indicate that increasing the from the default value of 0.0003 to 0.0004, 0.0005
and 0.001 progressively improves performance across all activations. Notably, for Ir values of 0.0004, 0.0005 and 0.001, GoLU achieves the lowest final test loss. Results for the optimum Ir=0.001 are highlighted in the right column of Table & and Figure 25 (Right). These findings in line with the trend observed in semantic segmentation and instance segmentation, where GoLU outperforms baseline activations under optimal Ir configurations.
Training and Inference Speed
Existing activation functions PyTorch leverage CUDA kernels in Eager mode to achieve optimal speedup. To ensure fair comparison of training and inference speeds, we developed a CUDA- optimized kernel for GoL.U, which was used for all training experiments described in the previous sections. Table in Appendix (eadxxx]ppeeseee presents the relative training and inference speeds of GoLU compared to the default activation function across various tasks.
Our results show that GoLU achieves a speed comparable to that of the default activation function across all architectures. The only exception is DeepLabV3-ResNet-55 trained on MS-COCO, where GoLU incurs slightly higher training time. However, this is consistent with other activation functions, all of which exhibit increased training times relative ReLU in this specific architecture
Conclusions
We have introduced GoLU, a new self-gated activation function based on the CDF of the Gumbel distribution as its gate function. Through extensive analysis and experiments, we have demonstrated

---

that GoLU provides regularising effect by reducing variance in the activation output, it enables the representation of diverse features through more distributed weight pattern, and encourages smoother and more robust loss landscape. Notably, our results show that GoL.U generally outperforms state-of-the-ait baseline activation functions across a wide range of tasks and domains, from computer vision to language modeling. Additionally, we implemented a custom CUDA kernel to optimize training and inference efficiency, minimizing latency and enhancing scalability. GoLU offers a robust, efficient, and scalable alternative to existing activation functions. Its integration into state-of-the-ar neural networks has the potential to improve performance across various applications, positioning GoLU as promising standard in modern deep learning.
Acknowlwdgements
This research was funded by the Deutsche Forschunsseemeisscaattt (DFG, German Research Foun- dation) under grant number 539134284, through EFRE (FEIH 2698644) and the state of Baden- Württemberg.
Co-funded by the European Union
Baden-Württemberg
The authors gratefully acknowledge the computing time made available to them on the high- performance computer NHR @ KIT Compute Cluster at the NHR Center NHR @ KIT. These Centers are jointly supported by the Federal Ministry of Education and Research and the state governments participating the NHR European Union (via ERC Consolidator Grant DeepLearning 2.0, grant no. 101045765). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them.
partterr..... We acknowledge funding by the
Funded by the European Union
The authors gratefully acknowledge the Gauss Centre for Supercomputing e. for funding this project by providing computing time on the GCS Supercomputer JUWELS at Jülich Supercomputing Center (JSC) and SuperMUC-NG Leibniz Supercomputing Centre (www.lrz.de). The authors acknowledge support by the state of Baden- Wireen--rtteeberrgg through HwHPC. We acknowl- edge the EuroHPC Joint Undertaking for awarding this project access to the EuroHPC supercomputer LEONARDO, hosted by CINECA (Italy) and the LEONARDO consortium through an EuroHPC Extreme Access grant BHHCCEXX-2023E00--66
Frank Hutter acknowledges the financial support of the Hector Foundation. We also thank Jörg Franke, Edward Bergman, Arbër Zela, André Biedenkapp and Lennart Purucker for their constructive feedback throughout the development of this work.
References
Steven Adriaensen, Herilalaina Rakotoarison, Samuel Müller, and Frank Hutter. Efficient bayesian learning curve extrapolation using prior-data fitted networks. Advances in Neural Information Processing Systems, 36, 2024.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.064500 2016.
Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv: 1700::0066.5877 2017.
Djork-Aréé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint 15 2015.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEFF/CVE conference on computer vision and pattern recognition, pages 113-123, 2019.
10

---

Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 702-703, 2020.
Janez Demšar. Statistical comparisons of classifiers over multiple data sets. J. Mach. Learn. Res., 7:
1-30, December 2006. ISSN 1532-4435.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition. pages
IEEE, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11992 2020.
Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english? arXiv preprint FrXXi:2305.07755 2023.
Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. Openwebtext corpus. http: //Skylion007.gitoa 1o o0/0peeneeTTextCorpuus 2019.
Benjamin Gompertz. On the nature of the function expressive of the law of human mortality, and on a new mode of determining the value of life contingencies. a letter to francis baily, esq. frs &c. Philosophical transactions of the Royal Society of London, (115):513-583, 1825.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition. pages 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08411 2016.
Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning lecture 6a overview mini-batch gradient descent. Cited on, 14(8):2, 2012.
Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4700-4708, 2017.
Sergey Toffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448-456. pmlr, 2015.
Andrej Karpathy. nanogpt. httpps//ggthaa.ccm///aaaaaaaaaaa 2023. Accessed: 2024- 09-10.
Jongbeom Kim. Ddpm. https://gthhhb.ccm////nnssssss/s
2023. Accessed:
2024-09-13.
Yann LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Robert Müller. Efficient backprop. Neural networks: Tricks of the trade, pages 9-50. Springer, 2002.
T Lin. Focal loss for dense object detection. arXiv preprint
2017.
11

---

Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollär, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014. doi: 1010077978-33311-1002211 48.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. arXiv preprini 1rXiv:1711.051000 5, 2017.
Andrew L Maas, Awni Y Hannum, Andrew Ng, et al. Rectifier nonlinearities improve neural network acoustic models. In Proc. ieml, volume 30, page 3. Atlanta, GA, 2013.
Diganta Misra. Mish: self regularized non-monotonic activation function. arXiv preprint
arXiv:1908.0860 2019.
Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter. Transformers can do bayesian inference. arXiv preprint
2021.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pages 807-814, 2010.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLL 2019: Demonstrations, 2019.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. arXiv preprint arXiv:1710.05941, 2017.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances neural information processing systems, 28, 2015.
David Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-propagating errors. nature, 32336089) 1986.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-2826, 2016.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference machine learning, pages 6105-6114 14. PMLR, 2019.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve supe-supe deep learning results. Advances in neural information processing systems, 30, 2017.
Torch Vision. Torchvision: Pytorch's computer vision library. vision 2016. Accessed: 2024-09-10.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint 1rxiv:1607.00000 2016.
Pierre François Verhulst. Notice sur loi que la population poursuit dans son accroissement. Correspondance Mathématique Physique, 10:1 10:113-121, 1838.
Ross Wightman. Pytorch image models.
http:://gghhub..omnnngiigggee//
pytorch- image- models, 2019.
12

---

Kan Wu, Jinmian Zhang, Houwen Peng, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Tinyvit: Fast pretraining distillation for small vision transformers. In European conference on computer vision (ECCV), 2022.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference computer vision (ECCV), pages 19, 2018.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVE international conference on computer vision, pages 6023-6032, 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146. 2016.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances Neural Information Processing Systems, 32, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2017.
13

---

A Properties of GoLU: Further Details
To further elucidate the concepts presented in Section and gain deeper insights into the properties of GoLU, we present additional details and visualizations this section.
Figure 7 compares the GoLUU activation with GELU, highlighting how the right-leaning inclination of the Gumbel distribution, in contrast to the symmetrie Gaussian distribution (Left column), results in smaller value of the Gompertz gate at the origin compared to the Gaussian CDF (Middle column). In fact, this behavior is not confined to the origin, and the Gompertz gate remains smaller than the Gaussian CDF across the entire input range.
Thesse
Seace
02
02
0.5
0.3
Sue
0.7
1bpe 0.27
03
Figure 7: Top row, from left to right: Gaussian distribution, Gaussian CDF, GELU, Bottom row, from left to right: Gumbel distribution, Gompertz function, GoLU.
This reduced value of the Gompertz gate at the origin directly translates into a lower slope for GoLU compared to GELU, illustrated in Figure ( (Right column). This can be readily seen by taking the derivative of the GoLU activation and evaluating it at zero
GoLU'(z) = x Gompertz' (ι) + Gompertz(z) GoLU' (0) Gompertz(0)
(5)
(6)
which shows that the slope of GoLU at the origin corresponds to the value of the Gompertz gate the origin. Similarly, the slope of GELU at the origin is determined by the Gaussian CDF at the origin.
Assuming the input distribution resembles zero-centered, rearly-Gaussiar form, which is likely particularly when employing batch normalization and appropriate weight initialization, the activations can be approximated by their tangents at the origin. Therefore reduced slope the origin translates into decreased sensitivity to input variations and lower output variance. We note that GoLU exhibits lower slope magnitude not only in neighborhood around the origin but across significant portion of the negative input domain as illustrated in Figure 2
More generally, it can be shown analytically that for given activation function f that is smooth in neighborhood of its input mean, and for a sufficiently localized imput distribution, the variance of the activation output is approximately proportional to the square of its slope evaluated at the mean input μ, with the input variance o² serving as the proportionality constant
Varlf(z)] so f'())002
This formally demonstrates how smaller activation slopes result reduced output variance.
To derive this connection, we apply the definition of variance to a scalar activation function f(x):
Varif [()((z) ( B[f()))) EI [=)()) [=)
(8)
14

---

Expanding fx) and fx))² 9 in a Taylor series around the mean input μt, gives:
f(x) f(n) + f') - n)) + [n)((( 12) +...
(9)
f(x)² = 2f μ) + ('( -1) (w) +... (10)
Taking expectations and using E[( - [²] and Ei(er - 11} = o² leads to:
E[f(z)] f(1) 1 {if()) = f(n) + f + (
( ( +...
(11)
2
(12)
Substituting these into the definition of the variance, and simplifying while retaining only the reading-order term, we obtain:
Var[f(=)] (f(m)* + f2) + f()) +...) (f(x) 1 (3([wo++...)) (13) 2 + (() +..)) -ff()) f(w) X14)
0²
(15)
which completes the proof.
Finally, Taylor expansion of the Sigmoid and Gompertz gate functions for large positive input values demonstrates that these two functions converge to each other exponentially fast in this regime, pointed out in Section 2.1
(i) Gompertz(1)
1 + e- -
(16)
-- 0le-2r)) e0 - Ole ((2-)))
2n))
(17)
O(e 2r)
(18)
Flipped Mish: a new self-gated activation with right-leaning distribution
Throughout this work, we have emphasized the influence of right-skewed asymmetry in the underlying distribution associated with an activation model performance. To further explore this property, we leverage the left-skewed distribution underlying Mish to construct a new self-gated activation function exhibiting fight-skewedd asymmetry. This is achieved by reflecting the Mish-associated distribution about the vertical axis. Specifically, denoting the Mish distribution by D(x) and its corresponding gate function by g(x), the gate function for the flipped distribution D(c) = D( is given by g(x)
g(-x), shown by the following derivation:
100 g(x) = I dy D(y) = -f dy D(-9) = { dy D(y) = | dy D(v) - dy D(v) = = 11 g(-x)
(19)
where the third equation we have redefined the dummy integration variable y - -y and in the last equation have used the fact that the integral of the distribution D(y) over its entire domain is equal to 1. The resulting activation function, which we refer to as Flipped Mish (FMish), is thus defined
FMish(æ) = I(1 - taat(ssftpluu((=z))
(20)
Figure compares FMish with GoLU, including their respective gate functions and the associated distributions.
15

---

Hina Moidecde
Figure 8: Comparison of activations (left), gate functions (middle), and associated distributions (right) for FMish and GoLU.
We evaluated FMish on ResNet18, ResNe150 and ViT-B/32 trained on ImageNet-Ike Ik. Remarkably, outperformed all baseline activations except GoLU, achieving 73.03 05 for ResNet18, 76.20:10.01 for ResNet50 and 75.67 = 0.04 for ViT-B/32 (compare with results in Table 2). This outcome aligns with expectations, the slope of FMish the origin is 0.4. lower than that of sigmoid and Gaussian CDF (0.5) but slightly higher than GoLU (0.37). These results further highlight the significance of right-leaning asy asymmetry and the resulting variance reduction.
Furthermore, we note that the Flipped Mish distribution does not decay as rapidly as the Gumbel distribution for large negative inputs, which may also contribute to its performance.
C Details of the loss landscape experiment
We analyze the loss landscape of neural network by quantitatively measuring and visualizing how the loss changes as the network's parameters are perturbed. Smoothness the loss landscape often indicates that small perturbations in the parameters do not cause large changes in the loss, which can make optimization more stable.
Specifically, we generate two random perturbation directions di and d2, each matching the shape of the model parameters. The elements of these directions are independently sampled from a Standard Normal distribution. To ensure controlled magnitudes, each perturbation direction is subsequently normalized.
We perturb the weights of the model along these directions in a linear combination:
W pertarbed = Wtrained + ad1 + 3dd2
(21)
where W Wtrained the trained weights of the model and and [ are scalar values that determine the perturbation magnitude and are chosen as a, 3 E [-1, 1]. For each pair of values (a, B), we compute the loss using the perturbed weights W perturbed repeat this for a grid of (ax, 3) values to create a 3D surface plot as shown in Figure 5
the full test set of the CIFAR- 10 dataset. We then
To provide more quantitative understanding of the loss landscapes in Figure 5 we have plotted the density functions of the loss values for each activation function and computed their variance. The results, shown in Figure 9 indicate that GoLU achieves both lower average loss and smaller variance compared to other activations, consistent with the observations from the 3D plots in Figure
5
D Learning Rate ablation
For various tasks, we conduct focused search over the learning rate to determine whether the default setting represents the optimal value and to assess its impact on the performance of models trained with differrnt activation functions. Figures 10 and III present heatmaps of test results for Semantic Segmentation and Diffusion tasks, comparing models trained with various activation functions across different learning rates. Figures and and show similar heatmaps for the Instance Segmentation task, reporting box mAP and mask mAP, respectively. For these tasks, the default learning rate, highlighted by a black box, differs from the optimal learning rate, indicated by a green box. Notably, while GoLU performs slightly below the best dest-performin activation under the default learning rate, it outperforms all other activation functions when evaluated at the optimal learning rate, which is consistent across all activations.
16

---

140
S411 (Ver: 6981
GELU Dear 8544
Swish 1.9000--28
(er
130
0:35
8:30
1.30
640
Loss
Figure 9: Comparison of loss value distributions across the loss landscape. GoLU achieves both a lower loss mean and variance.
66-222
+1221
+0312
61-1 T.34
0.200913
6-20-4010
00317
32.140
1..05
20.00
+30004
+0.000
46.000
840 M0101
6.20
+0.31
+2.
+32004
+00.00
+1.11
44
+0600
03303442
141 MUCC
0.000011
0.66
J
Figure 10: Test mloU D3 on MS- CÔCO. The default learning rate 0.02 which is colored in black and the best learning rate is 0.01 which colored in green.
Figure 11: Test Loss DDPM on CelebA. The default learning rate is 0.0003 which is colored in black and the best learning rate is 0.001 which is colored in green.
20.30
144
64
KM
23/00
01.11
200
40.00
48.33
00.00
050
$5.04
20.00
23:.00
ZACHE
1434
00.043
00:030
10.300
413
013
18:00
13/
25.003
60.000
00.343
-0.03
11/
202272
14110
+0.14
41:11
-0.044
10.000
212-100
29 4223
GLUREE
60.28
90.
306
10/17
-0.30
-0:301
14
22.8
34 0175
00.210
0.33
600
645
3,69
0.4
0.85
0311
0000
0355
Figure 12: Test Box mAP- Mask R-CNN-PFN ResNet-50 MS-COCO. The default learning rate is 0.02 which is colored in black and the best learning rate is 0.03 which is colored in green.
Figure 13: Test Mask mAP- - Mask R-CNN-PFN ResNet-50 on MS-COCO. The default learning rate is 0.02 which is colored in black and the best learning rate is 0.03 which is colored in green.
Motivated by these results, we further investigate the impact of learning rate on image classification tasks where GoLU demonstrated superior performance compared to bascline activations. Figures 14 and 15 present heatmaps of test accuracies for ResNet-50 and ViT-B/32 on ImageNet-Ik.
17

---

mM tesgpeal 6411
43.40
2.20
20.4
00.04
2.14
0110
+3.00
40.000
2040
6004
22040
43130
2112
2150
11304
40204
40004
20091
4003
2.21
40305
60005
40453
+9034
60.30
2.24
197.5
11.27
2.21
10.77
A.30
00-113
21-25
00.01
197.3
+1.00
2.34
18:07
20.01
2.15
00000
137.0
0.14
6:301
16001
6.11
0M0L
Figure 14: Test accuracies ResNet-50 on ImageNet-Ik. The default learning rate is 0.1 which is also the best and is colored in green.
Figure 15: Test accuracies ViT-B/32 on ImageNet-Ik. The default learning rate is 0.003 which is also the best and colored in green.
Notably, we observe that the optimal learning rate aligns with the default learning rate in this case. These findings reinforce the broader trend that, with few exceptions, GoLU consistently outperforms baseline activation functions across lasks when evaluated at the optimal learning rale.
E Critical Difference Analysis
In this section, we conduct Critical Difference analysis following Demsar (2066) to systematically rank activation functions based on experiments performed on ImageNet-Ik, MS-COCO, OWT, TS, and CelebA. As shown Figure T6 GoLU achieves the highest rank, followed by GELU. Notice that the confidence interval in this analysis is independent of the variance across multiple runs with different random seeds. Instead, is determined by the number of models and datasets, as well the significance level, which is set to = 0.05 here.
ELU
CoLU
Swish
CELU
LeaxyReLU
Figure 16: Critical Difference diagram, ranking activation functions based on average performance.
F Training and inference times
Table reports relative training and inference times with respect to baseline activations for our trained models. On average, GoLU achieves training and inference speeds on par with default activation functions, while offering improved performance. This makes GoLU a practical and effective alternative for training deep learning models.
G Experimental Details
This section outlines detailed information about the datasets and training pipelines used for the various tasks studied in this work. All experiments in this section were conducted on NVIDIA A100
18

---

Table 9: Relative training and inference time with respect to baseline activations for our trained architectures.
Baseline Activation
Relative Training Speed
Relative Inference Speed
Architecture
Dataset
ResNet-18
ImageNNet-1K
ReLU
1.00x
1.00x
ResNet-34
ImageNet-11
ReLU
1.01x
1.00x
ResNet-50
ImageNet-1k
ReLU
1.01x
1.01x
WideResNet-50-22
ImageNet-Ii
ReLU
1.03x
1.02x
DenseNet-121
ImageNet-Ik
ReLU
1.02x
1.02x
EfficientNet-B0O
ImageNet-
Swish
1.00x
1.00x
ViT
ImageNet-1k
GELU
0.99x
0.98x
VIT-B/32
ImageNet-Ik
GELU
0,99x
0.99x
VIT-B/16
ImageNet-Ik
GELU
0.98x
0.98x
babyGPT
TinyStories
GELU
1.00x
1.00x
GPT2-S
OpenWehText
GELU
1.01x
1.01x
DeepLab
MS-COCO
ReLU
1.14x
1.04x
RetinaNet
MS-COCO
ReLU
1.00x
1.00x
FasterRCNN
MS-COCO
ReLU
1.03x
1.00x
MaskRCNN
MS-COCO
ReLU
1.05x
1.02x
DDPM
CelebA
Swish
0.97x
0.97x
Average
1.01x
1.00x
GPUs, with an approximate total compute time of I 12K GPU hours, except for Tiny ViT, which was executed on an NVIDIA H100 GPU with a total runtime of 455 GPU hours.
G.1 Image Classification ImageNet
In image classification experiments on ImageNet-Ik, ResNets 18, 34, 50, WideResNeet50-22 and DenseNet-121 1 are trained for 90 epochs with a batch size of 256, SGD with momentum=0.9 (Nesterov for WRN-50-2 and DN-121), learning rate 0.1, and weight decay 1 x 10-1 Further, a Step learning rate scheduler is applied that reduces the learning rate by gamma 0.1 after every 30 epochs. trained using the timm library for 450 epochs with a batch size 1536 using RMSProp Hinton et al. [2012) with an initial learning rate of 0.048 and weight decay of 1 10" ViT models are trained for 300 epochs with a batch size of 4096 using Adam Loshchilov and Hutter (2117) with initial learning rate of 3 x 10-3 and weight decay of 0.3. Various regularization techniques are applied, including Exponentially Moving Averaged Weights Tarvainen and Valpola
EfficientNet-BD
(017), Autmmng Cubuk al. [2099] (ImageNet policy for ViTs), RandAugment Cubuk et al.
2020 MixUp Zhang et al. [201]] CutMix Yun et al. 2019) and Label Smoothing Szegedy et al.
(016 for EfficiietNNet-BO and VIT models. ViT-B/16 shows slight instability for seed for GELU.
Hence we further average seeds 2 and for both GELU and GoLU. We find that GELU shows top-1
accuracy of 80.61 0.06 while GoLU shows top-1 accuracy of 80.69 + 0.07 which higher than
GELU.
G.2 Image Classification CIFAR-100
The ResNet 20, 32, 44, 56 and 110 models are trained for 164 epochs with a batch size of 128, learning rate of 0.1, and SGD with momentum 0.9. A weight decay
x 10 4 is applied, along with MultiStep learning rate scheduler with gamma factor of 0. at epochs 81 and 122 (with an initial learning rate of 0.01 and additional gamma factor of 10 at epoch 2 for ResNet-110).
WideResNer28- and DenseNet40, were trained for 200 and 300 epochs, and batch sizes of 128 and 64, respectively. We employ SGD with Nesterov momentum 0.9 for both architectures, using a learning rate of 0.1. The weight decays are 10-4 for WideResNer28-2 and 1 x 10-4 for DenseNer40. Similar to ResNets, both WideResNer28-2 and DenseNet40 use the MultiStep learning rate scheduler. However, WideResNer28-22 reduces the learning rate by factor of 0.2 at epochs 60, 120, and 160, while DenseNer40 reduces the learning rate by 0.1 at epochs 150 and 225. To train ViT-Ti/16-22 2-24 from scratch, we leverage the Timm library.
19

---

G.3 Language modeling
Both, TinyStories and Open WebText datasets are popular benchmarks for training language models. The TinyStories dataset consists 2. 1,999999 data points in the training set and 21, 21,990 the test set, while the Open WebText dataset has 8,009,762 7,26 data points in the training set and 4,007 data points in the test set. Both babyGPT and nanoGPT have a vocabulary size of 50,304 and a maximum sequence length of 1024.
The babyGPT version of the GPT-2 series consists of layers, 6 attention heads, and embedding dimension of 384, with a feed-forward expansion dimension of 1536 output features. The model trained for 10,000 iterations with a batch size of 640, using the Adam W optimizer. The initial learning rate is x 10 with a minimum learning rate of 1 x 10 10-+ weight decay of 0.1, and gradient clipping norm of 1.0. A Cosine learning rate scheduler is applied with linear warmup for the first 100 iterations.
Similarly, the GPT2-S model consists of 12 layers, 12 attention heads, and embedding dimension of 768. It trains for 600,000 iterations with a batch size of 480, using the Adam W optimizer (with 32 - 0.95). The initial learning rate is x 10-4, with a minimum learning rate of 6 10-$, 5 weight decay of 0.1, and gradient clipping norm of 1.0. The Cosine learning rate scheduler is employed with linear warmup for the first 2,000 iterations.
G.4 Semantic Segmentation
The MS-COCO dataset with PASCAL-VOC labels contains 92,518 data points the training set and 5,000 data points in the test set. The original MS-COCO dataset contains 1 117,266 data points in the training set. However, the existing benchmark pre pre-processes and removes images that either lack valid annotations contain only small objects with an area coverage of less than 1,000 pixels. This ensures the retention of meaningful data points for training the model.
The Deepl ab..aba3V--- W--VV3--R-N---- model is trained for 30 epochs with batch size of 32, using SGD with momentum 0.9, learning rate of 10-2, weight decay of x 10-4, and polynomial learning rate scheduler with power of 0.9.
G.5 Object Detection
Unlike Semantic Segmentation, the MS-COCU dataset for object detection contains 117,266 images in the training set and 5,000 images in the test set. Additionally, we do not apply any pre- pre-processing that removes images from the training or test sets.
Faster R-CNN-FPN ResNet-50 and RetinaNet-FPN ResNet-50 are trained for 26 epochs with a batch size of 16, an aspect ratio group factor of 3, frozen batch normalization, and MultiStep learning rate scheduler that reduces the initial learning rate by a factor of 0.1 at epochs 16 and 22. Specifically, Faster R-CNN-FPN ResNet-50
SGD with momentum 0.9, learning rate of 2 x 10-2, and a weight decay of x 10-4, while RetinaNet-FPN ResNet-50 uses the Adam W optimizer with learning rate of 1 and a weight decay of 5 1
G.6 Instance Segmentation
The MS-COCO dataset for instance segmentation uses the same train and test sets as those used for Object Detection. Additionally, it trains with the exact same configurations used for Faster R-CNN-FPN the previous subsection G.5
G.7 Denoising Diffusion Probabilistic Models
The CelebA dataset, comprises of 162,770 training images and 19,867 test images of human faces. The Denoising Diffusion Probabilistic Model is trained on the CelebA dataset for 50 epochs with a batch size of 32 leveraging the DDPM Kim 2023) repository. The AdamW optimizer with learning rate of 0.0003, Cosine learning rate scheduler, and linear learning rate warmup for the first 1,000 iterations are applicd.
20

---

H
Test Loss Curves
To provide comprehensive view of GoLU's test performance the course of training, this section presents test curves, including loss and task-specific metrics, comparing GoLU with ReLU and GELU, and illustrating how their performance evolves throughout training.
GoLU
GoLU
GELU
GELU
Top-1 Accuracy (in %age)
ReLU
ReLU
3.3
L,655
1.50
098
Epochs
Epochs
Figure 17: ResNet-50 test loss (Leftt) and test top-1 accuracy (Right) on ImageNet- 1k.
GoLU
GoLU
GELU
GELU
Top-1 Accuracy (in %age]
ReLU
ReLU
LOSS
2.0
220
Epochs
Epochs
Figure 18: ViT-B/32 test loss (Left) and test top-1 accuracy (Right) on ImageNet-Ie l.
61.9
GoLU
GoLU
GELU
Token Accuracy (in %age)
GELU
Swish
Swish
338
01.2
Lo55
31.0
3.1
ana
3.49
1500
7503
1200
8000
9000
2008
20003
7000
7500
$300
42.00
5000
3803
10000
Iterations
iterations
Figure 19: babyGPT test loss (Left) and test token accuracy (Right) on TS.
21

---

455
GoLU
GoLU GELU
152
GFLU Swish
Token Accuracy (in %age]
Swish
LOSS
es
2.54
17593
07000
552206
/90000
175200
525065
S15030
50065
lterations
Iterations
Figure 20: GPT2-S loss (Left) and test token accuracy (Right) on OWT.
GoLU
GoLU
ReLU
ReLU
LOSS
mloU
035
Epochs
Epochs
Figure 21: DeepLabV3 ResNet-50 test loss (Left) and test mloU (Right) on MS-COCO with 1r=0.02.
0.5
GoLU
GoLU
550
GELU
GELU
ReLU
ReLU
800
LOSS
mlou
10.0
0.30
30.
Epochs
Epochs
Figure 22: Deepl.abV3 ResNet-50 test loss (Lefft) and test mloU (Right) on MS-COCO with 1r-.001.
22

---

GoLU
GoLU
GELU
GELU
mAP
273
21.
Epochs
Epochs
Figure 23: Faster R-CNN-FPN ResNet-50 (Left) and RetinaNet-FPY ResNet-50 (Right) test Box mAP on MS-COCO.
GoLU
GoLU
GELU
GELU
mAP
Epachs
Epochs
Figure 24: Test Box mAP (Left) and test Mask mAP (Right) for Mask R-CNN-FPN ResNet-50 trained on MS-COCO.
W
64134
6:230
002900
LOSS
61:00
811975
6:030
64339
GoLL
GoLU
GELU
611.8
GELU
Swish
Swish
61332
810075
Epochs
Epochs
Figure 25: Test loss for Denoising Diffusion Probabilistic Model trained on CelebA at LR=0.0003 (Left) and LR=0.001 (Right).
I Machine Translation
To further assess GoLU across diverse tasks, we evaluated its performance on machine translation using the WMT14 English-German benchmark, with approximately 4.5 million training pairs. Specifically, we trained Transformer-Big models using the Fairseq framework [Ott et al. 2019), comparing GoLU against baseline activations including ReLU, which is the default in this architecture. The architecture follows the standard configuration with encoder and 6 decoder layers, 1024- dimensional embeddings, 16 attention heads, and feed-forward hidden size of 4096. Models
23

---

were trained with three different random seeds for 50 epochs using the Adam optimizer (6) = 0.9, 31 0.98), an inverse square root learning rate schedule (base LR 5 10 4000 warm-up steps), label smoothing of 0.1, and gradient accumulation of 16 to simulate large-batch training. Evaluation was conducted using beam search with BLEUU as the performance metric. All runs were executed on single NVIDIA L40S GPU with a total runtime of roughly 1750 GPU hours. As shown in Table GoLU outperforms standard activation functions in terms of mean BLEU4 score, which highlights its effectiveness in sequence modeling as well.
Table 10: Mean and standard error of BLEU4 scores for Transformer-Bi on the WMT14 En-
Blish-German translation task.
Activation
Test BLEU4
RcLU
28.33+0.14 14
LeakyReLU
28.2640.04
ELU
27.49+0.08
GELU
28. 28.20+0.04
Swish
28.3440.08
Mish
28.3120.10
GoLU
28.44:0.15
J Case Study: Bayesian Learning Curve Extrapolation using Prior-data fitted Networks
In this section, we present an additional experiment on GoLU, initially conducted as an internal validation study. We report this as a "negative" result, with GoLU ranking second-to-last under the optimal learning rate. Due to the unconventional experimental setup, its niche focus, and suboptimal hyperparameter tuning, we have included these findings in the appendix rather than in the main text.
Experimental Details In this experiment, we assessed all 7 activation functions (including GoLU)) considered in the main article as activations for LC-PFN Adriaensen et al. (2024). LC-PFN is prior-data fitted network Müller et al. [2021] that functions as a decoder-only transformer, trained for in-context Bayesian prediction for a specific prior dataset distribution. Specifically, LC-PFN is trained for Bayesian Learning Curve extrapolation. We adopted the same setup used to train the best model presented in the original paper, a decoder-only transformer having 26.79M trainable parameters, 12 layers, 4 attention heads, an embedding dimension of 512, and a feed-forward expansion dimension of 1024 output features. was trained using 10M synthetically generated learning curves, (each containing 100 observations), employing the Adam optimizer (with default learning rate of 0.0001 and batch size of 100), using a cosine scheduler with a linear warmup during the first 25,000 steps (25%) of the training. At test time, it takes partial learning curve as input, and predicts the posterior predictive distribution (PPD) for possible continuations. The test performance the final model was measured using the log-score, which represents the log-likelihood of the true continuation, under the PPD, averaged across all 99 cutoff's for 10,000 curves from the prior.
Results Figure 26 presents the log-scores for the final models, utilizing all 7 activation functions at different learning rates, averaged over training runs. At the original and optimal learning rate of 0.0001, GoLU ranks 6th among the 7 activations. However, a closer examination reveals that the choice of activation function seems have minimal impact, as the differences between GoLU and the best (ELU) and worst (Swish) activation are within a single standard error. The learning rate ablation shows that GoLU ranks first at the highest stable learning rate (0.001), supporting previous findings that GoLU thrives in the high learning rate regime.
K Limitations
The effects attributed to GoLU, as described in Section 2.2 are not guaranteed to hold universally but rather represent general trends observed in our empirical findings. Moreover, while asymmetry has been highlighted as distinctive feature of GoLU, its high performance. detailed in Section
24

---

11 151L
402
0000
40
0000
+6
01111
60037
15 015
10:30 00-00
16 8C445
8010
000%
Figure 26:
Test log scores LC-PFN. The default learning rate is 0.0001, which is also optimal, is highlighted in green.
cannot be solely attributed asymmetry, but instead arises from an intricate interplay of properties described in Section 2.1
Additionally, note the importance of learning rate tuning in realizing the full benefits of GoLU. Although we generally observed that, under tuned learning rate, which also improves the perfor- mance of all baseline activations, GoLU achieves superior results, it is not guaranteed to outperform baseline activation functions under default hyperparameter settings.
L Broader Impact
This paper introduces GoLU, a novel activation function designed advance the field of Machine Learning. The primary objective of this work is to improve the performance and robustness of state-of-the-ard neural networks across diverse domains, including computer vision, natural language processing, and generative modeling. The societal impact of this work is primarily tied to the downstream applications machine learning models that may incorporate GoLU. By enhancing the robustness and performance of models, our activation function has the potential to positively influence critical areas such as medical imaging, autonomous systems, and other technologies that drive societal progress. While there are no immediate direct societal concerns specific to GoLU itself, as with any development in machine learning, there is a possibility of misuse. We therefore emphasize the importance of ethical and responsible deployment of machine learning technologies enhanced by our contributions.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in the abstract and introduction are supported in the main paper and appendix particularly in Sections and
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are discussed throughout the paper and summarized in Appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-speciiiiatinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
26

---

Answer: [Yes]
Justification:
We include the full derivation of equation 4 in Appendix A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section G provides the experimental details reproduce the results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
27

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: the end of the introduction we provide the link to our GitHub repository fttts://ggtthe com/ automl automl/GoLU which includes complete instructions to run the experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.com cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
wwwp://wweeeeeeeeeeeeeeeee Caaeae//ddoeSSbbbiica wwwpeddeuumisssioopoo. wwiiiisssiiiicyiyioo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section G provides the details of the experimental settings. We further provide the link to our public repository.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report standard errors for all the results in the paper, and this is highlighted in Section [.1] In Section D we further provide a Critical Difference analysis to rank activation functions and illustrate the corresponding confidence intervals.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
28

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Sections Gad
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
The research fully conforms with the NeurIPS Code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See Section
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
29

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All creators original owners of the assets used in the paper properly cited, and all assets are covered by open-access licenses.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
30

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta
latasets.papersstt
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification: The new asset introduced in the paper is the code for GoLU, which is currently provided through the GitHub repository httt::///gihabb..cc//tttt///// com/automm/CollJ and is well documented.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
31

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMS as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neurips hett::///eerrrpp.ccc//ooerrreere cc///onferreeee//222////... for what should or should not be described.
32


Reviews:
[{"review_id": "mMeUeBBrz6", "reviewer": "Reviewer_g3hi", "summary": "The authors propose Gompertz Linear Units (GoLU), a novel activation function based on the Gompertz function. The paper formally defines GoLU and analyzes its key properties. In particular, the authors discuss its role during training, highlighting that it induces a smoother loss landscape and a more spread-out weight distribution, they are considered better compared to previous activation functions. Experimental results show that the proposed method achieves better performance across a wide range of tasks, including but not limited to image classification, object detection, image generation, and language modeling.", "questions": "It should benefit the readability and clarity of the paper if some of the theoretical parts in the appendix are included in the main body.", "limitations": "NA", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Pros\n1.\tThis paper is well-written.\n2.\tSignificant improvements have been achieved in many commonly used benchmarks.\n3.\tIn addition to strong baseline performance, the paper includes numerous visual comparisons (including those in the appendix) that help illustrate the advantages of the proposed method.\n4.\tAnonymous code repo is provided, supporting reproducibility.\nCons\n1.\tThere could be more visualization results, such as Table 1, and perhaps an ImageNet one, which would be more convincing.\n2.\tThe proposed method shows impressive performance improvements. However, the main text lacks theoretical explanations that help readers understand why the method works. Also, I noticed that the appendix contains some important analysis, which would strengthen the paper if included in the main body", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their positive and constructive feedback. Below, we provide our response.\n\n**Con 1:**  \nWe have indeed conducted an ImageNet equivalent of the experiment in Table 1 and observed a qualitatively similar variance reduction effect. However, we chose to include the CIFAR-10 results because, to the best of our knowledge, ImageNet images cannot be shared publicly due to license restrictions. In the revised version of the paper we will explicitly mention that the same effect is observed on ImageNet.\nAdditionally, Figure 4 offers a similar analysis on randomly sampled ImageNet images but shows the full activation density distributions. We would also like to highlight that this variance reduction effect is not limited to the final activation, as shown in Figure 4, but is consistently observed across intermediate layers of ResNet-50 as well. We will add a note about this in the updated version.\n\n**Con 2 & Question:**  \nDue to space constraints, we had to place several interesting results in the appendix. If the manuscript is accepted, we will have an additional page to move some of the more important results from the appendix, including some theoretical arguments, into the main body.\n\nWe hope that our responses have addressed the concerns raised by the reviewer. If there are any further questions or points that could help strengthen your view of the paper, we would be happy to continue the discussion at any time before the end of the discussion period on August 6th. Thank you."}, {"review_id": "kZmhq1bUqy", "reviewer": "Reviewer_qitQ", "summary": "The paper introduces a novel activation function, Gompertz Linear Unit (GoLU). The authors present GoLU as a self-gated activation that reduces variance in latent representations and promotes smoother gradient flows.\nThe experimental results show improved performance across various tasks such as image classification, language modeling, semantic segmentation, object detection, etc.", "questions": "Q1. Could the authors offer deeper theoretical analysis or empirical evidence to rigorously justify the claimed advantages of rightward asymmetry in GoLU?\n\nQ2. Do the performance gains transfer with advanced training procedures as in [1] mentioned in W1? Further, can GoLU seamlessly replace ReLU in pretrained models and lead to better performance in a fine-tuning setting on a downstream task?\n\nIf Q2. is answered I could raise my score, because as the paper's contribution relies on the experimental part, I would like to see a significant gain before granting acceptance.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 1, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n- S1 : Well written paper that is easy to follow. The introduction of GoLU is clear and straightforward to understand.\n\n- S2 : The amount of experiments is commendable, with a comprehensive empirical evaluation on diverse benchmark datasets. As this paper contribution is empirical, this is a good points (however see W1).\n\n- S3 : The code is available and a Cuda kernel was developed. This allows for the community a seamless integration and use of GoLU.\n\n---\n\n**Weaknesses:**\n\n- W1 : The improvements in performance, while seamingly widespread, appear (very) modest in magnitude, raising questions about their real-world impact. Furthermore, as different well-known training procedures would lead to better baselines (with even better results than those reported for GoLU), I am not convinced by the usefulness of this new activation function. I give a couple of examples:\n\n    a) Table 2 and 3 baselines are a bit questionable.\n\n    - for table 2, the results and training procedure provided by [1] with even the cheapest procedure (A.3) lead for a ResNet50 on Imagenet to 78.1 % while the paper reports 75.44% for ReLU and 76.63% for GoLU.\n\n    - for table 3, as public implementations, for example [this git](https://github.com/akamaster/pytorch_resnet_cifar10) can lead to much better results with ReLU. I am a bit worried about the reported gains, see the table below:\n| Name       | Test err (original impl.) | Test err (public git impl.) | Test err (reported for ReLU) | Test err (reported for GoLU) |\n|------------|------------------|------------------------|------------------|------------------|\n| ResNet20   | 8.75            | 8.27                  | 8.59        | 8.23        |\n| ResNet32   | 7.51            | 7.37                  | 7.79        | 7.31        |\n| ResNet44   | 7.17            | 6.90                  | 7.42        | 7.15        |\n| ResNet56   | 6.97            | 6.61                  | 7.20        | 6.85        |\n| ResNet110  | 6.43            | 6.32                  | 6.79       | 6.75        |\n\n   b) The Perplexity gains reported in Table 4 are in my opinion, not significant enough to be convincing and one could wonder whether when applied on larger language models, GoLU can lead to better results.\n\n- W2 : The discussion on smooth/flat landscape is not an argument.\nFigure 5, meant to demonstrate smoother loss landscapes, lacks clarity and fails to convincingly support the authors' claims, regarding that point (projecting a ~250k dimensional space onto a 3 dimensional one is not very interpretable). Further, as discussed in [2], these metrics are to be handled with extra attention and not just given as arguments.\n\n- W3 : The theoretical contribution is limited, as the paper primarily offers incremental enhancements over existing methods based on an established mathematical function, rather than presenting fundamentally novel theoretical insights.\n\n[1] Wightman, R., Touvron, H., & Jégou, H. (2021). Resnet strikes back: An improved training procedure in timm. arXiv preprint arXiv:2110.00476.\n\n[2] Andriushchenko, M., Croce, F., Müller, M., Hein, M., & Flammarion, N. (2023). A modern look at the relationship between sharpness and generalization. arXiv preprint arXiv:2302.07011.", "was_revised": true, "final_justification": "See the comment \"answer to rebuttal\".", "rebuttal": "We sincerely thank the reviewer for their constructive feedback and questions. Below, we address the points raised in the review.\n\n**Q1 and W3 (theoretical insight and empirical evidence):**  \nWe would like to emphasize that, to the best of our knowledge, this is the first work to introduce asymmetry as a novel and relevant concept in the design of activation functions. This contribution is not merely an empirical observation. We provide a theoretical analysis that directly links rightward asymmetry to variance reduction in latent representations. Specifically, we show that the reduced slope around the origin, a result of the right-skewed asymmetry of the underlying Gumbel distribution, acts as a form of implicit regularization, which is desirable in deep learning.\n\nBeyond the consistently strong performance of GoLU across a wide range of tasks, including vision, language modeling, and generative modeling, Appendix B provides additional empirical evidence highlighting the advantages of rightward asymmetry. We applied our insights to Mish, which is based on a left-skewed distribution, and flipped its underlying distribution to construct Flipped Mish (FMish) with right-skewed asymmetry.\n\nDespite being a minimal modification, FMish significantly outperformed the original Mish, and also surpassed symmetric activations like Swish and GELU when evaluated on ResNet18, ResNet50, and ViT-B/32 trained on ImageNet-1k, though GoLU remained the top-performing activation. This trend aligns closely with our theoretical analysis regarding slope and variance control:\n- Mish: slope = 0.6 at the origin\n- Swish / GELU: slope = 0.5 at the origin\n- FMish: slope = 0.4 at the origin\n- GoLU: slope ≈ 0.37 at the origin\n\n**Q2 and W1 (Transfer gains with advanced training procedures):**  \nWhile we understand the reviewer’s concern that some of the performance gains may appear small, we respectfully emphasize that even fractional improvements in accuracy can lead to substantial impact when amortized across many different deep learning applications. Activation functions like GELU and Swish also showed small improvements over ReLU, but they are now widely adopted, improving results in thousands of code bases. We specifically emphasize that, unlike many activations that show task or architecture-specific benefits, GoLU provides more consistent improvements across domains, including vision, language modeling, and generative modeling, as demonstrated in our experiments.\n\nWe would also like to emphasize a trend we observed in our Table 2 that the difference in top-1 accuracy between GoLU and GELU on ImageNet increases for larger models:\n- RN18: +0.10\n- RN34: +0.27\n- RN50: +0.56\n- WRN50-2: +0.65\n\nRegarding the concern about baseline performance and training pipelines, we fully agree with the reviewer that modern training recipes can significantly improve baseline results. However, our goal in this paper is not to surpass SOTA, but rather to demonstrate that GoLU provides consistent benefits when substituted into established pipelines, without further tuning. For this reason, we standardized our training setup across activations (focusing primarily on original implementations when possible) to ensure a fair and controlled comparison.\n\nThat said, based on the reviewer’s suggestion, we trained ResNet-50 on ImageNet-1k using the A3 training recipe from [1]. We used the timm library and followed the hyperparameters listed in Table 2 of [1]. With three random seeds (0,1,2), the ReLU model achieved a mean top-1 validation accuracy of $\\bf{78.17  \\pm 0.039}$, consistent with [1]. We then simply retrained the model with GoLU, keeping everything else identical, and observed a mean top-1 validation accuracy of $\\bf{79.15 \\pm 0.024}$, which is a highly significant improvement over the default ReLU activation. This improvement is also consistently observed across the entire training curve.\n\nRegarding the question on fine-tuning, while fine-tuning a ReLU-pretrained model using GoLU may yield improvements, we have not verified this, and this is not a claim made in our paper. It is conceivable that the weights of a model pretrained with ReLU are already aligned with that activation, and simply replacing it during fine-tuning may not reveal the full strength or effect of GoLU. Instead, we expect that future foundation models pretrained with GoLU will show improved performance, which will in turn benefit the resulting fine-tuned variants.\n\n[1] Wightman, R., Touvron, H., & Jégou, H. (2021). Resnet strikes back: An improved training procedure in timm. arXiv preprint arXiv:2110.00476.\n\n**W2 (discussion on smooth loss landscape):**   \nAs noted in the manuscript, the smoother loss landscape observed with GoLU is an empirical observation (rather than a formal argument). While it aligns with properties like GoLU’s smaller slope and variance-reducing effect, it is meant to complement the main experimental results.\n\nWe agree that loss landscape visualizations must be interpreted cautiously. However, we would like to clarify that Figure 5 simply illustrates a smoother loss landscape along a 2D subspace spanned by random directions, in order to provide qualitative insight. While not definitive, this is consistent with GoLU’s observed generalization gains. We will clarify this in the final version to avoid potential over-interpretation.\n\nWe hope to have addressed the concerns raised by the reviewer. We respectfully ask that if you feel more positively about our paper, you kindly reconsider your rating accordingly. If not, please let us know if you have further questions or what can be further improved. We are happy to continue the discussion at any time before the end of the discussion period on August 6th. Thank you."}, {"review_id": "hptcjaNc8W", "reviewer": "Reviewer_bjVH", "summary": "This paper introduces a new self-gated activation function, Gompertz Linear Units (GoLU), using the Gompertz function as the gate function. It demonstrates the variance reduction and smoothing in loss landscape properties of GoLU, and its consistent effectiveness on wide applications.", "questions": "1. Can the authors provide explicit notations of the functions in the captions of Figure 1 and Figure 2? Now, the textual description could be confusing and misleading (gate vs. gated functions). \n2. Why only select three random images from ImageNet-1K for Figure 4? How about on a larger set of images or the whole ImageNet-1K? Similarly, it is better to include statistics at a larger scale for Table 1, e.g., more and larger images.\n3. Is the variance reduction and smoothing effect consistent in language modeling? More results will be favorable and will provide stronger evidence.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The paper provides insightful characterizations of the proposed GoLU, including its analytical form, the property to reduce variance based on curvature, and improvement on the smoothness of the loss landscape. \n2. GoLU consistently outperforms other commonly used activation functions on wide applications (with acceptable exceptions), providing a new avenue for broader research.\n3. Anonymized code facilitates reproducibility.\n\nWeaknesses:\n1. See my questions below.", "was_revised": true, "final_justification": "The authors have provided valid responses to my questions and additional results that further support the effectiveness of their methods.\n\nGiven the solid analysis on the properties, valuable insights and promising results for future application shown in the paper, I would keep my borderline accept rating unchanged.", "rebuttal": "We sincerely thank the reviewer for their constructive feedback and questions. Below, we address the questions raised in the review.\n\n1. We will update the captions of Figures 1 and 2 in the revised version of our paper to more clearly distinguish between the gated activations, their corresponding gate functions, and the distributions underlying those gate functions.\n2. The three images in Figure 4 are randomly selected and are meant to visually showcase the variance reduction effect and support the theoretical argument. Including more samples would show qualitatively the same behavior and would therefore add limited value. We would also like to note that we observe such variance reduction effects across all intermediate layers of ResNet-50 trained on ImageNet-1k, not just in the final activation. We will comment on this in the revised version of the paper.\n3. Motivated by the reviewer’s question, we selected babyGPT trained on TinyStories and conducted the same activation distribution analysis as in Figure 4. We observed that GoLU results in the lowest variance in the final layer’s activations: GoLU (Var: 0.042), GELU (Var: 0.066), ReLU (Var: 0.079), Swish (Var: 0.155).\nWe further performed the same loss landscape analysis as in Figures 5 and 9, and again observed the same qualitative behavior, with GoLU exhibiting both the lowest mean loss and the lowest variance across the landscape.\n| Activation | Mean Loss     | Loss Variance     |\n|------------|-----------|--------------|\n| Swish      | 1.6301    | 3.94e-05     |\n| ReLU       | 1.6259    | 3.93e-05     |\n| GELU       | 1.6133    | 3.93e-05     |\n| GoLU       | **1.6079**    | **3.91e-05**     |\n\nWe hope to have addressed the concerns raised by the reviewer. We respectfully ask that if you feel more positively about our paper, you kindly reconsider your rating accordingly. If not, please let us know if you have further questions or what can be further improved. We are happy to continue the discussion at any time before the end of the discussion period on August 6th. Thank you."}, {"review_id": "S0a0XGpLdr", "reviewer": "Reviewer_8T3r", "summary": "The document introduces the Gompertz Linear Unit (GoLU), a novel activation function for deep learning that leverages the Gompertz function as its gating mechanism. Its asymmetry, derived from the Gumbel distribution, enables reduced activation variance and preserves robust gradient flow. The experiments show that this activation function can help enhance model robustness and generalization.", "questions": "I don't have question towards the current shape of the paper but I hope the authors can provide more insights on future development.", "limitations": "yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Quality: The paper conducts extensive experiments on a lot of tasks, including image classification, language modeling, video segmentation and even image generation (diffusion models), showing substantial efforts in demonstrating the effectiveness of the proposed algorithm. The proposed activation function seems to be able to outperform existing activation functions by a consistent margin across different tasks. \n\nClarity: The writing of this paper is clear (partially due to the simplicity of this paper), easy to follow with a substantial level of details. It would be quite straight-forward to replicate the authors' experiments. \n\nSignificance: the experiment results look very promising as the advantages are clear and the margin are significant and consistent. It would be great if the authors can provide more insights, e.g. is there any good principle that can help further improve performance?", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their positive and constructive feedback. Below, we provide our response.\n\nOne of the insights presented in our work is the role of rightward asymmetry stemming from the underlying Gumbel distribution, which contributes to variance reduction and is associated with a smoother loss landscape. This mechanism appears to be an important (though not the only) factor behind the consistent improvements observed across a wide range of tasks.\n\nTo further investigate this property, in Appendix B, we applied this insight to Mish, which is based on a left-skewed distribution, and flipped its underlying distribution to construct Flipped Mish (FMish) with right-skewed asymmetry. Despite being a minimal modification, FMish significantly outperformed the original Mish, and also surpassed symmetric activations like Swish and GELU, though GoLU remained the top-performing activation. These results are consistent with our expectations based on the slope of each activation function: FMish has a slope of 0.4 at the origin, which is lower than that of Mish (0.6) and Swish/GELU (both 0.5), but slightly higher than GoLU (0.37). \n\nBased on our findings, the slope near the origin appears to be a meaningful control parameter that could serve as a guiding principle for designing future activation functions (as done for FMish), whether through theoretical analysis, automated methods, or by motivating parameterized activations (please also refer to the table on Asymmetry Ablation in response to reviewer AMoC).\n\nWe hope these insights help illustrate the broader potential of our contribution. If there are any further questions or points that could help strengthen your view of the paper, we would be happy to continue the discussion at any time before the end of the discussion period on August 6th. Thank you."}, {"review_id": "BXNDNmE17c", "reviewer": "Reviewer_AMoC", "summary": "The paper proposes the Gompertz Linear Unit (GoLU), a smooth, self-gated activation function defined as GoLU(x) = x · e^{-e^{-x}}, where the gate is the Gompertz CDF of a standard Gumbel distribution. This form introduces a right-skewed asymmetry that reduces the slope near zero, encouraging lower activation variance while ensuring stable gradient flow. The authors analytically link this behavior to implicit regularization and empirically show that GoLU produces tighter latent representations and more expressive weights, resulting in smoother loss landscapes. Across a wide range of vision, language, and translation benchmarks, GoLU consistently outperforms existing activations like ReLU, GELU, Swish, and Mish, all while maintaining efficiency through a custom CUDA kernel.", "questions": "- **Asymmetry Ablation** – Could you run an ablation to isolate the role of GoLU’s asymmetry? For example, (a) evaluate a symmetric version such as ½ [GoLU(x) + GoLU(−x)], and (b) rescale GoLU to match GELU’s slope near zero. This would help clarify whether the asymmetry is essential for the observed gains.\n- **Interaction with Optimizers** – Since modern optimizers like Adam and Lion are designed to stabilize gradients, how does GoLU interact with them? It seems that the current experiments are conducted with momentum-based optimizers. I’m wondering whether GoLU’s effect—such as producing a smoother loss landscape—would become more prominent with vanilla SGD or be overshadowed by specific optimizer dynamics.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- The proposed Gompertz Linear Unit (GoLU) introduces a novel activation function with a self-gating mechanism based on the Gompertz CDF, offering a fresh addition to the family of smooth activations.\n- The paper provides solid theoretical support, showing that GoLU’s reduced slope near zero leads to implicit regularization and smoother loss landscapes. The analytical derivations are thorough and well-motivated.\n- The authors conduct extensive empirical validation across diverse domains—including vision, language modeling, and machine translation—demonstrating consistent improvements over standard activations such as ReLU, GELU, Swish, and Mish.\n- The paper is clearly written and well-structured.\n\n**Weaknesses**\n\n- While performance improvements are consistent across tasks, they are generally small in magnitude. This limited headroom may reduce the practical impact and adoption of GoLU, especially given the added complexity of a new activation function.", "was_revised": true, "final_justification": "The authors' rebuttal successfully addressed my concerns. I have raised my score from 4 to 5.", "rebuttal": "We sincerely thank the reviewer for their constructive feedback and questions. Below, we address the points raised in the review.\n\n**Weakness:**  \nWhile we understand the reviewer’s point that performance improvements may appear small in some cases, we respectfully argue that even fractional improvements in accuracy can lead to substantial impact especially when amortized across many different deep learning applications. Activation functions like GELU and Swish also showed small improvements over ReLU, but they are now widely adopted, improving results in thousands of code bases. We specifically emphasize that, unlike many activations that show task or architecture-specific benefits, GoLU provides more consistent improvements across domains, including vision, language modeling, and generative modeling, as demonstrated in our experiments.\n\nWe would also like to emphasize a trend we observed in our Table 2 that the difference in top-1 accuracy between GoLU and GELU on ImageNet increases for larger models:\n\n- RN18: +0.10\n- RN34: +0.27\n- RN50: +0.56\n- WRN50-2: +0.65\n\nIn addition to its empirical performance, GoLU has a simple, closed-form definition and is theoretically well-motivated. Our provided CUDA kernel enables easy integration with no added latency, making GoLU a practical activation function that can be widely adopted by the community.\n\n**Question on Asymmetry Ablation:**  \nWe appreciate the reviewer’s suggestion to ablate the role of asymmetry. The specific example, ½ [GoLU(x) + GoLU(−x)] will symmetrize the full activation which is not desirable, however one can symmetrize the underlying distribution which will lead to ½ x [1+Gompertz(x) -Gompertz(−x)] (the derivation follows similar lines to our eq.19). This does not enjoy the properties of GoLU and is expected to lead to lower performance.\n\nInstead, in Appendix B, we have followed a relatively similar idea, but have taken advantage of the insights gained in this work to explore whether we can *improve* an existing activation. Specifically, we took Mish, which is based on a left-leaning distribution, and flipped its distribution to obtain an activation with right-leaning asymmetry, which we refer to as Flipped Mish (FMish). We observed that this significantly improves performance compared to the original Mish activation and, surprisingly, makes FMish outperform symmetric activations like Swish and GELU, while it still does not surpass GoLU. This result aligns closely with what their slopes at the origin suggest: the slope of FMish (0.4) is lower than Mish (0.6) and Swish/GELU (both 0.5), but still slightly higher than that of GoLU (0.37).\n\nAdditionally, following the reviewer’s suggestion, we also conducted a more comprehensive ablation on the slope of GoLU. We introduced a parameter $\\beta$ which controls the slope $\\mathrm{GoLU}_{\\beta}(x) = x e^{-\\beta e^{-x}}$ and conducted an ablation study on this parameter for ResNet-34, ResNet-50, ViT-B/32, and ViT-B/16. The results, provided in the table below, indicate that for all these models, decreasing $\\beta$ below 1 (which increases the slope) degrades performance, aligning with our expectations. Moreover, for ResNet-34, increasing the $\\beta$ parameter to 1.5 or 2 (which decreases the slope) improves performance before degrading again at higher values. This shows that the default value ($\\beta=1$) provides a balanced variance control in most cases, although some architectures may benefit from further reducing the slope.\n\n| β     | ResNet34            | ResNet50            | ViT-B/32           | ViT-B/16           |\n|-------|---------------------|---------------------|--------------------|--------------------|\n| 0.10  | 69.86 ± 0.044       | 71.26 ± 0.167       | 70.87 ± 0.116      | 77.34 ± 0.014      |\n| 0.50  | 73.18 ± 0.030       | 75.84 ± 0.086       | 75.52 ± 0.150      | 80.41 ± 0.048      |\n| 1.00  | 73.71 ± 0.043       | **76.63 ± 0.036**       |**75.74 ± 0.108**      |**80.72 ± 0.052**      |\n| 1.50  | **73.86 ± 0.050**   | 76.50 ± 0.148       | 75.70 ± 0.018      | 80.40 ± 0.056      |\n| 2.00  | **73.86 ± 0.074**   | 76.43 ± 0.147       | 75.58 ± 0.011      | 80.33 ± 0.051      |\n| 5.00  | 72.92 ± 0.056       | 73.03 ± 0.153       | 75.21 ± 0.070      | 80.00 ± 0.086      |\n| 10.00 | 70.26 ± 0.173       | 66.36 ± 0.142       | 74.54 ± 0.088      | 79.56 ± 0.031      |\n\n**Question on Interaction with Optimizers:**  \nWhile we did use momentum-based optimizers in many of our vision experiments, we also evaluated GoLU with adaptive optimizers such as AdamW in multiple settings:\n- TinyViT, ViT-B/32 and ViT-B/16 on ImageNet-1k\n- GPT language models on TinyStories and OWT\n- Denoising Diffusion Probabilistic Model on CelebA\n\nIn all of these cases, GoLU outperformed baseline activations, which suggest that GoLU’s improved generalization complements adaptive optimizers as well.\n\nAdditionally, as an example of a model trained with AdamW, we selected babyGPT trained on TinyStories and conducted the same loss landscape analysis as in Figures 5 and 9. We observed the same qualitative behavior, with GoLU exhibiting both the lowest mean loss and the lowest variance across the landscape:\n\n| Activation | Mean Loss     | Loss Variance     |\n|------------|-----------|--------------|\n| Swish      | 1.6301    | 3.94e-05     |\n| ReLU       | 1.6259    | 3.93e-05     |\n| GELU       | 1.6133    | 3.93e-05     |\n| GoLU       | **1.6079**    | **3.91e-05**     |\n\nWe hope to have addressed the concerns raised by the reviewer. We respectfully ask that if you feel more positively about our paper, you kindly reconsider your rating accordingly. If not, please let us know if you have further questions or what can be further improved. We are happy to continue the discussion at any time before the end of the discussion period on August 6th. Thank you."}]

Author rebuttal:

Meta-review: The paper introduces a new activation function, Gompertz Linear Unit (GoLU): $x e^{-e^{-x}}$. The authors claim that GoLU as a self-gated activation, which is able to reduce variance in latent representations and promotes smoother gradient flows. 
The comprehensive experimental results on image classification, language modeling, semantic segmentation, object detection, support the effectiveness of GoLU.

Extensive experiments across multiple tasks—including image classification, language modeling, semantic segmentation, and object detection—demonstrate the effectiveness of GoLU.

During the review process, several reviewers raised concerns regarding the consistency of results, the extent of performance improvements, and the role of hyperparameter tuning. In response, the authors provided additional experiments in the rebuttal, which successfully addressed these concerns and convinced all reviewers.

The AC, while acknowledging the positive outcome, noted that most of the presented experiments are image-centric (e.g., classification, segmentation, detection). To further **strengthen the significance and impact** of the work, the AC suggested that the authors evaluate GoLU on large language models (e.g., a 1B-parameter model).
Decision: Accept (poster)