Title: Gompertz Linear Units: Leveraging Asymmetry for Enhanced Learning Dynamics
Venue: NeurIPS 2025
Accepted: yes
Keywords: Deep Learning

Activation Functions
Abstract: Activation functions are fundamental elements of deep learning architectures as they significantly influence training dynamics. ReLU, while widely used, is prone to the dying neuron problem, which has been mitigated by variants such as LeakyReLU, PReLU, and ELU that better handle negative neuron outputs. Recently, self-gated activations like GELU and Swish have emerged as state-of-the-art alternatives, leveraging their smoothness to ensure stable gradient flow and prevent neuron inactivity. In this work, we introduce the Gompertz Linear Unit (GoLU), a novel self-gated activation function defined as $\mathrm{GoLU}(x) = x \\, \mathrm{Gompertz}(x)$, where $\mathrm{Gompertz}(x) = e^{-e^{-x}}$. The GoLU activation leverages the right-skewed asymmetry in the Gompertz function to reduce variance in the latent space more effectively compared to GELU and Swish, while preserving robust gradient flow. Extensive experiments across diverse tasks, including Image Classification, Language Modeling, Semantic Segmentation, Object Detection, Instance Segmentation, and Diffusion, highlight GoLU's superior performance relative to state-of-the-art activation functions, establishing GoLU as a robust alternative to existing activation functions.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes the Gompertz Linear Unit (GoLU), a smooth, self-gated activation function defined as GoLU(x) = x · e^{-e^{-x}}, where the gate is the Gompertz CDF of a standard Gumbel distribution. This form introduces a right-skewed asymmetry that reduces the slope near zero, encouraging lower activation variance while ensuring stable gradient flow. The authors analytically link this behavior to implicit regularization and empirically show that GoLU produces tighter latent representations and more expressive weights, resulting in smoother loss landscapes. Across a wide range of vision, language, and translation benchmarks, GoLU consistently outperforms existing activations like ReLU, GELU, Swish, and Mish, all while maintaining efficiency through a custom CUDA kernel.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The proposed Gompertz Linear Unit (GoLU) introduces a novel activation function with a self-gating mechanism based on the Gompertz CDF, offering a fresh addition to the family of smooth activations.\n- The paper provides solid theoretical support, showing that GoLU’s reduced slope near zero leads to implicit regularization and smoother loss landscapes. The analytical derivations are thorough and well-motivated.\n- The authors conduct extensive empirical validation across diverse domains—including vision, language modeling, and machine translation—demonstrating consistent improvements over standard activations such as ReLU, GELU, Swish, and Mish.\n- The paper is clearly written and well-structured.\n\n**Weaknesses**\n\n- While performance improvements are consistent across tasks, they are generally small in magnitude. This limited headroom may reduce the practical impact and adoption of GoLU, especially given the added complexity of a new activation function.\n\nQUESTIONS:\n- **Asymmetry Ablation** – Could you run an ablation to isolate the role of GoLU’s asymmetry? For example, (a) evaluate a symmetric version such as ½ [GoLU(x) + GoLU(−x)], and (b) rescale GoLU to match GELU’s slope near zero. This would help clarify whether the asymmetry is essential for the observed gains.\n- **Interaction with Optimizers** – Since modern optimizers like Adam and Lion are designed to stabilize gradients, how does GoLU interact with them? It seems that the current experiments are conducted with momentum-based optimizers. I’m wondering whether GoLU’s effect—such as producing a smoother loss landscape—would become more prominent with vanilla SGD or be overshadowed by specific optimizer dynamics.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
The document introduces the Gompertz Linear Unit (GoLU), a novel activation function for deep learning that leverages the Gompertz function as its gating mechanism. Its asymmetry, derived from the Gumbel distribution, enables reduced activation variance and preserves robust gradient flow. The experiments show that this activation function can help enhance model robustness and generalization.

STRENGTHS AND WEAKNESSES:
Quality: The paper conducts extensive experiments on a lot of tasks, including image classification, language modeling, video segmentation and even image generation (diffusion models), showing substantial efforts in demonstrating the effectiveness of the proposed algorithm. The proposed activation function seems to be able to outperform existing activation functions by a consistent margin across different tasks. 

Clarity: The writing of this paper is clear (partially due to the simplicity of this paper), easy to follow with a substantial level of details. It would be quite straight-forward to replicate the authors' experiments. 

Significance: the experiment results look very promising as the advantages are clear and the margin are significant and consistent. It would be great if the authors can provide more insights, e.g. is there any good principle that can help further improve performance?

QUESTIONS:
I don't have question towards the current shape of the paper but I hope the authors can provide more insights on future development.

LIMITATIONS:
yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

comments: SUMMARY:
This paper introduces a new self-gated activation function, Gompertz Linear Units (GoLU), using the Gompertz function as the gate function. It demonstrates the variance reduction and smoothing in loss landscape properties of GoLU, and its consistent effectiveness on wide applications.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The paper provides insightful characterizations of the proposed GoLU, including its analytical form, the property to reduce variance based on curvature, and improvement on the smoothness of the loss landscape. 
2. GoLU consistently outperforms other commonly used activation functions on wide applications (with acceptable exceptions), providing a new avenue for broader research.
3. Anonymized code facilitates reproducibility.

Weaknesses:
1. See my questions below.

QUESTIONS:
1. Can the authors provide explicit notations of the functions in the captions of Figure 1 and Figure 2? Now, the textual description could be confusing and misleading (gate vs. gated functions). 
2. Why only select three random images from ImageNet-1K for Figure 4? How about on a larger set of images or the whole ImageNet-1K? Similarly, it is better to include statistics at a larger scale for Table 1, e.g., more and larger images.
3. Is the variance reduction and smoothing effect consistent in language modeling? More results will be favorable and will provide stronger evidence.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel activation function, Gompertz Linear Unit (GoLU). The authors present GoLU as a self-gated activation that reduces variance in latent representations and promotes smoother gradient flows.\nThe experimental results show improved performance across various tasks such as image classification, language modeling, semantic segmentation, object detection, etc.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- S1 : Well written paper that is easy to follow. The introduction of GoLU is clear and straightforward to understand.\n\n- S2 : The amount of experiments is commendable, with a comprehensive empirical evaluation on diverse benchmark datasets. As this paper contribution is empirical, this is a good points (however see W1).\n\n- S3 : The code is available and a Cuda kernel was developed. This allows for the community a seamless integration and use of GoLU.\n\n---\n\n**Weaknesses:**\n\n- W1 : The improvements in performance, while seamingly widespread, appear (very) modest in magnitude, raising questions about their real-world impact. Furthermore, as different well-known training procedures would lead to better baselines (with even better results than those reported for GoLU), I am not convinced by the usefulness of this new activation function. I give a couple of examples:\n\n    a) Table 2 and 3 baselines are a bit questionable.\n\n    - for table 2, the results and training procedure provided by [1] with even the cheapest procedure (A.3) lead for a ResNet50 on Imagenet to 78.1 % while the paper reports 75.44% for ReLU and 76.63% for GoLU.\n\n    - for table 3, as public implementations, for example [this git](https://github.com/akamaster/pytorch_resnet_cifar10) can lead to much better results with ReLU. I am a bit worried about the reported gains, see the table below:\n| Name       | Test err (original impl.) | Test err (public git impl.) | Test err (reported for ReLU) | Test err (reported for GoLU) |\n|------------|------------------|------------------------|------------------|------------------|\n| ResNet20   | 8.75            | 8.27                  | 8.59        | 8.23        |\n| ResNet32   | 7.51            | 7.37                  | 7.79        | 7.31        |\n| ResNet44   | 7.17            | 6.90                  | 7.42        | 7.15        |\n| ResNet56   | 6.97            | 6.61                  | 7.20        | 6.85        |\n| ResNet110  | 6.43            | 6.32                  | 6.79       | 6.75        |\n\n   b) The Perplexity gains reported in Table 4 are in my opinion, not significant enough to be convincing and one could wonder whether when applied on larger language models, GoLU can lead to better results.\n\n- W2 : The discussion on smooth/flat landscape is not an argument.\nFigure 5, meant to demonstrate smoother loss landscapes, lacks clarity and fails to convincingly support the authors' claims, regarding that point (projecting a ~250k dimensional space onto a 3 dimensional one is not very interpretable). Further, as discussed in [2], these metrics are to be handled with extra attention and not just given as arguments.\n\n- W3 : The theoretical contribution is limited, as the paper primarily offers incremental enhancements over existing methods based on an established mathematical function, rather than presenting fundamentally novel theoretical insights.\n\n[1] Wightman, R., Touvron, H., & Jégou, H. (2021). Resnet strikes back: An improved training procedure in timm. arXiv preprint arXiv:2110.00476.\n\n[2] Andriushchenko, M., Croce, F., Müller, M., Hein, M., & Flammarion, N. (2023). A modern look at the relationship between sharpness and generalization. arXiv preprint arXiv:2302.07011.\n\nQUESTIONS:\nQ1. Could the authors offer deeper theoretical analysis or empirical evidence to rigorously justify the claimed advantages of rightward asymmetry in GoLU?\n\nQ2. Do the performance gains transfer with advanced training procedures as in [1] mentioned in W1? Further, can GoLU seamlessly replace ReLU in pretrained models and lead to better performance in a fine-tuning setting on a downstream task?\n\nIf Q2. is answered I could raise my score, because as the paper's contribution relies on the experimental part, I would like to see a significant gain before granting acceptance.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
The authors propose Gompertz Linear Units (GoLU), a novel activation function based on the Gompertz function. The paper formally defines GoLU and analyzes its key properties. In particular, the authors discuss its role during training, highlighting that it induces a smoother loss landscape and a more spread-out weight distribution, they are considered better compared to previous activation functions. Experimental results show that the proposed method achieves better performance across a wide range of tasks, including but not limited to image classification, object detection, image generation, and language modeling.

STRENGTHS AND WEAKNESSES:
Pros
1.	This paper is well-written.
2.	Significant improvements have been achieved in many commonly used benchmarks.
3.	In addition to strong baseline performance, the paper includes numerous visual comparisons (including those in the appendix) that help illustrate the advantages of the proposed method.
4.	Anonymous code repo is provided, supporting reproducibility.
Cons
1.	There could be more visualization results, such as Table 1, and perhaps an ImageNet one, which would be more convincing.
2.	The proposed method shows impressive performance improvements. However, the main text lacks theoretical explanations that help readers understand why the method works. Also, I noticed that the appendix contains some important analysis, which would strengthen the paper if included in the main body

QUESTIONS:
It should benefit the readability and clarity of the paper if some of the theoretical parts in the appendix are included in the main body.

LIMITATIONS:
NA
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03