Polynomial, trigonometric, and tropical activations
Authors: Ismail Khalfaoui-Hassani
Stefan Kesselheim
Venue: ICLR 2026 Poster (2026)
Keywords: Orthogonal function bases
Tropical polynomials
Polynomial mapping
Deep neural networks
ImageNet-1K
OpenWebText
Transformers
GPT2
Convolutional networks
ConvNeXt
Initialization scheme
PyTorch
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: New activation functions based on orthonormal bases (like Hermite and Fourier) and Tropical polynomials can train deep models effectively, avoid gradient issues, and approximate standard activations.

Abstract: Which functions can be used as activations in deep neural networks? This article explores families of functions based on orthonormal bases, including the Hermite polynomial basis and the Fourier trigonometric basis, as well as a basis resulting from the tropicalization of a polynomial basis. Our study shows that, through a simple variance-preserving initialization and without additional clamping mechanisms, these activations can successfully be used to train deep models, such as GPT-2 for next-token prediction on OpenWebText and ConvNeXt for image classification on ImageNet. Our work addresses the issue of exploding and vanishing activations and gradients, particularly prevalent with polynomial activations, and opens the door for improving the efficiency of large-scale learning tasks. Furthermore, our approach provides insight into the structure of neural networks, revealing that networks with polynomial activations can be interpreted as multivariate polynomial mappings. Finally, using Hermite interpolation, we show that our activations can closely approximate classical ones in pre-trained models by matching both the function and its derivative, making them especially useful for fine-tuning tasks. These activations are available in the torchortho library.

Published as a conference paper at ICLR 2026
POLYNOMIAL, TRIGONOMETRIC, AND TROPICAL ACTIVATIONS
Ismail ShalfaouuiHHssseen & Stefan Kesselheim
Jülich Supercomputing Centre
Forschungszentrum Jülich
Jülich, Germany
(i. 11.khal faoui, wwe.ekesseelle...... (ttchheeinmeeaefeeee feesselhhhimeff-uuuul
ABSTRACT
Which functions be used activations in deep neural networks? This article explores families of functions based on orthonormal bases, including the Hermite polynomial basis and the Fourier trigonometric basis, as well basis resulting from the tropicalization of a polynomial basis. Our study shows that, through simple crrtance-preserving initialization and without additional clamping mechanisms, these activations can successfully be used to train deep models, such as GPT-2 for next-token prediction on OpenWebText WebText and ConvNeXt for image classification on ImageNet. Our work addresses the issue of exploding and vanishing activations and gradients, particularly prevalent with polynomial activations, and opens the door for improving the efficiency of large-scale learning tasks. Furthermore, approach provides insight into the structure of neural networks, revealing that networks with polynomial activations mappings. Finally, using Hermite interpolation, we show that our activations can closely approximate classical ones in pre-trained models by matching both the function and its derivative, making them especially useful for fine-tuning tasks. These activations are available in the torchortho library.
be interpreted multivariate polynomial
INTRODUCTION
Modern deep learning is largely built upon the Multi-Layer Perceptron (MLP) McCulloch Pitts (1943); Rooenblatt] (1958) and the gradient backpropagation algorithm Rumelhart et al. (1986). The MLP can be described as combination of a multiplication by a matrix of learnable weights and the application of nonlinear activation function. Gradient hackpropagation, on the other hand, relies on the chain rule to compute partial derivatives necessary for optimizing weights through gradient descent. In a deep neural network, preserving variance across layers critical ensure stable training dynamics. Glorot & Bengio (2010) and He et al. (2015) were the first to consider a variance- preservvng analysis for deep neural networks.
The analysis shown in He et al. (2015) could be stated as the output signal of each MLP block should have the same variance as the input signal. And since learning performed with backpropagation. this same rule should apply to the gradients as well, meaning that the variance of the gradient of the input should also be equal the variance of the gradient of the output of the MLP.
In this manner, He et al.) (2015) demonstrated the methodology for initializing the weights of a deep neural network, thereby attaining performance on ImageNet classification that exceeds that of humans. This process entails the calculation of the ratio between the variance pre- and post-activation, called forward gain, as well as the ratio variance with respect to the derivative of the activation, called backward gain. Remarkably, for the ReLU function, both forward and backward gains are equal 2.
Recently, Yang & Wang (2025) employed the same principle to train learnable rational activations. However, they encountered challenge: the second-order moment has no closed formulation in the case of rational fractions. The authors' solution for ensuring the convergence of such rational activation networks consisted in initializing them by fitting the polynomial coefficients a classical

---

Published as a conference paper at ICLR 2026
activation such as ReLU or SiLU Ramachandran et al. (2018): Elfwing al. (2018). Here, propose a solution to the aforementioned problem by employing orthogonal basis functions (Fig. E. specifically polynomial and trigonometric functions. Orthogonal basis functions in chosen L space, as will be elucidated in the subsequent sections, facilitate the calculation of the second-order moment integral, thereby yielding a closed and straightforward formula. Additionally, we demonstrate that rational functions are unnecessary, asserting that polynomial activation functions are sufficient.
More generally, the convergence of polynomial networks shown in this work proves that deep neural networks can be seen as multivariate polynomial mappings. Indeed, the successive layers of a feed-forward network activated by a polynomial activation can be seen a composition of weighted sums of multivariate polynomials, ultimately resulting in a polynomial mapping. A parallel representation was made by Zhang et al. (2018) for ReL U-attivviedeed networks, demonstrating that they are tropical rational mappings. In a later section, we also explore tropical polynomial functions as activation functions. We demonstrate that these can be interpreted as the discrete convex conjugate of learnable function, thus encoding the convex hull of its epigraph (the set of points lying on or above the function's 's graph). The contributions of this paper span theoretical proofs, technical developments, and empirical confirmations, and can be summarized in the following list:
A novel pariance-preserving initialization method introduced for orthogonal learnable activations in neural networks. Assuming an orthonormal function basis, this method ensures that the output variances are unitary and match those of the derivative, leading to stable training-
Empirically showing that deep neural networks like ConvNeXt (Liu et al. (022) and GPT-2 (Radford et al. (209) can be trained using orthogonal learnable activations for tasks like image classification on ImageNetlk (Deng et al. (209) and language modeling on WebnWteTeext
(Gokaslan & Cohen 2019). The innovation eliminates the need for additional mechanisms (c.g.. ReLU, SoftSign...) to maintain training stability.
Proving in Appendix that polynomially activated neural networks are polynomial mappings. Developing Hermite, Fourier, and Tropical activations, addressing finite-precisie floating- point issues, and designing efficient parallel algorithms and kernels for their implementation.
2
RELATED WORK
The use of polynomial activations has long been denigrated, probably by the rise of works such as Pinkus (1999) and Leshno al. (1993) which have mathematically demonstrated that the universal approximation property is equivalent to the use of a non ponyooollomial activation function. The Universal Approximation Theorem Cybenko (1989); Hornik et al. (1990) holds for neural networks of arbitrary width and bounded depth. However, recent work such as Kidger & Lyons (2020); Gao jet al. (2025) show that in the framework of bounded width and arbitrary depth, every nonaifine continuous function is possible to use in practice, including polynomial activation functions. We show empirically in this work that polynomial activations can converge in the context of large-scale deep networks and datasets, provided coefficients learnable, and initialization is suitable. The empirical demonstration of the effectiveness of polynomial activations made here was achieved without the use of other functions intended regularize convergence, such as the SoftSign function borrowed from Turian et al. (2009) and used in Lokhande et al. (2020) for Hermite activations, or a ReLU function, or any normalization, as recently done in Zhuo et al. (025). This confirmation that polynomial activations are practicable opens the way to representing deep neural networks as multivariate polynomial mappings. As in Kileel et al. (2019) and Kubjas al. (2024). which see that these types of networks have greater expressive potential, we show in Appendix that deep polynomially activated neural networks are indeed multivariate polynomial mappings.
The adoption of polynomial bases is further justified by recent theoretical advancements in neu- roalgebraic geometry, particularly through the notions of neuromanifolds, which are the image of polynomially activated neural network, and neurovarieties, which are defined as the closure of neuromanifolds in the Zariski topology Amari (1994): (Amari et al. (2001 2006): Calin (2020). A key advantage of polynomial activations is that they make neural networks identifiall: under appropriate assumptions, the network parameters are uniquely determined by the represented function, up to finite set of permutations.
2

---

Published as a conference paper at ICLR 2026
Recent results by Usevich al. (2025 provide a constructive proof for finite identifiability for MLPs with monomial activations: for a general choice of learnable parameters (i.e., outside a set of Lebesgue measure zero), the network admits finite number of con-eiuvvalent representations. The proof proceeds inductively, showing that if pairs of layers are identifiable, the whole network is identifiable. In particular, non-increasing-widee networks may require large and linearly increasing polynomial degrees, which motivates stable training for higher-degree polynomial activations, explored in our work. Similarly, (Shahverdi al. (2026) show that general polynomial activations in MLPs and CNNs are identifiable, and that singularities of their neuromanifolds correspond to sparse subnetworks. For MLPs, these singularities often coincide with critical points of the mean-squared error loss. Together, these results suggest that the algebraic structure of polynomials allows for a principled understanding of the loss landscape. This perspective is consistent with the findings of Ainsworth et al. (2023), who studied neural network basins and demonstrated that the weights one trained model can be permuted to align with those another. Such alignment effectively establishes identifiability up to permutation symmetries, making the parametrization finite-to-one.
shown to be identifiable, though they
Furthermore, quadratic forms can be interpreted as MLPs with 2egree-2 polynomial activations, which have recently been successfully employed deep architectures Fan et al. (2023); Chen et al. (0025).
The subject of learnable activations has seen a resurgence thanks to the popularity enjoyed by the KAN article Liu et al. (2025). In Appendix []] we'll digress for while to explain how these inspired by the Kolmogorov-Amoo theorem Kolmogoroy (1957). Further related work appears in Appendix K
METHODS
3.1
VARIANCE PRESERVING INITIALIZATION
The variance-preservingg principle He et al. (015)) mentioned in the introduction is expressed in the following. Consider an input vector x mutually independent and uniformly distributed. Preserving the variance in an MLP layer with learnalle weight tensor W of inner dimension C. and an activation function F amounts to:
(Top..., Lis...... .. ) RCi.. Cin E N*, where all x;
Var[æ] C.n Var[W F(r)]
If suppose that and W are independent and of finite variance, we have:
Var [x] = Cin.
(ar E F(x)²) + Var[F(x)] E (W]?)
(2)
Assumption 3.1. We initialize W such as IE [W] = 0.
Since always assume that W is initialized with zero mean, Eq. simplifies into:
Var = Cin Var[W] [F())*]
(3)
Thus, to calculate the variance of the weights, we should calculate the following ratios: Definition 3.2. The forward gain of the MLP layer is defined by:
Var [a][x] E [F(x))1-1
(4)
Similarly, and a backward manner, Definition 3.3. The backward gain is the gain of the derivative of the activation with respect to æ, which we denote by F", and is defined as:
a' = Var[a] E [F'(x)]]-
Since a deep neural network essentially a composition of MI.P layers, an appropriate initialization method must avoid reducing or amplifying the input signals He et al. (2015).
Assumption 3.4. From now on, we assume that both the input signal and its gradient Ax follow a distribution of mean 0 and variance 1.
3

---

Published as a conference paper at ICLR 2026
Therefore, calculating the gains and a' in an MLP (or equivalently, convolution layer) involves calculating only the inverse of the second-order moments of the activation functions and their derivatives. Interestingly, for the ReLU function, have a' 2. Hence, the scaling of the standard deviation of the weights W in He al. (0055) by factor /2/Cin- More details can be found in Appendix
Given an arbitrary activation, equality of forward and backward gains is not always achieved by default, as in ReLU. In the next section, we show the conditions for an activation function written in orthonormal coordinate system verify the forward-backward gain equality. To illustrate this point, we will calculate the second moment for Hermite and Fourier basis decompositions, given their compatibility with the normal and uniform distributions, respectively.
3.2 VARIANCE PRESERVING INITIALIZATION FOR THE HERMITE ACTIVATION FUNCTION
Definition 3.5. Vn E the probabilist Hermite polynomials can be defined as follows:
(x) (-1)"
(6)
dr
is called the degree of the Hermite polynomial, and we have the first terms:
Heo( Hei (I:) = Hez(z) = 12 Hez(æ) = 3x
Hermite polynomials constitute a suitable choice for calculating the moment of order when follows standard normal distribution (0, 1) as evidenced by the following property 3.6
Property 3.6. Vm, n E N?, we have:
Hem (r) Hen (x)e 4 dx 2
With 8am the Kronecker delta.
Definition 3.7. We define the Hermite activation F: RR - R with its learnable coefficients Vn E N. Vk E [0, nI. 0g IR as:
ak
+
F(x)
(ex
(8)
Theorem 3.8. Variance-preservinn coefficient initialization of Hermite activation. Let
Vk [1, a* and ag
n!
Then, using this initialization, the forward and backward gains become the same and are equal to:
= a'
(10)
Proof. The proof is provided in Appendix
Corollary 3.9. In the limit case n - +oo, the coefficient initialization in Theorem 3.8 could be divided by factor Ve, with 2.7182... ... in order have unitary forward and backward gains. vk E [1.n]]
a2
Ve and ag Ve
-eV1-2 n!
(11)
Remark 3.10. The choice of an orthonormal family of functions depends on the input's probability distribution. For normally distributed input, Hermite polynomials simplify the computation of second-order moments and related gains. For a uniform distribution over [- wr, trigonometric functions (Fourier series) are appropriate. the input follows Wigner semi-circle distribution (of measure vi - dzx), then the Chebyshev polynomials of the second kind are the suitable choice.
4

---

Published as a conference paper at ICLR 2026
3.3 VARIANCE PRESERVING INITIALIZATION FOR THE FOURIER ACTIVATION FUNCTION
The forward and backward gains for a Hermite activation have been calculated under the assumption that the input follows normal distribution, such that the initial coefficients provide equal gains. The subsequent analysis will establish the same result for a truncated Fourier series expansion of order n E N.
Assumption 3.11. The input is assumed now to follow uniform distribution the interval [-,, TJ, with и zs 3. 3.1415.... denoted as x ~ U (-T, T..
Definition 3.12. We consider the following Fourier activation F: IR ++ IR:
(ax cos(kar) + be sin(ka)) (kxr))
+ F(z) dg
(12)
where (aE)REN and (by)keN. real learnable coefficients. Theorem 3.13. Variance-preservingg coefficient initialization of Fourier activation. Let
vk E [1, d2 1, by and (gg
(13)
(nl)²
Then, using this initialization, the forward and backward gains become the same and are equal to:
a
(14)
(!)
Proof. The proof provided in Appendix
Corollary 3.14. In the limit case - +oc. in order have unitary forward and backward gains, the coefficient initialization Theorem 3.13 could be divided by a factor v [2(2), with I, (x) the modified Bessel function of the first kind of order and we have [2) = 2.2795.... Vk [1,n]
1
0g
(2)2)
by
V (T)(2)
and dg VI((2)
(n!)²
(15)
Remark 3.15. In both Definitions 3.7 and 3.12 the terms inside the sum are scaled by factor of kl, yielding exponential series. In practice, is possible to scale the terms using other converging series such as K2 with p > 1. We experimented with this last alternative and observed no statistically significant impact on loss convergence, though we did observe better stability for higher polynomial degrees in the exponential variant.
3.4 VARIANCE PRESERVING INITIALIZATION FOR THE TROPICAL ACTIVATION FUNCTION
Definition 3.16. The max-tropical semiring T is the semiring T = (R U (+oo), 0 8), with the operations, Vx, E R U {+
x + y = max y} and x 2 x+
(16)
Equivalently, we could define the min- min-tropical semiring by substituting the max operation + with min operation. By extension, define for all a N the tropical power of z raised to as multiplying x to itself a times:
(17)
Definition 3.17. The tropicalization of polynomial of degree n E N is defined F: R + R. with Vn E N, Vk [0, n|] ag E R are the polynomial learnable coefficients:
F(x)
al X σ
máx {ar + kzx k=0
(18)
K=o
With max {ax + kx} max(au, 0g + x. ..., On + nx). k=0

---

Published as a conference paper at ICLR 2026
Definition 3.18. Convex conjugate Fenchel))-Fenchel) Let x I, f R is the convex conjugate of f: - R if and only if:
(x) sup f(k))
(19)
kER
Theorem 3.19. Variance-preserving coefficient initialization of Tropical activation. Let
Wk E [0, ni]
ag =
(20)
Then, applying this initialization the limit case of + oc yields an equal unitary gain both forward and backward for the following "scaled" definition of the tropical activation:
V2
F(x)
max {ax + kx}
(21)
k=o
Proof. The proof is provided in Appendix E
The tropical polynomial activation can be viewed as a generalization of the ReLU activation. Further- more, can be interpreted discrete version of the convex conjugate of a function / whose values at the natural integers E N are f(k) effectively encoding the convex hull of the epigraph of
f. as illustrated in Figures
3.5 PRACTICAL IMPLEMENTATION
In what follows, outline the considerations we have taken to implement Hermite, Fourier, and Tropical polynomial activations efficiently in PyTorch.
Weight decay. An important aspect of training learnable activations is that their learnable coefficients should be trained without weight decay, as it could bias them toward zero.
Explicit Hermite formula. We can show by induction that the following definition is equivalent to the one in Eq.
Hea (x) !
(-1)
x= 2m
(22)
n!
ml(n 2m)) 2
m==
We can see that the formula 22 can be parallelized, and is, therefore, the core of the algorithm we have developed in native PyTorch to compute Hermite activations (see Algorithm 2}.
dedicated Hermite kernel. Along with the parallel implementation of the Hermite activation,
developed a dedicated kernel that leverages the derivation established in C.5 for the backward
pass exploiting the fact that the derivative of polynomial is polynomial of lower degree and the following recurrence formula in the forward pass to optimize performance and memory usage (see Algorithm
(x)+1 = x He,, (x) Hcn-1(z)
(23)
Alternative Fourier formula. The definition of Fourier activation given in Definition 3.12 is under the Sine-Cosine form. In practice, we use the following equivalent Amplitude-Phine Phase formulation (see Algorithm 5):
I x F(x) ag + V2 ag cos( far ox)
(24)
A!
as it is less expensive in terms of FLOP. The learnable parameters here are initialized as follows: vk E No fc = ki &b - and and initialized as in 3.14 In our implementation of Fourier activation, not only were the coefficients learnable, but also The frequencies, yielding to what known "cosine basis" Mallat (2099) rather than the Fourier series.
Initializing by fitting a classical activation Function. Using a family of orthonormal functions permits an easy calculation of the initialization gain without resorting the trick of fitting function to an activation whose gain is known or easy to calculate, as in Yang & Wang (2025) with Safe Padé activation Molina et al. (2020). However, in some cases, such as continuing tnnettnnig model that was pretrained with a classical activation, using one of the learnable activations presented here to fit a classical activation could still be relevant. By fitting mean performing Lagrange interpolation.
CoovvationMMoiaaaa

---

Published as a conference paper at ICLR 2026
This could be accomplished via direct method involving the inversion of a Vandermonde matrix (Lagrange, or Newton's 's methods), or by an iterated gradient descent method (Gauss-Jordan method).
Two precautions need be taken, however, when performing such interpolation. The first concers the maximum degree that should be considered order to the function on a given interval. Figure (left) shows how far Hermite activation of degree can be accurately fitted, while [ shows the extent to which a Hermite activation of degree 8 can be accurately fitted. The second precaution concerns the derivative of the activation with respect the derivative of the target function to be interpolated. A Lagrange interpolation of a function is not always sufficient to fit its k-th derivatives. If we want to interpolate a function and its derivative(s) simultaneously, we refer this as Hermite interpolation.
4
3
GELU
GELU
GELU deriv.
GELU deriv.
2
2
Hermite
Hermite
Hermite deriv.
1
Hermite deriv.
0
2
2
x-axis
T-axis
Figure 1: Fitting a GELU with a Hermite Activation of degree 3 (left) and of degree 8 (right).
In the case of the Fourier activation, we observe in Figure (left) that a Lagrange interpolation not sufficient and that higher-order frequencies occur in the derivative approximation. This phenomenon can be likened to aliasing and can be circumvented by performing a simple Hermite interpolation instead of Lagrange interpolation, shown in Figure [Berrut & Welscher (207)) examined the exact solutions to this last problem for trigonometric functions.
GELU
GELU
10
10
GELU deriv.
GELU deriv.
Fourier
Fourier
Fourier deriv.
5
Fourier deniv.
0
0
10
10
- 10
0
10
r-axis
Æ-axis
Figure 2: Lagrange interpolation (left) and Hermite interpolation (right) of a GELU with Fourier Activation of degree
The success in fitting classical activations with Padé approximants in Yang & Wang 2025) could be attributed the fact that Padé approximant by definition the rational function that coincides with a function to be interpolated to the highest possible order, thus naturally achieving a Hermite interpolation. A good fit of non-convex function by tropical polynomial activation impossible since tropical polynomials are convex by definition. Therefore, in Appendix H we show how rational tropical activations (an extension of tropical polynomials) could, principle, achieve this fitting.
EXPERIMENTS
4.1 PRELIMINARY IMAGE CLASSIFICATION RESULTS ON CIFAR10
We trained Conv-Net-T (Liu et al. 2022 on CIFAR- 10 Krizhevsky et al. 2009) for 300 epochs,
averaging results over 10 random seeds. The experimental setup and results for CIFAR10 classification can found in Appendix L The three proposed learnable activations consistently outperformed baseline activations on test metrics. Results are shown in Table 7 and Figures and

---

Published as a conference paper at ICLR 2026
4.2 DECISION BOUNDARIES ON NOISY CLASSIFICATION DATASETS
We compared the decision boundaries of four single-layer neural networks trained on a simple, noisy classification dataset, each using different activation function evaluate how the choice of activation function affects classification behavior and boundary smoothness. Details of the visualizations of decision boundaries on multiple noisy datasets are provided in Appendix M
(a) Hermite
(b) Fourier
(c) Tropical
(d) GELU
Figure 3: Decision boundaries for different activation functions
4.3 VISION TASK: CONVNEXT-T IMAGE CLASSIFICATION ON IMAGENETIK
We evaluated the ConvNeXt-T model on the ImageNetlk dataset Deng al. 2009) for single-class image classification. The baseline ConvNeXt-T model employed GELU as the activation function in its MLP blocks. To analyze the impact of our learnable activations, replaced GELU with Hermite polynomial, Fourier trigonometric, and Tropical polynomial activation functions under our proposed initialization scheme. Each model was trained under identical conditions with fixed random seeds to ensure reproducibility and comparability. The evaluation metrics included: training loss, Top-1 and Top-5 validation accuracy. Table I and Figures II 12] and 13 summarize our results. We reproduced all experiments using five different random seeds. For each trial, we report the mean standard deviation at a fixed epoch. The experimental setup followed the approach and hyperparameter configuration detailed in Liu et al. (2022).
Ablation Studies. Additionally, ablation studies were performed on this vision task establish the impact of the degree for the learnable activations (Table the impact of our proposed initial- ization scheme (Table ( and making the activation coefficients learnable was useful (Table 5). Higher degrees generally improved performance, with all proposed activations showing consistent improvements in Top-1 and Top-5 accuracy as the degree increased. Furthermore, making activation coefficients learnable consistently resulted in better performance across all activation functions. Initialization with the proposed method led to improvements, especially for Hermite activation, where derived initialization scheme outperformed GELU-basse initialization.
8

---

Published as a conference paper at ICLR 2026
Table Training and validation results of ConvNeXt-T (28M) model on ImageNet-Ik Ik classification.
Values are reported mean + standard deviation over 5 seeds. p-values (two-tailed Student's 's t-test assuming equal variances) are for Val Top-1 accuracy compared to GELU.
Act.
Deg.
Train Loss 4
Val (
Val Top-5(%) f
FLOP
FLOP/Act.
p-value (Top-1)
GELU
2.824 + 0.0051
82.06 0.072
95.92 +: 0.038
4.57G
12
Tropical
6
2.854 = 0.0080
82.17 + 0.063
95.95 + 0.072
4.62G
Id+1= 19
0,0345 (*)
Fourier
6
2.759 = 0.0167
81.64 = 0.153
95.47 = 0.049
4.83G
7 ==13
0.0005 (*e2)
Hermite
3
2.788 a 0.0072
82.22 # 0.064
95.97 # 0.045
4.58G
4d+ 13
0,0062 (**)
4.4 LANGUAGE TASK: GPT-2 (124M) NEXT TOKEN PREDICTION ON OPENWEBTEXT
For the language modeling task, we trained the GPT-2 model Radford al. 2019) on the Open Text dataset Gooaslan & Cohen (019) for next-token prediction. The baseline GPT-2 used GELU activation, and we compared against SiLU (Elfwing et al. 2018). Hermite, Fourier, and Tropical activations under our proposed initialization scheme. All models were trained with identical hyperpa- rameters and initialization seeds to ensure consistent and reproducible comparisons. The evaluation metrics included: training and test losses and perplexities (which are simply the exponential of the loss). Table 2 and Figures [4 and L3 summarize our results. We reproduced all experiments using five different seeds. For each trial, report the mean + standard deviation a fixed iteration. The experimental design followed the guidelines established in Radford et al. (2019) and the open source reproduction available at Karpathy (2022). We used a total batch size of 786, 432 of which context length of 1024 tokens for total of 210, 000 iterations.
Table 2: Training and validation results for next-token prediction using GPT-2 (124M) model with different activations. Values are reported as mean + standard deviation over different seeds. Perplexity is computed as exp(loss). p-values (two-tailed Student's 's t-test assuming equal variances) compare each activation's 's validation loss against GELU,
Act.
Deg.
Train PPL
Train Loss |
Val PPL +
Val Loss J
FLOP
p-value (Val Loss)
GELU
19.003 = 0.156
2.944 = 0.0082
19.319 + 0.076
2.961 = 0.0039
87.52G
SiLU
19.324 = 0,106
2,962 = 0.0055
19.664 # 0.088
2.0045
87,37G
0.0001 (***)
Tropical
6
18.840 = 0.107
2.936 = 0.0057
19.027 # 0.055
2.946
87.75G
0.0001 (***)
Fourier
6
18.761 + 0.071
2,930 = 0.0038
18.965 + 0.154
2,941 2.9418000086
88 68.69G
0.0014 (*4)
Hermite
3 18.678 0.093
2.926 0.0049
18.821 0.293
0.0175
87.56G
0.0067 (*9)
All experiments were conducted under fixed configurations to ensure that any observed differences were solely due to the choice of activation function, allowing for fair and reproducible comparison
4.5 FINETUNING EXPERIMENT ON CIFAR10
Using the insights from Sec. 3.5 we conducted fine-tuning experiment in a transfer learning setting. Specifically, we investigated whether initializing a learnable activation by fitting classical one, using Hermite interpolation, can improve performance when adapting a pretrained model to new dataset. This experiment complements our theoretical analysis by demonstrating how fitting classical activations can serve as an effective initialization strategy. The experimental procedure and results for activation finetuning are available in Appendix 0
PARAMETERS, MEMORY, FLOP COUNT, AND EXECUTION TIME
The proposed activation functions introduce negligible number of additional parameters. For example, Hermite activations of degree d = 3 add only 72 parameters to ConvNeXt- -iny (28M total), corresponding to 0.0002%, with similarly minimal overheads for Tropical and Fourier activations. Hermite activations leverage a recursive formulation (Alg. that reduces both FLOP and required
Thhe code reproduce the experiments available http://pihh.aaoooo/oHsssssaaannn

---

Published as a conference paper at ICLR 2026
memory (vRAM) complexity from (dd²) to Odd), requiring only simple arithmetic per term. Fourier and Tropical activations also scale linearly with degree (O(d), as illustrated in Figure 17 and Table 9 measured on CPU. On GPUs, smaller degrees benefit from vectorized computation, leading to
reduced runtime and near-constant (1) scaling for low degrees (Figure 18 Table 10).
We further evaluated average training times per epoch across varying MLP widths and depths (Table IT). The proposed activations can incur higher latency compared to GELU in deep networks, but are often faster in shallower ones. Slowdowns relative to GELU were analyzed across widths (Figure (9) and depths (Figure 20p. Slowdowns are largely independent of width but increase approximately linearly with depth, with Hermite activations showing the largest slope, followed by Fourier and Tropical. This suggests that the proposed activations are more suitable for shallow, wide MLPss This observation aligns with Appendix [F] where polynomially activated MLP of arbitrary depth is shown to be equivalent to high-degree single-layer multivariate polynomial.
6 DISCUSSION
The results presented in this paper demonstrate the potential of using learnable activation functions based on orthogonal function bases and tropical polynomials in large-scale neural network tasks. Our experiments on ImageNet- I and WebTex with deep models such as ConvNeXt and GPT-2 show for the first time that such activations can lead to improvements over traditional static functions like ReLU and GELU, both in terms of image classification and language modeling.
This challenges the deep learning, as demonstrated by prior work. Our approach provides empirical evidence that, with appropriate initialization, polynomial activations can indeed be competitive. One of the key takeaways from our findings is the effectiveness of our proposed variance- variance-preserving initialization scheme. The choice of orthogonal functions plays an essential role in achieving a closed-form expression for the second-order moment. Furthermore, the use of tropical polynomials, which are not orthogonal, introduces a FLOP-light alternative approach to polynomial activations.
notion that polynomial activations are inherently unsuitable for
While our approach shows promise, there are several avenues for future exploration. Extending the framework to other activation families, such as wavelets, is straightforward. Multiplying the Hermite activation presented in this work by the term exp (-2² /2) gives what is known as Hermitian wavelets Brackx et al.) (2008), and applying the same to the Fourier activation yields the Morlet wavelet Grossmann & Morlet (1984) (or Gabor wavelet Gabor (1946)). Wavelets retain good orthogonal properties with respect to the adequate scalar product, and the calculation of the second moment is slightly modified take account of the additional decaying exponential term. Using wavelet activations instead of polynomials could enhance variance stability by providing finite function support, with potential plouusuiibblity implications. By expressing a Fourier series in its complex form, a network with Fourier activation can be viewed as a complex-valued neural network, offering a framework for modeling neuronal synchronization through the phase and amplitude relationships of oscillatory brain activity. Extension to other non-orthogonal functions, such as rational functions, could be done, for example, by means Laplace transform the Fourier activation.
7 CONCLUSION
In this work, we introduced novel framework for integrating learnable activation functions based on orthogonal function bases and tropical polynomials into deep neural networks, addressing challenges like variance preservation and stable gradient flow. Extensive experiments with the ConvNeXt model on ImageNetlk and the GPT-2 model on Open WebText showed that learnable polynomial activations match or exceed traditional activation functions during aageescale training and fine-tuning on smaller tasks, demonstrating their practical viability and challenging conventional beliefs about polynomial activations in neural networks. Importantly, we adopt a broad notion of "polynomial", encompassing not only classical algebraic polynomials but also trigonometric expansions and tropical polynomials. Our results pave the way for representing deep neural networks as polynomial mappings whose hypothesis classes correspond to geometric objects akin to algebraic varieties, or, in the tropical setting, piccewise-lincd polyhedral complexes, with future work focused on exploring a careful relaxation of these last.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDD
In alphabetical order, we thank Emile de Bruyn, Jan Ebert, Jiangtao Wang, and Oleg Filatov for their helpful discussions and feedback this manuscript. We also thank David Guo and @iissak for helpful technical discussions. This work would not have been possible without financial and computational support. This research was supported by the German Federal Ministry for Economic Affairs and Climate Action through the project "NXT GEN AI METHODS" and by the Helmholtz AI Cooperation Unit (HAICU) through the Helmholtz Foundation Model Initiative part of the Synergy Unit. We gratefully acknowledge the Gauss Centre for Supercomputing e. (www.gaus, centre.eu) eu) for funding this project by providing computing time on the Supercomputers JUWELS and JURECA at Jülich Supercomputing Centre (JSC).
REFERENCES
Alireza Afzal Aghaei. fkan: Fractional kolmogorov-arnold networks with trainable jacobi basis functions. arXiv preprint arXiv:2406.07445 2024a.
Alireza Afzal Aghaei. rkan: Rational kolmogorov- networks. arXiv preprint arXiv:2406. 14495, 2024b.
Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. re-bbains, Merging models modulo permutation symmetries. In The Eleventh International Conference on Learning Representations, 2023. URL
net. forum? d=CQsmMYm PPT
Javad Alavi and Hossein Aminikhah. Orthogonal cubic splines for the numerical solution of nonlinear parabolic partial differential equations. MethodsX, 10: 02100, 2023.
Shun-ichi Amari. Information geometry and manifolds of neural networks. In From Statistical Physics Statistical Inference and Back, pp. 113-138. Springer, 1994.
Shun-ichi Amari, Hyeyoung Park, and Tomoko Ozeki. Geometrical singularities in the neuromanifold of multilayer perceptrons. Advances in neural information processing systems, 14, 2001.
Shun-ichi Amari, Hyeyoung Park, and Tomoko Ozeki. Singularities affect dynamics of learning in neuromanifolds. Neural computation, 1875)1000 2006.
VI Arnold. On the presentation of continuous functions of three variables by superpositions of continuous functions of two variables. Sbornik: Mathematics, 48(90):3-74, 1959.
Vladimir Amold. On functions of three variables. Collected Works: Representations of Functions, Celestial Mechanics and KAM Theory, 1957-1965, pp. 5-8, 2009a.
Vladimir I Aroll.. On the representation of functions of several variables as a superposition of functions of smaller number of variables. Collected works: Representations of functions, celestial mechanics and KAM theory, 1957-1965, pp. 25 25-46, 2009b.
Jean-Paul Berrut and Annick Welscher. Fourier and barycentric formulae for equidistant hermite trigonometric interpolation. Applied and Computational Harmonic Analysis, 23(3):307-320, 2007.
Koushik Biswas, Shilpak Banerjee, and Ashish Kumar Pandey. Orthogonal-pade activation functions: Trainable activation functions for smooth and faster convergence in deep networks. arXiv preprint arXiv:2106.09699 096993. 2021.
YEVGENTY Bodyanskiy and SERHIT Kostiuk. Learnable extended activation function for deep neural networks. International Journal of Computing (Oct. 2023), pp. 311-318, 2023.
Pakshal Bohra, Joaquim Campos, Harshit Gupta, Shayan Aziznejad, and Michael Unser. Learning activation functions in deep (spline) neural networks. IEEE Open Journal of Signal Processing, 295-309, 2020.
Fred Brackx, Hennie De Schepper, Nele De Schepper, and Franciscus Sommen. Hermitian clifford- hermite wavelets: an alternative approach. Bulletin of the Belgian mathematical Society-Simat Stevin, 15(1):87107,, 2008
11

---

Published as a conference paper at ICLR 2026
Jürgen Braun. An application of Kolmogorov's superposition theorem to function reconstruction in higher dimensions. PhD thesis, Universitäts und Landesbibliothek Bonn, 2009.
Jürgen Braun and Michael Griebel. On a constructive proof of kolmogorov's superposition theorem. Constructive approximation, 30:653-677 2009.
Adhemar Bultheel, Pablo Gonzalez- Verza, Erik Hendriksen, and Olav Njästad. Orthogonal rational functions and continued fractions. Special Functions 2000: Current Perspective and Future Directions, pp. 87-109, 2001.
Ovidiu Calin. Deep learning architectures. Springer, 2020.
Yueyang Cang, Li Shi, al. Can kan work? exploring the potential of kolmogorov-arnold networks in computer vision. arXiv preprint arXiv:2411.0672. 2024.
D Castellanos and William E Rosenthal. Rational chebyshev approximations of analytic functions. The Fibonacci Quarterly, 31(3):205-215 1993.
Qian Chen, Linxin Yang, Akang Wang, Xiaodong Luo, and Yin Zhang. Quadenhancer: Leveraging quadratic transformations to enhance deep neural networks. In The Thirty-ninth Anmual Conference on Neural Information Processing Systems, 2025. URL. stttps://openret ew. net/for
um d= xx
Ghinwa F Choueiter and James R Glass. An implementation rational wavelets and filter design for phonetic classification. IEEE Transactions on Audio, Speech, and Language Processing, 15(3): 939-948, 2007.
Grigorios G Chrysos, Stylianos Moschoglou, Giorgos Bouritsas, Yannis Panagakis, Jiankang Deng, and Stefanos Zafeiriou. P-nets: Deep polynomial neural networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7325-333357 7335, 2020.
Grigorios G Chrysos, Bohan Wang, Jiankang Deng, and Volkan Cevher. Regularization of polynomial networks for image recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16123-16132, 2023.
Charles K Chui and Jian-zhong Wang. A cardinal spline approach to wavelets. Proceedings of the American Mathematical Society, 11333)77557733 1991.
Madison Cooley, Shandian Zhe, Robert M Kirby, and Varun Shankar. Polynomial-augmmentee neural networks (panns) with weak orthogonality constraints for enhanced function and pde approximation. arXiv preprini arXiv:2406.0233 02336 2024.
George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems, 2(4):303-31 314, 1989.
M Deepthi, GNVR Vikram, and P Venkatappareddyy. Development of a novel activation function based on chebyshev polynomials: an aid for classification and denoising of images. The Journal of Supercomputing. (9(18):2011522
2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference compuler vision and puttern recognition, pp. 248-255. leee, 2009.
Stefan Elfwing. Eiji Uchibe, and Kenji Doya. Weggoid-weighted linear units for neural network function approximation in reinforcement learning. Neural networks, 107:3-11, 2018.
Daniele Fakhoury, Emanuele Fakhoury, and Hendrik Speleers. Exsplinet: An interpretable and expressive spline-based neural network. Neural Networks, 02:332-3466, 2022.
Feng-Lei Fan, Mengzhou Li, Fei Wang. Rongjie Lai, and Ge Wang. On expressivity and trainability of quadratic networks. JEEE Transactions on Neural Networks and Learning Systems, 36(1): 2228-224, 2023.
12

---

Published as a conference paper at ICLR 2026
Haishuo Fang, Ji- Ji-Ung Lee, Nafise Sadat Moosavi, and Iryna Gurevych. Transformers with learnable activation functions. In Findings of the Association for Computational Linguistics: EACL 2023, pp. 2382-23929 2023.
Dennis Gabor. Theory of communication. part 1: Institution of Electrical Engineers-por III: radio and communication engineering, 93(26):429- 441, 1946.
The analysis of information. Journal of the
Tianxiang Gao, Siyuan Sun, Hailiang Liu, and Hongyang Gao. Global convergence in neural odes: Impact of activation functions. In The Thirteenth International Conference on Learning Representations, 2025.
James F Geer. Rational trigonometric approximations using fourier series partial sums. Journal of Scientific Computing, 10:325-355 1995.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
Aaron Gokaslan and Vanya Cohen. Openwebtext corpus, 2019. URL ithub. htt.:a/eoooopeeeeeeeetCorrrrprp
Mohit Goyal, Rajan Goyal, and Brejesh Lall. Improved polynomial neural networks with normalised activations. In 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1-8, 2020.
doi: 10.1 D0.1000/UCNN446052200000000
Alexander Grossmann and Jean Morlet. Decomposition of hardy functions into square integrable wavelets of constant shape. SIAM journal on mathematical analysis, 15(4):723-736, 1984.
Boris Hanin and David Rolnick. How to start training: The effect of initialization and architecture. Advances in neural information processing systems, 31, 2018
Baran Hashemi, Roderic Guigo Corominas, and Alessandro Giacchetto. Can transformers do enumerative geometry? In The Thirteenth International Conference on Learning Representations, 2025. httt:ss///penneeei.eeeeetettfrrs
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Robert Hecht-Nielsen. Kolmogorov's mapping neural network existence theorem. In Proceedings of the international conference on Neural Networks, volume 3, pp. 11-14. IEEE press New York, NY, USA, 1987.
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks. Neural networks, 3(5):551-560, 1990.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning. pp. 2790-2222220 2799. PMLR. 2019.
Boris Igelnik and Neel Parikh. Kolmogorov's spline network. IEEE transactions on neural networks,
14(4):725-733, 2003.
Vugar E Ismailov. Addressing common misinterpretations of kart and uat in neural network literature. Neural Networks, pp. 108361, 2025.
Jean-Pierre Kahane. Sur le théorème de superposition de kolmogorov. Journal of Approximation Theory, 1975.
Andrej Karpathy. NanoGPT, 2022. URL wttp///tts///ggtt...././//tttttt
Patrick Kidger and Terry Lyons. Universal approximation with deep narrow networks. In Conference learning theory, pp. 2306-2327. PMLR, 2020.
13

---

Published as a conference paper at ICLR 2026
Joe Kileel, Matthew Trager, and Joan Bruna. On the expressive power of deep polynomial neural networks. Advances in neural information processing systems, 32, 2019.
Andrei Nikolaevich Kolmogorov. On the representation of continuous functions of many variables by superposition continuous functions of one variable and addition. In Doklady Akademii Nauk, volume 114, pp. 953-956. Russian Academy of Sciences, 1957.
Mario Köppen. On the training of kolmogorov network. In Artificial Neural Networks-ICANN 2002: International Conference Madrid, Spain, August 28-30, 2002 Proceedings 12, pp. 474-479. Springer, 2002.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. Technical report, 2009.
Kaie Kubjas, Jiayi Li, and Maximilian Wiesmann. Geometry of polynomial neural networks. Algebraic Statistics, 15(2):295-328, 2024.
Véra Kúrková. Kolmogorov's ' theorem and multilayer neural networks. Neural Networks, 5(3):501- 506, 1992. ISSN 0893-6080. doi:
URL https www.:wccience...c rect http://ccttcccccceeeeceeecc http::/tcwccc.ceeee///o/oooo///a
Miklós Laczkovich. A superposition theorem of kolmegorov type for bounded continuous functions. Journal of Approximation Theory, 269:105609, 2021.
Ming-Jun Lai and Zhaiming Shen. The kolmogorov superposition theorem can break the curse of dimensionality when approximating high dimensional functions. arXiv preprint arXiv:2112.09963, 2021.
Yann LeCun, Corinna Cortes, and Christopher Burges. The mnist database of handwritten digits, 1998.
Moshe Leshno, Vladimir Ya Lin, Allan Pinkus, and Shimon Schocken. Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural networks, 6(6):861-867, 1993.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xic. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 11976-11988 2022.
Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruehle, James Halverson, Marin Soljacic, Thomas Hou, and Max Tegmark. KAN: Kolmogorov- networks. In The Thirteenth International Conference on Learning Representations, 2025, URL et 2t//formm2dd=020000500
Vishnu Suresh Lokhande, Songwong Tasnecyapant, Abbay Venkatesh, Sathya N Ravi, and Vikas Singh. Generating accurate pseudo-labels in semi-supervised learning and avoiding overconfident predictions via hermite polynomial activations. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 1435-11443, 2020.
GG Lorentz. Approximation of functions, athena series. Selected Topics in Mathematics, 1966.
Kyle Luther and H Sebastian Seung. Sample variance decay in randomly initialized relu networks. arXiv preprint arXiv: 11X.::9902.04422. 2019.
Liying Ma and Khashayar Khorasani. Constructive feedforward neural networks using hermite polynomial activation functions. IEEE Transactions on Neural Networks, 16(4):821-833, 2005.
Stéphane Mallat, Chapter 4 time meets frequency. In Mallat Stéphane (ed.), A Wavelet Tour of Signal Processing (Third Edition), pp. 89-153. Academic Press, Boston, third edition edition, 2009. ISBN 978-0-12-374370-11 doi:
1.00008-2 URL https
www. /ttp://eeeaaaaaoo...oooooooo//oa
Mare Martinez-God Gost, Ana Pérez-Neira, and Miguel Ángel Lagunas. Enn: A neural network with det adaptive activation functions. IEEE Journal of Selected Topics in Signal Processing, 2024.
14

---

Published as a conference paper at ICLR 2026
JC Mason, Giuseppe Rodriguez, and Sebastiano Seatzu. Orthogonal splines based on b-splines- wwith applications to least squares, smoothing and regularisation problems. Numerical Algorithms, 5: 25 25-40, 1993.
Warren McCulloch and Walter Pitts. A logical calculus the ideas immanent in nervous activity. The bulletin of mathematical biophysics, 5:115-133, 1335--3,,.10 1943.
Ali Mehrabian, Parsa Mojarad Adi, Moein Heidari, and Ilker Hacihaliloglu. Implicit neural repre- sentations with fourier kolmogorov-aroodd networks. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp- 1-5. IEEE, 2025.
Alejandro Molina, Patrick Schramowski, and Kristian Kersting. Padé activation units: End-to-end learning of flexible activation functions in deep networks. In International Conference on Learning Representations, 2020. URL https:::: http:///ppeeeeeeeeeeeeeeee
Hadrien Montanelli, Haizhao Yang, and Qiang Du Deep relu networks overcome the curse of dimensionality for generalized bandlimited functions. Journal of Computational Mathematics, 39 (6):801-815, 2021.
Nafise Sadat Moosavi, Quentin Delfosse, Kristian Kersting, and Iryna Gurevych. Adaptable adapters. In Proceedings of the 2022 conference of the North American chapter of the association for computational linguistics: human language technologies, pp. 3742-3753, 2022.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pp. 807-814,
2010.
Burak Nebioglu and Alexander I Iliev. Higher order orthogonal polynomials as activation functions in artificial neural networks. Serdica Journal of Computing. 17(1):1-16, 2023.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, pp. 7. Granada, 2011.
Kurt Pasque, Christopher Teska, Ruriko Yoshida, Keiji Miura, and Jefferson Huang. Adversarially robust neural network decision boundaries via tropical geometry. Neural Networks, pp. 108624, 2026.
Nikolaos Passalis, George Mourgias-Alexandir Apostolos Tsakyridis, Nikos Pleros, and Anastasios Tefas. Variance preserving initialization for training deep neuromorphic photonic networks with sinusoidal activations. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1483-1187. IEEE, 2019.
Allan Pinkus. Approximation theory of the mlp model neural networks. Acta numerica, 8:143-195, 1999.
Evgenii Pishchik. Trainable activations for image classification. Preprints, 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl Blog, 2019.
Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. In The Sixth International Conference on Learning Representations, 2018. URL /ttl:t::///ppeeeeelew....
Reza Rezaeian, Moein Heidari, Reza Azad, Dorit Merhof, Hamid Soltanian Zadeh, and Ilker Haci- haliloglu. S12a-inr: Single-layer learnable activation for implicit neural representation. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pp. 26065-26074, 2025.
Frank Rosenblatt. The perceptron: a probabilistic model for information storage and organization in the brain. Psychological review, 65(6):386, 1958.
15

---

Published as a conference paper at ICLR 2026
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-propagating errors. nature, 523(6088):533--36 1986.
Seyd Teymoor Seydi. Exploring the potential of polynomial basis functions in kolmogorov-aroold net-
works: A comparative study of different groups of polynomials. arXiv preprini arXiv:2406602588, 2024.
Vahid Shahverdi, Giovanni Luca Marchetti, and Kathlén Kohn. Learning on razor's s edge: Identifia- bility and singularity polynomial neural networks. The Fourteenth International Conference on Learning Representations, 2026. URL
WeycAx
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in neural information processing systems, 33:7462-7473, 2020.
Georgios Smyrnis and Petros Maragos. Tropical polynomial division and neural networks. arXiv preprint arXiv:1911.1292.. 2019.
David A Sprecher. On the structure of continuous functions of several variables. Transactions of the American Mathematical Society, 115:340-355 3555, 1965.
David A Sprecher. A numerical implementation of kolmogorov's superpositions. Neural networks, (5):765-772, 1996.
David A Sprecher. A numerical implementation of kolmogorov's superpositions Neural networks, (133)447-457, 1997.
Mohammadamin Tavakoli, Forest Agostinelli, and Pierre Baldi. Splash: Learnable activation
functions for improving accuracy and adversarial robustness. Neural Networks, 140:1-12, 2021.
L. N. Trefethen and M. H. Gutknecht. Padé, stable Padé, and Chebyshev-Pade approximation, pp. 227-264. Clarendon Press, USA, 1987. ISBN 0198536127.
Joseph Turian, James Bergstra, and Yoshua Bengio. Quadratic features and deep architectures for chunking. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Companion Volume: Short Papers, pp. 245-248 2009.
Konstantin Usevich, Ricardo Augusto Borsoi, Clara Dérand, and Marianne Clausel. Identifiability of deep polynomial neural networks. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL
Venkatappareddyy, Jayanth Culli, Siddharth Srivastava, and Brejesh Lall. legendre polynomial based activation function: An aid for modeling of max pooling, Digital Signal Processing, 115: 103093, 2021.
Ross Wightman. Pytorch image models. http:://ggttuuu..oomm/vvggaaaaa/
mage-models 2019.
Tingxiong Xiao, Weihang Zhang, Yuxiao Cheng, and Jinli Suo. Hope: High-order polynomial expansion of black-box neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Xingyi Yang and Xinchao Wang. Kolmogorov-armold transformer. In The Thirteenth International Conference on Learning Representations, 2025. URL httpp:///ooeeeeeeee.eeeeeeeeeeee
?id=BCeock53nt
Ruriko Yoshida, Georgios Aliatimis, and Keiji Miura. Tropical neural networks and applications to classifying phylogenetic trees. In 2024 International Joint Conference Neural Networks (ICCN), pp. 1-9. IEEE, 2024.
Runpeng Yu, Weihao Yu, and Xinchao Wang. Kan or mlp: fairer comparison. arXiv preprint arXiv:2407.16674 2024.
16

---

Published as a conference paper at ICLR 2026
Liwen Zhang, Gregory Naitzat, and Lek- Lek-Heng Lim. Tropical geometry of deep neural networks, In International Conference on Machine Learning, pp. 5824-5832 PMLR, 2018.
Kuang Zheng and Cui Minggen. Rational filter wavelets. Journal of mathematical analysis and applications, 239(2)22772444, 1999.
Jun Zhou, Huimin Qian, Xinbiao Lu, Zhaoxia Duan, Haoqian Huang. and Zhen Shao. Polynomial activation neural networks: Modeling, stability analysis and coverage bp-training. Neurocomputing, 359:227-240, 2019.
Zhijian Zhuo, Ya Wang. Yutao Zeng, Xiaoqing Li, Xun Zhou, and Jinwen Ma. Polynomial com- position activations: Unleashing the dynamics of large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https:::: htt.httts:///peeeeeee //tp://eeeeeeee.nn SE/Eoruum2.GCCCppPbYYb
17

---

Published as a conference paper at ICLR 2026
A
SCHEMATIC OF BASIS-MLP
Basis-MPP
Lincar layer X WX +
Linear layer x WX +
Learnable basis Σ Lig axx() fir by are leamed (X0)
Activation function Rx)U(X)
Figure 4: A classical MLP (linear + ReLU) vs Basis-MLP (linear learnable basis function) blocks.
B FORWARD AND BACKWARD SECOND MOMENT CALCULATION FOR THE RELU ACTIVATION FUNCTION
B. SECOND MOMENT OF THE RELU ACTIVATION FUNCTION
The Rectified Linear Unit (ReLU) activation function Nair & Hinton 2010)), defined as:
(eLU(x) = max(0, r)
(25)
commonly used in neural networks due to its simplicity and effective gradient propagation. When x drawn from standard normal distribution ~ ~N(00, 1), the second moment of the ReLU function is:
E [ReLU(x))] = 1/0 x² 2
x² 22
(26)
2
B.2 SECOND MOMENT THE DERIVATIVE OF RELU
The derivative of ReLU, given by:
ReLU(x) 1,
x
(27)
00 x 5 0,
acts as binary indicator of positive inputs. The second moment of this derivative when æ ~ ~N(0 1)
is:
1 }
E d
ReLU()))
(28)
2T
This result matches the variance of the ReL.U function itself and validates the gain of 2 for variance- preserving weight initialization with ReLU activations.
C PROOF OF THE THEOREM 3.8
Definition C.1. We define the Hermite activation F: + R with its learnable coefficients Vn E N, vk E [0, n| 0g IR as:
x =+ =
ag
Hex(z)
(29)
k!
k=0
Property C.2. Vm, n N?, we have:
Hem (x) He, (x)e 3 dx 2mm15...
(30)
With 8mm the Kronecker delta.
18

---

Published as a conference paper at ICLR 2026
Proposition C.3. The second moment of this activation with respect N(, 1) is:
72
E
[F
(31)
k!
Proof. The proof relies on the orthonormality property
C.2
The orthonormality property C.2 means that: Vm, n2 E N?,
Hen (x)) e- =/4
dx
(32)
n!
22T
and if m # n
Hem (x) He (x)
dr
(33)
22T
Given the definition (Def. C.1 of a Hermite activation F. we have:
[F(x)*] = |
(x)
e
dr
(34)
2
2
- 2
Σ
Ok
(x)
dr
(35)
!!
2
Using the orthogonal property Eq, 33 the cross terms cancel out, and we have:
E
[F(())
He (x)
22T dx
(36)
(k!)
a
(e)
dx
(37)
(k!)2
22x
k=
Given the normality property Eq.
32
have
Hep (2)(2)² e dx
4/2
E [F(x)
(38)
hd
2m
k=U
a
(39)
Having found the initialization gain for the activation F, we now need to enforce this same gain for its derivative. Indeed, we are going to the gradient descent algorithm to train our learnable activation networks, and having an activation gradient of high (respectively low) variance could lead to exploding (respectively vanishing) gradients, nondesirable property for deep neural networks trained with gradient backpropagation.
Property C.4. The following recurrence property is derived directly from the equation 6 Yk e Vr E R.
N
(e)( = x Hey(x) - (x))
(40)
Property C.5. The The following property shown by induction and by using the previous property Vk E N' Va E R:
C.4
Hel, (x) = Heg-1(z)
(41)
Proposition C.6. Using the last property and by the linearity of the integral, the derivative of F (Def. C.I. F²: + R is written as follows:
x + F'(x)
Σ
0k
Hcx-1(x)
(42)
(k - 1)!
R=
19

---

Published as a conference paper at ICLR 2026
Remark C.7. A first remark here is that Vn > 2: F' is unbounded [im F'(x) + oo). This means that F is not Lipschitz continuous. Lipschitz continuity is often desired (or even required) when training a deep neural network using gradient backpropagation. However, by suitable initial choice of the coefficients (ax (as))E[0.n] we can keep the Lipschitz constant under control.
Proposition C.8. The second moment of the derivalive of the Hermite activation is:
2
E {F)
Σ
(43)
(k:
1)!
Proof. Knowing that Ak E N Vx ER:
(e) k Hee (1)
(44)
The definition of F" becomes:
R R
11 kak (1
x x F*))
(45)
Thus, the second-order moment of F' is:
c dx
E [F
1}! Hes H()) (3r)
(46)
2n
By the orthogonal property Eq.
33 the cross terms cancel out, and we have:
E [x(()))
((k 0 1)!22
HeR 1 (x)20T 2nT dx
(47)
az
400
He (x) e - dx
(48)
(k- 1)!
(k: - 1)! 2T
1=1
(49)
By the normality property Eq. 32
we finally have:
E (F)
k
(50)
(k 1)!
Proposition C.9. Equality between propositions
C.3
and C.8
imposes that:
n2
0
(51)
k=1
1
(52)
1)!
To satisfy the as Vn E N
gain equality, we could initialize the coefficients (ax)kefo..) such
Vk E [[, au = and
(53)
n!
This initialization works in practice for all n. Furthermore, as the term in as vanishes quickly with + +oo, for larger could initialize all the coefficients to 1 including og.
In the limit case, by a simple injection of ag = 1 in Prop. C.9 and then in Prop. C.8 we obtain the result.
20

---

Published as a conference paper at ICLR 2026
D PROOF OF THE THEOREM 3.13
Definition D.I. We consider the following Fourier activation F: IR + R:
(ax cos(())) + by sin (kxz))
x - F(x) ag +
(54)
where (aE) and (by) are real learnable coefficients.
Property D.2. The equivalent of the C.2 property for trignometric, functionns is given by Vm, n E Z²:
cos(maa cos(ner) Tmâm
sin(ma) (in(nx)) dx T8nm
(55)
cos (mx) sin sin(nn))d dr 0
With 8am the Kronecker delta function. Proposition D.3. The second moment of this activation is:
(a bg)
E
F(((()))) = ag
(56)
(R:)2
A=
Proof. The proof relies on the orthonormality property D.2
The random variable æ is assumed to follow a uniform distribution on the interval [-x, T.. denoted as:
T u (-
(57)
To compute the second moment of the Fourier activation F(x), we need compute the expected value of F
B[F(z)²1= = [" F(x) da
(58)
where p(z) is the probability density function (PDF) of the uniform distribution:
p(x) 2x
x E [-x,
(59)
Taking the square of the definition Eq, 54 gives::
(ax cos(kar)) + by sin (k(()))) 2
F(x))= 0g
(60)
k!
Using the orthogonal property and and the linearity of the integral, we have:
B[F(z) -0 +
a? cos? cos2((xx)) + by sin sinr(ka)) da
(61)
2T
(k:))2
kml
a
sin (2
b4
sin (2
+
+ +
π
(62)
2T
(k:)²
2k
(k!)²
2k
The second moment simplifies to:
(a + b2)
E[F(s) o/
E
(63)
(k!))
21

---

Published as a conference paper at ICLR 2026
Next, we compute the second moment of the derivative of the Fourier activation F". The derivative of is given by: Proposition D.4. The derivative of the Fourier activation F'; - IR is given by:
((
(-ax sin(ker + by cos(kz))
(64)
kk 1)!
Remark D.5. Contrary to the remark in C.7 F' is bounded.
Vr E R: (x)) S max axl bu DREIL.]
(k 1}! S e max(laub 1bx DREgA.
(65)
R=
This means that in the case of a Fourier activation, F is Lipschitz continuous.
Proposition D.6. The second moment of the derivative of the Fourier activation is:
" Σ
E F (1))
1)!2 (1 + +b)
(66)
((k
Proof. An orthonormality argument as for the proof in the forward case suffices.
Proposition D.7. Equality between
D.3 lannDD. imposes that:
(k2 1) (0 bb)
dg
(67)
2
(k!)²
To satisfy the bawward-ackward gain equality, we could again initialize the coefficients such as Vn E N
Vk E [1. ax
and do
(68)
(n2)²
This initialization works in practice for all n. Furthermore, the term (nt) in ag vanishes quickly with n > +00, for larger n we could initialize all the coefficients to including ag.
In the limit case, by simple injection of a = 1 in Prop. D.7 and then in Prop. D.6 we obtain the result.
Remark D.8. For an input æ of distribution x ~ U1 v 3. v3), which has a variance of Var[æ] = and which more in line with deep neural networks that seek unitary variance preserving property across layers, we could rescale the fundamental frequency given in the definition of F in Def. D. by redefining it as:
1 1 k! 1
TH F(z) + Σ
" x) + by sin(k " x)) ax cos(k ²²²²²
(69)
/3"
k=
The computation of the second moment stays the same except for factor V In general if U(- u(-1,1), RJ, and if e the fundamental frequency, this last should be scaled by w' Fw.
E PROOF OF THE THEOREM 3.19
Proof. Consider the function:
V2 max {1+ kz) F(x)
V2 (1+ mäx ka k=
(70)
n k=0
71
Note that since R, the maximum over i depends the sign of z:
If r > 0, then max {kx] = nz.
If < 0, then max (kar) 0 (achieved at A 0).
22

---

Published as a conference paper at ICLR 2026
Thus, write
V2 + nx) = /2r
x>
F(x)= =
n
n
(71)
1/2
J 0
We now analyze the variance of F(x) under the assumption that ~ (O, 1) (standard normal input):
As + oo, the function becomes approximately:
{
[22,,
x 0
F(x) 2
(72)
10,
I 5
This is similar to a scaled ReLU: F(x) = v2- max(0, x), which we know from Appendix B has unitary forward and backward gains.
F DEEP POLYNOMIALLY ACTIVATED NEURAL NETWORKS ARE MULTIVARIATE POLYNOMIAL MAPPINGS
Deep MLPs compositions of affine transformations and activation functions applied layer by layer. When the activation functions are polynomial, the entire network can be expressed as a polynomial mapping.
Definition F.1. Let m E N. A function F R" + R. is called a polynomial mapping if each component function F; RR" + R, for 1,... m, a polynomial in n variables. Explicitly, this means that for each i, F; has the form:
F Rn)
Σ |a/Sd:
Ci
01 x22
.... on
(73)
where the sum is taken all multi-indices (01,..., an) No such that |a|| an + 02
... + an S di, Ci E R are real coefficients, and d; E N.
Definition F.2. A deep neural network with layers, input dimension n, and output dimension m is function F: Rn + mm of the form:
F (x) = WLo(WL-io(....o)
b1) + by-1) + by,
(74)
where Vi E [I C, E N Each W; CR a weight matrix, by E RR a bias vector, and is activation function applied element-wise.
Proposition F.3. Let F' IR" + R. be deep neural network with polynomial activation functions of degree d. Then F is a polynomial mapping of degree most d'. Furthermore, any L-layer MLP could be collapsed into an equivalent 3-layer network with the middle layer being a polynomial mapping of degree at most d'.
Proof. The proof proceeds by induction the number of layers L. and is detailed in what follows.
Base case: For L = 1, the network takes the form
F(x) Wio + bb) + b,.
Since is polynomial of degree d, applying it the affine transformation Wox + by yields polynomial mapping of degree at most d. Therefore, F(x) is a polynomial mapping of degree at most d.
Inductive step: Assume the statement holds for I - layers, meaning the network Fr.-1(x) is polynomial mapping of degree at most d--1, For the T.-layer case, have
F(x) (Lo + by.
Since o is a polynomial degree d, applying it to Fı.-1()) results in a polynomial of degree at most d. d--1 d4 Thus, by induction, the statement holds for all 7. > 1.
23

---

Published as a conference paper at ICLR 2026
Corollary F.4. Any deep neural network with polynomial activation functions realizes potynomial mapping whose degree grows exponentially with the number of layers.
Remark F.5. The total number of monomial terms in this mapping is (dt d'
Remark F.6. An equivalent considcration for trigonometric polynomials be established by approximation, but will not be covered here.
G ALGORITHMS
Algorithm Initialization of Hermite Grid and Coefficients
Input: Polynomial degree
Output: Coefficients tensor coeffs, Grid of powers tensor grid
Initialize coeffs and grid as zero matrices of shape (n + 1, 1,,//22+ for i 0 to do for j = to 2 do
if: j S then
coeffs|[][]] + (-1)ee(- (ooodloooo)--eeee (ogg())) 25)!) log(2))
grid[1]
[ t i 2j
else
coeffs[]][]] < 0 [rid
end if
end for end for return coeffs, grid
Algorithm 2 Hermite Activation Function Forward Pass
Input: Input tensor x, polynomial degree n
Parameters:
Learnable polynomial coefficients A R.
Output: Output tensor after applying Hermite activation function
coeffs, grid +- Initialize cirtiflizeecoeffssssc _grd()
Procedure Forward(zz):
+- x.repeat(n + r).repeat(nn/2 1)
x |a grid @ sign(a))end
< @Cooeffs
x r@A
return
End Procedure
Algorithm 3 Hermite Forward CUDA Kernel
Input: Input tensor æ, degree n, output tensor out
Output: Computed Hermite polynomials up to degree
Procedure HermiteForwarCUUAAL n., out):
for in parallel index size(z):
outli n < 1.0
if n > 1: out n + 1] - x]]]
for A 2 to
out(i n + k] t -ot(i: n n 1] - (I - 1) outli - n 1: 2]
End Procedure
24

---

Published as a conference paper at ICLR 2026
Algorithm Hermite Backward CUDA Kernel
Input: Input tensor x, degree n, output tensor out, gradient tensor grad_out out
Output: Computed gradients for Hermite polynomials
Procedure Heemiie 71, out, grad_ for in parallel index size(grad_our
grad <- 0.0 for to n:
grad grad + ari k] k outfi + k 1]
grad_out[i] < grad
End Procedure
Algorithm 5 Fourier Activation Function Forward Pass
Input:
Input tensor v, degree
Parameters: Learnable coefficients A E R", fundamental E R, phases P E RR, frequencies F E R",
Output: Ouiput tensor after applying Fourier activation function
Procedure FourierActivationn())
Ø - r.repeat(n (n + 1)
F Fca x P
x - 22cos(z) xx)
A
T + I +
return End Procedure
Algorithm 6 Tropical Activation Function Forward Pass
Input: Input tensor æ, degree Parameters: Learnable coefficients A E g
Output: Output tensor after applying Tropical activation function
powers range(0,n+1) 1)
Procedure Forward(z):
x. repeat(n 22/n max(# @ powers + A,dim -1)
return æf End Procedure
25

---

Published as a conference paper at ICLR 2026
H RATIONAL TROPICAL ACTIVATION
Definition H.1. The tropical quotient 2 of x over y is defined as:
x 2 y == x -
(75)
Definition H.2. The tropical rational activation defined as the quotient of two tropical polyno- mials Fi and F22 of degree m, E N² respectively.
F: - F(x) Fi F2(x) (z F2(x)
(76)
An example of fitting a classical activation (GELU) with rational tropical activation is shown in Figure Rational tropical activation understood here in the general sense, i.e., with real powers.
CIRLA
GEUU deri.
Trspical Rai
Tropical Rat.
4
F(X)
2
0-
2-axis
Figure 5: Hermite interpolation GELU with Tropical Rational Activation of degree both the numerator and the denominator.
example of fitting a convex function (a - ) with polynomial tropical function (in the general sense) is shown in Figurel6
10
F(X)
5
2
0
-2
2
T--axi
2.2 Figure 6: Interpolation of function by the Tropical-Laurent polynomial (with potentially negative
2
powers) max (kx k=-3
3
} of degree 6.
2
26

---

Published as a conference paper at ICLR 2026
I
ABLATION STUDIES
Table Ablation studies the degree of the activation on ConyNeXt-T model.
Activation
Degree
Train Loss
Val Top-1 (%)
Val Top-5 (%)
Tropical
1
2.925
81.60
95.73
Tropical
3
2.866
82.01
95.91
Tropical
5
2.863
82.18
96.00
Tropical
2.857
82.20
95.90
Fourier
2.872
80.29
95.03
Fourier
2.850
80.61
95.26
Fourier
5
2.844
80.69
95.41
Fourier
6
2.837
80.93
95.44
Hermite Hermite
2
2.833
81.66
95.71
3
2.790
82.34
96.03
Table 4: Ablation studies for the initialization the activation on ConvNeXt-T model.
Activation
Degree
Initialized from
Train Loss
Val Top-1 (%)
Val Top-5 (%)
Fourier Fourier
6
GELU
2.775
81.91
95.77
Thrm. 3.13
2.837
80.93
95.44
Hermite Hermite
3
GELUU
2.809 2.790
82.04
95.91
Thrm. 3.8
82.34
96.03
Table 5: Ablation studies for the learnability of the parameters of the activation on ConvNeXt-T
model.
Activation
Degree
Learnable?
Train Loss
Val Top-1 (%)
Val Top-5 (%)
Tropical Tropical
X
3.560
76.31
93.09
6
V
2.857
82,20
95.90
Fourier Fourier
6
X
3,181
79.51
94.60
6
V
2.837
80.93
95.44
Hermite
3
X
3.411
78.48
94.20
Hermite 3
2.790
82.34
96.03
Table 6: Ablation studies for the clamping the Hermite activation on ConvNeXt-T model.
Activation
Degree Clamped?
Train Loss
Val Top-1 (%)
Val Top-5 (%)
Hermite
2.772
81.98 82.34
95.81
Hermite
X
2.790
96.03
J A BRIEF DIGRESSION ON KOLMOGOROV ARNOLD NETWORKS (KANs)
Kolmogorov-Armold networks Liu et al. 20255) have been presented potential alternative to Percepptrosrrs (MLPs), promoting several merits such as greater accuracy, fewer learnable parameters, and better interpretability. While the first two advantages could only be demonstrated for simple cases in the (Liu et al. 2025) article, the third benefit more straightforward, as these networks overcome the "black-box" aspect of traditional non-linear activations MI.Ps by allowing
27

---

Published as a conference paper at ICLR 2026
the activation to be polynomial, piece-wise polynomial, rational, in Yang & Wang (2025). From there, having learned the weights of the network and those of the activation, it becomes clear what approximation these functions (polynomial, rational, trigonometric have converged to.
Rather than providing a direct application of the celebrated Kolmogorov-Amon rem (KART) Kolmogorov (1957); Arnold (1959), the recent work on KAN et al. (2025) appears to take inspiration from it in more figurative sense. For clarity, we recall that the Kolmogorov-Amoll representation theorem, cited below, states that any continuous multivariate function f: [0, 1 I
representation theo-
can be represented as a composition of addition and some functions of one variable denoted by Va. and by:
Theorem J.1. (Arold (2009b(a)) Let J [0, 1Jth IR be an arbitrary multivariate continuous
function. Then il can be represented as follows:
2n
f (TI, In)
Σ ap (1p)
(77)
with continuous one-dimensional functions Øg: - Te. and uh funcions and ap called inner functions. The inner functions Vapp independent of the function f.
[0, &g are called outer
This differs substantially from KAN'S formulation Liu et al. (2025), where the outer functions disappear, the inner functions are replaced by weighted sum of a SiLU MLP MLP[Elfwing et al. (2018)
and B-spline, and the networks are composition of multiple feed-forward layers to accommodate recent neural network architectures.
Since the KART proof not constructible, and is essentially based on Baire's theorem Kahane (1975), )
the first efforts to implement constructive proof of the KART were made by Sprecher in Sprecher (1996] 1997). These latest works are based on a more economical variant of the KART terms of the number of outer and inner functions due to both Sprecher 1965) and Lorentz (1966).
This was followed by the first article on the practical training of this type of network by Köppen (0002), pointing out at the same time that the inner function & constructed in this theorem was continuous but fractal! This limited its use in gradient-based learning algorithms. [Braun & Griebel (2009) gave rigorous proof of termination, continuity, and monotonicity for the construction of the inner and the outer functions given by Sprecher (1997).
As acknowledged by both Liu al. 2025)) and Yang Wang (2025), the original "KAN" layer defined in Liu et al.] (2025) could be seen as a sum of SiLU MLP and weighted B-Spline combination. Let define a linear function Lw: + Wx, with W a learnable weight matrix. The
"KAN" layer Liu et al. (2025) is then defined as follows:
KAN (x) = Lw, (w.LU((())) + Lw.
(78)
With W, and W, two learnable weight matrices, (B.)ie family of B-spline functions of order d + 1, (c))ego.4 the learnable spline weights and si LU:
Ite-r
Indeed, if we follow the line of thought set out in KAN Liu et al. 2025). an MLP with learnable activation, equivalently learnable activation network (LAN) would be a sort of KART formulation, with the aa inner functions being a linear combination of ReLU functions. However, this is not what the KART theorem suggests. Constructing a Kolmogorov- Arnold superposition requires a maximum of two layers formulated by inner and outer functions as in theorem J.I [smailov 2025)
worth noting that the concept of using splines to approximate inner functions in a Kolmogorov- Arnold network or more generally analogy between KANs and MLPs has been noticed since Hecht-Nielsee (1877) and Kúrková (1992). Earlier research, such as Igelnik & Parikh (2003), introduced Kolmogorov's 's Spline Network, which employed splines for flexible function approximation. In his PhD thesis, Braun (009) corrected the constructive proof of the KAT and gave practical examples using B B-splines. Further developments in this area include Bohra et al. (2020) and Fakhoury et al.] (2022), who focused on learning adaptive activation functions through splines, thus enhancing the network's expressiveness.
representation of an activation function isn' 't entirely new. The
28

---

Published as a conference paper at ICLR 2026
Additionally, the use of the Kolmogorov superposition theorem to tackle high-dimensional problems has been explored by Laczkovich (2021) and Lai & Shen (202I). who showed its potential in overcoming the curse of dimensionality. Similarly, [Montaneli al. (202I) demonstrated how structured networks like Deep ReI.U models can efficiently approximate bandlimited functions, thus expanding the practical applications of spline-based methodologies in neural networks.
With an equivalent number of parameters or FLOP, Yu et al. (2024) observed that KAN surpasses MLP solely in symbolic formula representation, while falls short of MLP in other machine learning
tasks, including computer vision, NLP, and audio processing. CCag el al. (2024) confirmed the same finding.
Nevertheless, KANs have had the merit of rekindling interest in learnable activations in neural networks, among them polynomial and trigonometric activations.
Since the interest in KANs began, numerous researchers have proposed a multitude of learnable functions for activations, spanning a diverse range of mathematical functions, including splines, classical orthogonal polynomials, rational functions, Fourier bases, and wavelets... Despite this, in some instances, the safety of these operations, the boundedness of their gradients, their initialization, and their computational properties in the context of gradient descent have sometimes received less emphasis. Instead, many studies have highlighted
results, often demonstrating that such functions can achieve strong performance benchmark datasets like MNIST LeCun ct al. (1998). This line of work has produced a rich body of literature. A common observation, however, is that much of it focuses adapting specific interpolation function within relatively shallow architectures and evaluating on small-scale datasets (such as the MNIST dataset, for example). Because a wide variety of functions can achieve test accuracies exceeding 97% on MNIST with networks of depth three or less, it becomes challenging to distinguish which approaches provide the most robust or generalizable benefits.
K EXTENDED RELATED WORK
The subject of learnable activation
-ell- one, but has seen a resurgence thanks to the popularity enjoyed by the KAN article Liu Liu et al. (2025). Examples of works in which the main theme is learning the activation function include Houlsby et al. (2019); Goyal et al. (2020); Tavakoli et al.
(2021): Moosavi et al. 2022): Fang al. 2023) Bodyanskiy & Kostiuk (2023) Pishchik (2023).
The use of orthogonal polynomials as activation functions predates many recent developments. For example, Ma & Khorasani (2005) introduced Hermite polynomial activations within constructive 1-
layer neural network framework, while Venkatappareddy et al. (2021) explored Legendre polynomial- based activations better approximate max-pooling behavior.
Earlier works exploring polynomial activations in deep neural networks trained using the backpropa- gation algorithm include Zhou et al. (2019) and Chrysos al. (2020), which empirically demonstrate that polynomially activated neural networks, even without non-linear activation functions, can per- form well across multiple tasks. Building on this, Chrysos al. (023) sought regularize such networks to compete with deep ReLU networks.
More recently, Nebioglu recently,Ncbiogiu & Iliev (023) investigated the use of Chebyshev and Hermite orthogonal polynomials as activation functions, demonstrating that Chebyshev activations are computationally efficient but sensitive to problem types, while Hermite activations exhibit greater robustness and generalization. Additionally, Xiao et al. (2024) introduced HOPE (High-order Polynomial Expansion), a novel method that represents neural networks as high-order Taylor polynomials, enabling improved interpretability, low computational complexity, and applications such as function discovery, fast inference, and feature selection.
Other recent works utilizing Chebyshev activation include Deepthi et al. (2223) and Rezacian et al. (2025), which employed single-layer shallow networks. Seydi (2024) conducted comparative study of exotic polynomial activations the MNIST dataset, while Cooley et al. (224)) applied mambomiil-auggmette neural networks for approximating solutions to partial differential equations.
On the rational activation front, notable works include Trefethen & Gutknecht (987), which intro- duced stable-Pade and Chebyshev-Pars approximators, and Molina et al. (2020), which proposed the Padé activation by ensuring the denominator of the rational activation remains nonzero. An
29

---

Published as a conference paper at ICLR 2026
orthogonal variant of the Padé approximant presented in Biswas al. 2022 while Chebyshev rational functions Castellanos & Rosenthal (1993) and Fourier rational functions Geer (1995) were explored in subsequent studies. More recently, advancements in rational activation using general Jacobi functions were introduced in Aghaei 2024b a).
Polynomial piecewise functions (such B-splines) and rational functions (such as the Padé approx- imant) can exhibit finite support properties. On the other hand, these last lack the orthogonality property. Several works have aimed to formulate orthogonal splines Mason al. (1993); Alavi &
Aminikhah (2023) and orthogonal rational functions Bultheel et al. (2001), or even a theory of spline
wavelets Chui & Wang 199I) and rational wavelets Zheng & Minggen 1999); Choueiter & Glass (2007).
Learning with periodic function or Fourier series has also been the subject of many anterior works, such as Sitzmann et al. (2020), and more recently Mehrabian et al. (2025). and Martinez-Gost et al. (2024) using Discrete Cosine Transform (DCT). Recently, Hashemi et al. (2025) introduced the Dynamic Range Activator (DRA), an activation function that combines harmonic (trigonometric) and hyperbolic components to capture the highly recursive and high-variance behavior within a deep problem in enumerative algebraic geometry.
In the context of tropical activations, prior work has been done to establish connections between tropical geometry and neural networks. For instance, Zhang al. (2018) demonstrated that feedfor- ward neural networks with ReLU activation can be interpreted as tropical rational maps, relating their decision boundaries to tropical hypersurfaces and showing how deeper networks leverage zonotopes to achieve exponentially greater expressiveness. Building on this geometric foundation, [Smyrnis] Maragos| (2019) introduced tropical polynomial division, an approach inspired by the max-plus semiring, and applied it to neural networks with ReLU activation. Recent work also developed tropical activation functions Yoshida et al. (2024), which were subsequently applied to convolutional neural networks (CNNs) for image classification tasks on MNIST[LCuur et al. 1998). CIFARIO Krizhevsky et al. (2009) and SVHN Netzer et al. (20IT) in Pasque al. (2026).
The stability of deep networks with learnable or unconventional activations is closely tied to variance- preserving initialization. Prior analyses have established theoretical conditions for stable signal propagation across depth |Hanin & Rolnick (2018), while empirical and theoretical studies have examined variance decay phenomena in ReLU networks Luther & Seung (2019). For sinusoidal activations, Passalis et al. (2019) introduced dedicated variance-preserving initialization scheme.
L IMAGE CLASSIFICATION RESULTS ON CIFAR10
We conducted an experiment using ConvNeXt-T on CIFAR10 for 300 epochs and averaged the results over 10 different seeds. The experiment shows that Hermite, Fourier, and Tropical activations significantly above the GELU baseline in terms of test metrics. In addition, we added three different seeds for ResNet50 with ReLU, and the results of the latter are clearly inferior to those of To, ConvNeXt-T being a modernized version of ResNet50. The results, in table form, reported in Table Jnd and in graphical form in Figures and
Table 7: Comparison of the proposed activation functions on ResNet50 and ConvNeXt-T models for CIFAR-10 image classification task. Values are reported as mean # standard deviation 10 different seeds (3 seeds for the ResNet50 case). p-values (two-tailed Student's 's t-test assuming equal variances) compare each activation's 's Topl-accuracy against GELU.
Model
Activation
Top-1 Ace. (%)
Top-5 Acc. (%)
p-value vs GELU (Top-1)
ResNet50
ReLU
88.9 + 0.04
99.43 : 0.47
< 0.0001 (*+**)
ConyNeXt-Ting
Baseline (GELU)
90.47 + 0.20
99.62 + 0.06
ConvNeXt-Tiny
Tropical
90.87 0.19
99.63 + 0.04
0.0002 (++*)
ConvNeXt-Tim
Fourier
91.23 + 0.65
99.60 + 0.05
0.0023 (**)
ConvNeXi-Tiny
Hermite
91.35 1 0.29
99.63 1 0.05
< 0.0001 (**8*)
30

---

Published as a conference paper at ICLR 2026
train_loss
ConvNeXt-T Tropical
ConvNeXt-T Fourier
ConvNeXt-T GELU
ConvNeXt-T Hermite
ResNet50
1.9
1.7
1.5
1.3
epoch
1.2
100
150
200
250
300
Figure 7: Training loss curves for ConvNeXt-T on CIFAR-10. The solid lines represent the mean of the metric over 10 different seeds (3 seeds for the ResNer50 case), and the shaded areas show the range (min max).
eval_top1
ConvNeXt-T Tropical
ConvNeXt T Fourier
ConvNeXt GELU
ConvNeXt T Hermite
ResNet50
92
90
88
96
84
epoch
82
100
150
200
250
300
Figure 8: Evaluation Top1 accuracy curves for ConvNeXt-T on CIFAR-10. The solid lines represent the mean of the metric over 10 different seeds (3 seeds for the ResNet50 case), and the shaded areas show the range (min to max).
31

---

Published as a conference paper at ICLR 2026
eva evalttop5
ConvNeXt-T Tropical
ConvNeXt-T Fourier
ConvNeXt-T GELU
ConvNeXI-T Hermite
ResNet50
99.6
99.4
99.2
epoch
99
150
300
Figure 9: Evaluation Top5 accuracy curves for ConvNeXI-T on CIFAR-IO. The solid lines represent the mean of the metric over 10 different seeds (3 seeds for the ResNet50 case), and the shaded areas show the range (min to max).
For this experiment we used the TIMM library Wightman (2019) with the following configuration S
Table Configuration for the pretraining experiment on CIFAR-10.
Hyperparameter
Value
Input Size
3x32x32
Number of Classes
10
Batch Size
128 GPUs
Optimizer
AdamW
Learning Rate
4e-3
Epochs
300
Scheduler
Cosine
Drop Path Rate
0.0
Mean
0.491, 0.482, 0.446
Std
0.247, 0.243, 0.261
Warmup Epechs
20
Weight Decay
0.0
Mixup
0.8
Label Smoothing
0.1
Auto Augmentation
rand-m9-mstd0.55
Re-mode
Pixel
Random Erasing Prob
0.25
Gradient Clipping
5.0
CutMix
1.0
32

---

Published as a conference paper at ICLR 2026
M DECISION BOUNDARIES
To evaluate how the choice of activation function affects classification behavior and boundary smoothness, compare the decision boundaries of four single-layer neural networks trained on simple 2D classification datasets (e.g., two moons, circles, etc.). Each network uses different activation function. The Hermite activation function produces globally smooth and coherent decision boundary, reflecting its polynomial nature. Notably, when the regions of the classes intersect, boundary can develop critical point corresponding to the intersection, where the local geometry degenerates. The first sub-figure in the top left shows that the decision boundary is rather plane curve resembling an elliptic curve with two connected components and no singular points. This curve delimits three regions of the 2D space. The network uses two of these regions to classify/cluster one of the two classes of points, while the complementary region classifies the points belonging to the second class. In contrast, the Fourier activation leads quasi-periodic patterns that adapt well to the underlying structure of the data, capturing fine- fine-graines ---iinannd details and even fitting noisy points. The tropical activation yields piecewise affine boundary, resembling the behavior of ReLU, with sharp transitions and linear segments that reflect its max-plus (tropical) structure.
Hermite
Fourier
Tropical
GELU
Figure 10: Decision boundaries datasets: Top row: classification; middle moons; bottom row: circles, using four different activation functions.
33

---

Published as a conference paper at ICLR 2026
N
LINE PLOTS
Global Train/train, loss
ConvNeXt-T Hermite
ConvNeXt-T Tropical
ConvNeXt-T GELU
ConvNext- Fourier
2.95
2.9
2.85
epoch
2.75
250
260
270
280
290
Figure 11: Training loss curves for ConvNeXt-T on ImageNet The solid lines represent the mean of the metric over 5 different seeds, and the shaded areas show the range (min to max).
Global Test/test _cc___cc_____ ema
ConvNeXt-T Hermite
ConvNeXt-T Tropical
ConvNekt-T GELU ConvNeXt-T Fourier
82
81.5
81
epoch
200
220
240
260
280
Figure 12: Top1 evaluation accuracy for ConvNeXt-T on ImageNetlk. The solid lines represent the mean of the metric over 5 different seeds, and the shaded areas show the range (min to max).
34

---

Published as a conference paper at ICLR 2026
Global Test/test_acee acc5__ acc5_emma _emaa
ConvNeXt-T Hermite
ConvNeXt-T Tropical
ConvNeXt-T GELU ConvNeXt-T Fourier
96
95.8
95.6
95.4
epoch
200
240
260
Figure 13: Top5 evaluation accuracy for ConyNeXi-T on ImageNetlk. The solid lines represent the mean of the metric over different seeds, and the shaded areas show the range (min max).
train/loss
GPT2 GELU
GPT2 Fourier
GPT2 Hermite GPT2 SiLU
GPT2 Tropical
3
2.98
2.96
2.94
2.92
2.9
iter
100k
120k
140k
160k
180k
200k
Figure 14: Comparison of the train losses of the GPT2 model (124M) Open with GELU, SiLU, Hermite, Fourier, and Tropical activations. The solid lines represent the mean of the metric over 5 different seeds, and the shaded areas show the range (min to max).
35

---

Published as a conference paper at ICLR 2026
val/loss
GPT2 GELU
GPT2 Fourier
GPT2 Hermite
GPT2 SiLU
GPT2 Tropical
3
2.98
2.96
2.94
iter
100k
180k
200k
Figure 15: Comparison of the validation losses of the GPT2 model (124M) on OpenWebTex( with GELU, SiLU, Hermite, Fourier, and Tropical activations. The solid lines represent the of the metric over 5 different seeds, and the shaded areas show the range (min to max).
O FINETUNING ACTIVATIONS EXPERIMENT ON CIFAR10
We conducted an experiment for fine-tuning ConvNeXt-tinn (pre-trained on ImageNetlk) on CI- FARIO. We froze all the weights except those of the last linear layer and the ones of the learnable activations, which were initialized by fitting GELU with a Hermite interpolation. The results hereby show clear superiority of the proposed learnable activations:
fine-Tuning pretrained ConvNext-Tiny on CITAR-10: Comparing Learnable Activations with GEW
Train Loss
Test Loss
lermite
2.80
Hermite
Rounter
Founte
1.2
Toopicaltational
0.75
Tropica Rabinnl
0.9
0.8
0.65
0.5
0.60
0.3
spocts
Train Accuracy
Test Accuracy
Top1 nocuraay
Harmitte
Pousierr
Tropicallen (tatioon
Tropice Rtaionaal
Spoxchc
Figure 16: Performance of a pretrained ConvNecX-T (on ImageNetlk) on CIFARIO when fine-tuning only the final linear layer and the learnable coefficients of the activations.
36

---

Published as a conference paper at ICLR 2026
P
TIMING RESULTS
Hermite Fourier
10-1
Forward Backoarrd Time
20-2-
2
2
Degres
Forward backward pass averaged times (in seconds and log-log scale) for Hermite, Tropical, and Fourier activations across varying degrees benchmarked on an AMD EPYC 7402 CPU.
Figure 17:
Table Forward backward pass averaged times (in seconds) for Hermite, Tropical, and Fourier activations across varying degrees as benchmarked on an AMD EPYC 7402 CPU.
Activation
Degree
Forward+Backward Time
Hermite
0.00047354
Hermite
0.00188882
Hermite
0.00462004
Hermite
16
0.0168367
Hermite
32
0.0496985
Hermite
64
0.191425
Tropical
0,000300329
Tropical
0.000337174
Tropical
0.000356829
Tropical
16
0,000405076
Tropical
32
0.000512147
Tropical
64
0.000788548
Fourier
0.000270133 38
Fourier
0.000599022
Fourier
0.000644515
Fourier
16
0.000787303
Fourier
32
0.00121512
Fourier
64
0.00262779
37

---

Published as a conference paper at ICLR 2026
Hermite Hermite CUDA Tropical Fourler
10"1
Fonvord Badkward
20-7-
10a4
Degree
Figure 18: Forward + backward pass averaged times (in seconds) for Hermite (explicit Alg. and recurrence-based CUDA implementation Alg. Eq. 23). Tropical, and Fourier activations varying degrees as benchmarked on a single NVIDIA A100 GPU/40GB.
Eq. 22
Duptimg
Distim1b
Dipli-52
Figure 19: Relative slowdowns of Hermite, Tropical, and Fourier activations compared GELU across different widths.
Witt--1024
Wath 2048
Figure 20: Relative slowdowns of Hermite, Tropical, and Fourier activations compared GELU across different depths.
38

---

Published as a conference paper at ICLR 2026
Table 10: Forward backward pass averaged times (in seconds) for Hermite (explicit Alg. Eq. 22 and recurrence-basee CUDA implementation Alg. 3 Eq. 23). Tropical, and Fourier activations across varying degrees as benchmarked on a single NVIDIA A 110 GPU/40GB.
Activation
Degree
Forward-Backward Time
Hermite
0.000350997
Hermite
4
0.00035552
Hermite
0.00035341
Hermite
0.000355005
Hermite
32
0.000355887
Hermite
64
0.000795927
Hermite
128
0.00221812
Hermite
256
0.00794526
Hermite
512
0.0306861
Hermite
1024
0.122032
Hermite Rec. CUDA
0.000887356
Hermite Rec. CUDA
0,000887024 4
Hermite Rec. CUDA
0,000893834 8
Hermite Rec. CUDA
0.000912833 16
Hermite Rec. CUDA
0.000956004
Hermite Rec. CUDA
64
0,00100782
Hermite Rec. CUDA
128
0,001 10659
Hermite Rec. CUDA
256
0.0017534
Hermite Rec. CUDA
512
0.00233119 19
Hermite Rec. CUDA
1024
0.00555244
Hermite Rec. CUDA
2048
0.0119172
Tropical
0.000296884
Tropical
4
0.000295298
Tropical
0.000295942 8
Tropical
0.000296063
Tropical
32
0,000302484
Tropical
64
0.000303783
Tropical
128
0.000304883
Tropical
256
0.000305769
Tropical
512
0.000305247
Tropical
1024
0.000306931
Tropical
2048
0.000375793
Fourier
0,000470023
Fourier
4
0.0004714B0 444
Fourier
0.000478096
Fourier
0.00047 1656
Fourier
32
0.000489211
Fourier
64
0.000487039
Fourier
128
0.000486042
Fourier
256
0.00048811
Fourier
512
0.000489757
Fourier
1024
0.000666652
Fourier
2048
0.00125729
39

---

Published as a conference paper at ICLR 2026
Table 11: Training times (in seconds) and relative slowdowns compared to GELU across different MLP network widths and depths, for Hermite, Tropical, and Fourier activations. The reported times were averaged per epoch and were obtained using single NVIDIA A100 GPU with 40 GB of memory.
Activation
Degree
Width
Depth
Training Time
Slowdown vs Baseline
GELU
1024
16.13s
1.00x
Hermite
3
1024
8
16.03s
0.99x
Tropical
6
1024
8
13.27s
0.82x
Fourier
1024
13.93s
0.86x
GELU
1024
16
12.86s
1.00x
Hermite
3
1024
16.15s
1.26x
Tropical
1024
16
14.06s
1,09x
Fourier
6
1024
16
15.13s
1.18x
GELU
1024
32
13.96s
1.00x
Hermite
3
1024
32
20.65s
1.48x
Tropical
6
1024
32
16.29s
1.17x
Fourier
1024
18.50s
1.33x
GELU
2048
12.36s
1.00x
Hermite
3
2048
14.08s
1.14x
Tropical
6
2048
12.98s
1.05x
Fourier
2048 6
13.51s
1.09x
GELU
2048
16
12.96s
1.00x
Hermite
3
2048
16
16.29s
1.26x
Tropical
2048
16
14.05s
1.08x
Fourier
2048
15.15s
1.17x
GELU
2048
13.98s
1.00x
Hermite
3
2048
32
20.65s
1,48x
Tropical
6
2048
32
16.64s
1.19x
Fourier
2048 6
32
18.53s
1.33x
GELU
4096
8
12.43s
1.00x
Hermite
3
4096
14.12s
1.14x
Tropical
6
4096
13.02s
1.05x
Fourier
4096 6
13,58s
1.09x
GELU
4096
13.10s
1.00x
Hermite
3
4096
16
17.07s
1.30x
Tropical
6
4096
16
14.30s
1.09x
Fourier
4096 6
16
15.42s
1.18x
GELU
4096
32
23.93s
1.00x
Hermite
3
4096
33.41s
1.40x
Tropical
4096
26.10s
1.09x
Fourier
6
4096
32
28.21s
1.18x
40

---

Published as a conference paper at ICLR 2026
Q LARGE LANGUAGE MODEL USAGE DISCLOSURE
We used large language models to assist translating, rewording, and polishing the text for clarity and readability. The models were not used for idea generation, experiments, analysis, or contributions at the level of scientific authorship.
41


Reviews:
[{"review_id": "uxd4dwfuwl", "reviewer": "Reviewer_MYHC", "summary": "This paper explores the utilization of orthogonal polynomial activation functions in neural networks, specifically, first deriving the variance preserving initialization for Hermite, Fourier, and Tropical activation function, then conducting experiments on image classification and NLP tasks.", "questions": "Please see the 'Weaknesses' section.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well written, presented with a clear structure.\n2. The theorem-proof logic is clear and rigorious.\n3. The visualization helps explain the conclusion.", "weaknesses": "Despite the strengths, here are some weaknesses:\n1. The motivation is not clear, is it just an exploration on activations? And I am not sure if the proposed activation functions solve any existing problems. (Although I do know that not all innovative thought must solve something discrete, but I do suggest the author to refine this part.) \n2. Since the paper is not the first to design a new kind of activation function, even not the first to use orthogonal  polynomials, I am not sure what is the core innovation.\n3. The experiments show very little improvements when Hermite activation function is used. The tropical and Fourier activation function even have worse performance then GELU. These results restrict the value of the paper.\n4. More experiments should be conducted, like, more tasks, more models, and more benchmarks.\n5. In Proposition C.3, when computing the expectation of $F(x)^2$, the paper uses $\\int F^2(x) \\dfrac{e^{-x^2/2}}{\\sqrt{2\\pi}}dx$. I do not think adding the $\\dfrac{e^{-x^2/2}}{\\sqrt{2\\pi}}$ term is rigorious, although it is based on the definition of Hermite. This problem is common for orthogonal polynomials, like Legendre, Hermite, and Chebyshev polynomials, that when adding the orthogonal term, the derivation is much easier.\n6. Although the newly designed functions have negligible redundent parameters, it may cause low numerical stability. I am not sure how the authors resolve this problem.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EF8fHIARNh", "reviewer": "Reviewer_CZi7", "summary": "This manuscript introduces a family of learnable activation functions based on orthogonal function bases (Hermite and Fourier) and tropical polynomials. The authors propose a variance-preserving initialization scheme to ensure stable gradient propagation and demonstrate the feasibility of using these activations in deep architectures such as ConvNeXt and GPT-2. The paper combines theoretical analysis, efficient implementations, and empirical validation, suggesting that polynomial activations can indeed yield competitive results with proper initialization.", "questions": "1. The variance-preserving initialization ensures equal forward and backward gains for orthogonal activations. How sensitive is this balance to deviations from the assumed input distributions (e.g., non-Gaussian inputs during training)?\n2. When replacing activations in large pretrained models (e.g., GPT-2), how does initialization interact with layer normalization and residual scaling? Are any stability adjustments required?\n3. Have the authors explored methods to reduce computational cost, such as approximate polynomial evaluation (e.g., Chebyshev truncation, low-rank projection, or kernel-based approximation)? Could these reduce FLOPs while preserving stability?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper provides a rigorous variance-preserving initialization framework that unifies different activation families under an orthogonal function perspective. This is both mathematically elegant and practically meaningful.\n2. By addressing Hermite, Fourier, and tropical bases, the study gives a broad view of orthogonal and piecewise-linear activations, including insightful links to classical activations (ReLU, GELU).\n3. Experiments on ImageNet (ConvNeXt) and OpenWebText (GPT-2) convincingly demonstrate that the proposed activations can be trained stably and achieve comparable or slightly better performance than standard nonlinearities.\n4. The inclusion of recursive formulations, efficient kernels, and open-sourced code (torchortho) greatly improves the work’s reproducibility and potential impact.", "weaknesses": "1. The reported 30–90% slower training speed (Section 6) is significant. The paper would benefit from more detailed timing analyses and GPU utilization comparisons to quantify the trade-off between performance and efficiency.\n2. The experiments focus on classification and next-token prediction tasks. Additional ablations (e.g., fine-tuning, transfer learning, adversarial robustness) could help demonstrate broader applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L8yMf4fJ5P", "reviewer": "Reviewer_4XrB", "summary": "The authors introduce a novel framework to enable learnable activation functions in deep neural networks. In particular, they focus on functions based on orthogonal bases and tropical polynomials. An initialization method for the activation functions  is introduced and the results showcase improvements over static functions. The efficacy of the method is benchmarked across vision and language tasks.", "questions": "I would appreciate if the authors address the issues raised in the weaknesses section.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The main idea is novel and well-motivated.\n\n- The thorough theoretical support on the initialization methods is a valuable contribution to the community.\n\n- I appreciate the benchmarking of the method across both text and vision tasks.\n\n- The latency analysis is an important addition.", "weaknesses": "- I am missing an ablation over different backbones for both vision and language benchmarks.\n\n- Although not a major weakness, additional experimental support on challenging benchmarks would increase the impact of the paper, e.g., on COCO for vision related tasks.\n\n- A discussion on the application of the proposed activation functions for generative models (e.g., diffusion-based models) would be interesting.\n\nMinor:\n\n- The last sentence in ln. 485 seems to end abruptly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JbhjlSXERz", "reviewer": "Reviewer_3rgG", "summary": "This work reopens the discussion on which functions can be used in deep neural networks (or, more specifically, whether a few of the highlighted functions in this paper can). The paper covers theory and some empirical results. Chiefly, perhaps, is empirical work showing ConvNeXt and GPT-2 can be trained using orthogonal learnable activations, at least on specific shown datasets, which \"eliminates the need for additional mechanisms\".", "questions": "- Why do you choose Hermite, Fourier, and Tropic activations? Are there other possibilities? What main problems with existing methods to these overcome?\n- Are there assumptions regarding distributions that aren't reasonable to expect, and to what extent do you need to check for them? E.g., are Gaussian or uniform distributions assumed by these activation functions, and would that be realistic?\n- The Fourier is described in sine-cosine form, but the recommended initialization seems to only make use of a_k? Which coefficients are actually initialized and trained?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- the theoretical underpinnings of Sec 3 are good -- 3.1 leads naturally to 3.2, 3.3, and 3.4. I may have missed _why_ Hermite, Fourier, Tropic activations are focused on. These families are unified cleanly.\n- The practical implementation in Sec 3.5 seems like it's reproducible. At least, no obvious flaws with that secion was found.", "weaknesses": "- It's not *technically* a weakness, but convention seems to be moving away from older datasets like CIFAR-10, and older models like GPT-2. This does not sway the decision much, but it seems to upset 'the community' and it does limit the generalizability of your claims, especially to larger scales.\n- This is perhaps more of a question, but I'm flagging it as a weakness because lack of clarity on this seems to undermine one of the main points -- that exploding/vanishing gradients are addressed. I.e., it seems that equal-gain guarantees are only guaranteed at the initialization? What evidence is provided for stability long-term?\n- Not exactly a weakness either (to the extent to which it just describes an observation!) but the activations (esp Hermite) seem to add a high computational overhead.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)