Deep Learning Alternatives Of The Kolmogorov Superposition Theorem
Authors: Leonardo Ferreira Guilhoto
Paris Perdikaris
Venue: ICLR 2025 Spotlight (2025)
Keywords: Kolmogorov-Arnold Representation Theorem
Function Approximation
Physics Informed Neural Networks
AI4Science
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: This paper introduces ActNet, a neural network architecture based on alternative KST formulations, overcoming limitations of KANs and showing improved performance in PINNs for simulating partial differential equations.

Abstract: This paper explores alternative formulations of the Kolmogorov Superposition Theorem (KST) as a foundation for neural network design. The original KST formulation, while mathematically elegant, presents practical challenges due to its limited insight into the structure of inner and outer functions and the large number of unknown variables it introduces. Kolmogorov-Arnold Networks (KANs) leverage KST for function approximation, but they have faced scrutiny due to mixed results compared to traditional multilayer perceptrons (MLPs) and practical limitations imposed by the original KST formulation. To address these issues, we introduce ActNet, a scalable deep learning model that builds on the KST and overcomes some of the drawbacks of Kolmogorov's original formulation. We evaluate ActNet in the context of Physics-Informed Neural Networks (PINNs), a framework well-suited for leveraging KST's strengths in low-dimensional function approximation, particularly for simulating partial differential equations (PDEs). In this challenging setting, where models must learn latent functions without direct measurements, ActNet consistently outperforms KANs across multiple benchmarks and is competitive against the current best MLP-based approaches. These results present ActNet as a promising new direction for KST-based deep learning applications, particularly in scientific computing and PDE simulation tasks.

Published as a conference paper at ICLR 2025
DEEP LEARNING ALTERNATIVES OF THE KOLMOGOROV SUPERPOSITION THEOREM
Leonardo Ferreira Guilhoto
Graduate Group in Applied Mathematics and Computational Science University of Pennsylvania
Philadelphia, PA 19104, USA
guilhoto@sas. wwwp:essssupeneee..m upenn.
Paris Perdikaris
Department of Mechanical Engineering & Applied Mechanics University of Pennsylvania Philadclphia. PA 19104, USA pgpêseas, upenn. edu
ABSTRACT
This paper explores alternative formulations of the Kolmogorov Superposition Theorem (KST) as foundation for neural network design. The original KST for- mulation, while mathematically elegant, presents practical challenges due its limited insight into the structure of inner and outer functions and the large number of unknown variables it introduces. Kolmogorov-Arnol. Networks (KANs) lever- age KST for function approximation, but they have faced scrutiny due to mixed results compared to traditional multilayer perceptrons (MLPs) and practical lim- itations imposed by the original KST formulation. To address these issues, we introduce ActNet, scalable deep learning model that builds on the KST and over- comes many of the drawbacks of Kolmogorov's 's original formulation. We evaluate ActNet in the context of Physics-Inforee Informel Neural Networks (PINNs), a frame- work well-suited for leveraging KST's strengths in low-dimensional function ap- proximation, particularly for simulating partial differential equations (PDEs). this challenging setting, where models must learn latent functions without direct measurements, ActNet consistently outperforms KANs across multiple bench- marks and competitive against the current best MI.P-based approaches. These results present ActNet as a promising new direction for KST-based deep learning applications, particularly in scientific computing and PDE simulation tasks.
INTRODUCTION
The Kolmogorov Superposition Theorem (KST) powerful and foundational result in mathe- matics, originally developed to solve Hilbert's 13th problem (Kolmogorov 196T). Over time, it has gained significant attention as tool for function approximation, showing that any multivariate continuous function can be represented exactly using finite sums and compositions of univariate functions. This theoretical insight has inspired numerous applications mathematics and computa- tional sciences since its inception in 1957, particularly in problems requiring efficient representation
of complex functions low-dimensional domains (Sprecher 1996 Kürková 1992 Köppen 2002)
Recently, Kolmogorov-Amold Networks (KANs) LLu al., 2024) have renewed interest in the practical use of KST, especially the context of physics-infone thysics-informed machine learning (PIML) (Krr- niadakis al. 2021). Several papers have been published this topic in the span of just few
months (Shukla et al., 2024 Wang et al.) 2024c Howard et al.) 2024: Koenig et al.] 2024 Rigas et al. 2024 Patra et al. 2024). demonstrating the current relevance of the KST for PIML research.
The KST's strength in approximating functions aligns well with problems in scientific computing and partial differential equation (PDE) simulation, where often encounter functions representing

---

Published as a conference paper at ICLR 2025
physical fields (e.g., velocity, temperature, pressure) that live in low-dimensional domains. Physics- Informed Neural Networks (PINNs) (Raissi et al. 2019 Wang et al. (023) have emerged as a popular deep learning framework for such problems, offering unique approach to learning latent functions without direct measurements by minimizing PDE residuals. By exploiting the KST, KANs offer mathematically grounded approach function approximation, sparking new research and development in the area of neural networks for numerically solving PDEs.
Despite its mathematical elegance, the original KST formulation is often impractical for real-world applications like PINNs, as the implementation of its univariate functions can be cumbersome Shukla et al. (2024); Yu et al.] (2024). As a result, alternative formulations of the KST may be better equipped to address the challenges faced in modern approximation tasks, motivating ongoing research into more adaptable and scalable methods.
Alternative formulations of KST, which we will collectively refer to Superposition Theorems, offer greater flexibility and more robust guarantees, making them better suited for modern deep learning applications, particularly scientific computing. These variants relax some of the original theorem's constraints, enabling more efficient computation and training of neural networks. Su- perposition Theorems open up new possibilities for designing deep learning architectures that can improve performance and scalability in challenging settings like PINNs.
The summary of contributions in this paper are as follows:
We argue for using alternative versions of the KST for building neural network architec- tures, as shown in Table [ Despite being the most popular iteration, Kolmogorov's original formula is the least suited for practical implementations.
We propose ActNet, a novel neural network architecture leveraging the representation the- orem from Laczkovich (2021) instead of the original KST employed in KANs. We prove ActNet's universal approximation properties using fixed depth and width, and propose a well-balanced initialization scheme, which assures that the activations of the network scale well with the size of the network, regardless of depth or width. Furthermore, ActNet does not have vanishing derivatives, making it ideal for physics-informed applications.
We evaluate ActNet in the context of PINNs, demonstrating its potential in setting that uniquely challenges current deep learning practices. Our physics-informed informed experiments showease ActNet's 's ability to learn latent functions by minimizing PDE residuals, task that aligns well with KST's strengths. ActNet outperforms KANs in every single experiment we examined, and appears to be competitive against the current best MLP-based approaches, including advanced methods such as PirateNets (Wang et al., 2024a).
Taken together, these contributions not only demonstrate the potential of ActNet as a novel architec- ture for tackling challenging problems in scientific computing and PDE simulation, but also open up a broader question: which formulation of the KST is most suitable for deep learning applications? While ActNet, leveraging Laczkovich's 's theorem (Laczkovich) 202T). shows promising results, it represents just a first step in exploring this rich space of possibilities. By addressing key limita- tions of existing KST-based approaches while maintaining their theoretical strengths, work with ActNet aims to pave the way for future research into optimal KST formulations for neural networks.
2 SUPERPOSITION THEOREMS FOR REPRESENTING COMPLEX FUNCTIONS
Proposed in 1957 by Kolmogorov (196I), the Kolmogorov Superposition Theorem (KST. also known Kolmogorov-Amoen representation theorem) offers powerful theoretical framework for representing complex multivariate functions using sums and compositions of simpler functions of a single variable. This decomposition potentially simplifies the challenge of working with multivariate functions, breaking them down into manageable components.
In its original formulation, the KST states that any continuous function f : 10, 1]d - R can be represented exactly as
2d
d
f(z1..... .......,,))) =
Σe py
Σ
(1)
9=0
p=1
2

---

Published as a conference paper at ICLR 2025
Superposition formulas for a continuous function f(z1,..., za). Kolmorogov's original result lead to many different versions of superpositions based on univariate functions. Although KANs employ the original formulation, other versions suggest distinct architectures. Other than the underlying formula, different theorems also use different assumptions about the domain of ʃ and regularity conditions of the inner/outter functions.
Table 1:
Inner Functions
Outer Functions
Other
Version
Formula
Parameters
Kolmogorov
2d p
(>n (ap(ip))
2d
2d
N/A
(1957)
Lorentz
Ex=0g 9
Epp {p0)(Ep))
2d
1
A E Rd
(1962)
Sprecher
Eg=0 I1
Ep=z Ap + ga))
1
2d
R, X E Rdd
(1965)
Laczkovich (2021)
E (Ep=) (pp))(0p))
m ((d)
1
gdxm
where bg R + I and 04p [0, 1] + R are univariate continuous functions known as outer functions and inner functions, respectively.
The first version of the KST was created to address Hilbert's 13th problem (Hilbert 2000) on the representability of solutions to 7th-degree equations via continuous functions of at most two vari- ables. Since its inception in the mid-20th century, the theorem has inspired numerous connections to approximation theory and neural networks, due its potential for studying multivariate functions.
Over time, a multitude of different versions of KST have emerged, each with slightly different formulations and conditions, reflecting the evolving attempts to adapt the theorem to different ap- plications and mathematical frameworks. Some of the most important variants are outlined in Table I For more information, we recommend the excellent book written by Sprecher (2017). who has worked on KST related problems for the past half century.
KST And Neural Networks. There is substantial body of literature connecting the Kolmogorov Superposition Theorem (KST) to neural networks (Lin & Unbehauen
1993 Köppen| 2002). notably
pioneered by Hecht-Nielsen in the late 1980s (Hecht-Nielsen 1987). KST seen as promising for providing exact representations of functions, unlike approximation- based methods such as those of Cybbnkko (1989). More recently, Kolmogorov-Armol Networks (KANs) (Liu et al. 2024) were proposed, attracting attention as a potential alternative to MLPs. Despite the initial enthusiasm, KANs have faced scrutiny due to mixed empirical performance and questions about experimental fairness (Yu et al. 2024 Shukla et al. 2024).
As detailed in Table [ the KST exists in multiple versions, each with unique conditions and for- mulations, offering significant flexibility over how functions can be decomposed and represented. These KST variants provide theoretical foundation for designing new neural network architectures, rooted in well-established function approximation theory. Despite serving as the basis for KANs, the original formulation of the KST in equation (1) offers the least information about the structure of the superposition formula. As can be seen in Table I Kolmogorov's formulation is in fact, the only one with (dd²) unknown functions that have to be inferred, whereas other formulations scale linearly with the dimension d. This discrepancy motivates the use of other formulas from Table in order to design neural network architectures based on superposition theorems.
Kolmogorov's Theorem Can Be Useful, Despite Its Limitations. Previous efforts to apply KST in function approximation have had mixed results (Sprecher) 1996. 2013 Köppen 2002) Kürková, 1992) as KST's non-constructive nature and irregularity of the inner functions (c.g., non- differentiability) can pose significant challenges. Vitushkin (1944) even proved that there exists certain analytical functions cannot be represented by r times continuously differentiable functions with less than d variables, suggesting that in certain cases, the irregularity of inner functions is unavoidable.
[o, 1]d -> IR that are times continuously differentiable which
The limitations of Kas-based representations seem to be at direct odds with the many attempts over the past 50 years to use it the basis for computational algorithms. The primary argument
3

---

Published as a conference paper at ICLR 2025
against the use of KST for computation is the provable pathological behavior of the inner functions, as detailed in Vitushkin (1964). However, this impediment may not be directly relevant for most applications for a few different reasons.
The pathological behavior of inner functions mainly applies to exact representations, not approximations, which are more relevant in machine learning (Kürková 1992).
Approximation vs. Exact Representation:
2. Function-Specife Inner Functions: Instead of using the same inner functions for all func- tions as suggested by most superposition theorems, allowing them to vary for specific target functions might address the irregularity problem in some cases.
3. Deeper Compasitions: Deep learning allows for more layers of composition, which could help alleviate the non-smoothness approximation tasks
While further research is needed to verify these claims, they offer practical insights that motivate continued exploration, including the work presented this paper.
ACTNET KOLMOGOROV INSPIRED ARCHITECTURE
With renewed interest in KST and deep learning, many open questions remain about its applicability, limitations, and effective integration. light of these questions, we propose ActNet, a deep learning architecture inspired by the Laczkovich (202T) version of the KST. This formulation implies several differences from KANs and other similar methods. The basic building block of an ActNet is an ActLayer, whose formulation is derived from the KST and can additionally be thought of as a multi- head MLP layer, where each head has a trainable activation function (hence the name ActNer).
3.1 THRORETICAL MOTIVATION
Theorem 1.2 in Laczkovich 2021) presents the following version of Kolmogorov's superposition theorem, which is the basis of the ActNet architecture:
Theorem 3.1. Let C(R²) denote the set of continuous functions from R& - R and > (2 + (2) 1) be integer. There exists positive constants Xig > 0, j. 1,..., d; m continuous increasing functions o, E C(R),, = I,..., m such that for every bounded function f C(RR), there is continuous function C(R) such that
1,..,, and
f(z ..)
Σ9
Σ Aushb(ts))
1=1
Using vector notation, this can equivalently be written as
m [(x) Σ (A, (x())
(2)
i=1 where the o; are applied element-wize and X; = (Ari...., dsa) E Rd.
As corollary, we have that this expressive power is maintained if we allow the entries of À to be any number IR, instead strictly positive, and do not restrict the inner functions to be increasing Furthermore, if we define the matrices (x()); ex(xr, and A.j == Xij and let S Ra l be the function that returns row sums of a matrix, or, if the input a vector, the sum of entries, then equation 2 can be cquivalently expressed
m
f(x)
Σ g (2: (x()))
(3)
9 ((S(A xbx))))
(4)
= S (g IS(A O (z))))
(5)
'n earlier version of ActNet attempted impose these constraints to À and ø, but this approach did not lead any practical gains, and potentially limited the expressive power of the network.
4

---

Published as a conference paper at ICLR 2025
where @ is the Haddamard product between matrices.
From the formulations described in Table I we believe this one from Laczkovich (2021) is the best-suited for practical applications in deep learning. Some of the reasons for this
This version of the theorem is valid across all of R as opposed to being only applicable the unit cube [0, 1]4, as is the case for most superposition theorems.
It presents flexible width size m, which yields exact representation long
(2 + (2d 1). which scales linearly with the dimension d.
It requires (d)) inner functions, as opposed to (dd²) in the case of KANs.
This formulation has other attractive interpretations in the light of recent deep learning advances, detailed in section 3.4
3.2 ACTNET FORMULATION
The main idea behind the ActNet architecture is implementing and generalizing the structure of the inner functions implied by theorem 3.1 That is, we create a computational unit called an ActLayer, which implements the forward pass ((A ((x)), and then compose several of these units sequen- tially. This done by parametrizing the inner functions .1,..., of basis functions.
......., Pm as linear compositions a set
IR Nxd return the basis expansion matrix defined by B(x), bx()) The forward pass of the Actlaayer component given an input Rd is then defined
Given a set of univariate basis functions b by....,b by R R. let B
Rd
ActLayyerg.A (a) (A BB(x))
where the output vector in R and the trainable parameters are 3 E R and E Rx The graphical representation of this formula be seen in Figure Under this formulation, say that the matrix ((x) = (x) E mxd where ((r); = (I(z, represent the inner function expansion of the layer and write
(x) (#1( pm (x)) (Bx)),
where O& : IR + R are real-valued functions that are applied element-wise, taking the form
N
(t) Σ (1)b
(8)
Thus, the htth entry an ActLayer output can be written as
(ActLayer 9.a (x=)
Σ
Aus Σ
Bayby(or)
(9)
1=1
==1
Anidh( Ak. sxr)
(10)
where Ae.: = (AkI...., Ate) Idd and (x(x)) = (.. ERd
Additionally, before passing the input to ActLayers and before outputting the final result, ActNet uses linear projections from the latent dimension used for the ActLayers We have empirically found this improved the performance of ActNet some tasks by mixing the information of the different entries of æ. It is also possible to include additive bias parameter at the end of the forward pass of each ActLayer. visualization of the ActNet architecture can be seen in Figure I
Thus, the hyperparameters of ActNet are: the embedding dimension E N, which is used as the input and output dimension of the ActLayers; the number N of basis functions used at each ActLayer; and the amount L N of ActLayers. Such network has OLLm(m parameters. For comparison, a KAN with L layers of width m, spline order K and grid definition G has (Tmm² (G + K)) trainable parameters, which grows faster than ActNet. On the other hand,
N)) trainable
5

---

Published as a conference paper at ICLR 2025
ActNet times
ActLayer
Astivation Specifi Densp Leret
Ox
À +
Linear Projection
Linear Projection
Input
ActLayer
Output
A (x)
X
(ctNet(x)
Me (x)
Am (xx
vector in RR
scalar in B
vector in gd
layer
vector in @p
Figure 1: (left) Visual representation of individual Act ActLay The Actlayyer architecture be seen as a MultiHead MLP layer with tunable activations. (right) Visual representation of the ActNet architecture. The input vector is first projected an embedding dimension, then passed into composed blocks of ActLayer, and finally linearly projected into the desired output dimension.
an MLP of L layers of width has OLm²²) trainable parameters. Our experiments indicate that N can often be set as low as
basis functions, which case the number of parameters of an ActNet is roughly similar that of MLP of the same width and depth, whereas KANs typically use G > 100, implying a much faster rate of growth. A comparison of the number of trainable parameters and flops between ActNet, KAN and MLPs can be seen in Appendix B and Table
Another advantage of the ActLayer over a KAN layer is that of simplicity in implementa- tion. For example, using the einsum function found in packages like NumPy, PyTorch and JAX the forward pass of the ActLayer can even be implemented using single line of code as out=einsun; um (' ij, jk, ik->k' B(x), ))
3.3 UNIVERSALITY
Definition 3.2. Given a set S Rd. neural network architecture is called a universal approximator on if for every ʃ E C(Q) and every 6 > there exists a choice of hyperparameters and trainable parameters 0 such that the function So computed by this neural network satisfies
sup [(())(() fo(#)) <. sen
With this definition, we now state the universality of ActNet, with proof presented in Appendix
Theorem 3.3. A composition of two ActLayers is universal approximator for any compact set X C Rd. In other words, for any continuous function with compact domain f + R and any > 0, there exists composition of two ActLayers, where ff Act(x)) < for all x E X.
In the construction implied by this proof, the first ActLayer has output dimension m > (2 + (2d 1), while the second ActLayer has output dimension 1. This means that we obtain uni- versality using width size that scales linearly with the input dimension d, potentially at the cost of requiring a large number N of basis funcitons.
3.4 OTHER INTERPRETATION OF THE ACTLAYER
Although the representation formula from theorem 3.1 is the primary inspiration for the ActLayer, this unit can also be seen through other mathematically equivalent lenses. For example, the strategy of using several small units parallel, followed by concatenating their outputs is exactly the idea behind multi-head attention employed in the transformer architecture from Vaswani (2017). In the same sense, the ActLayer can seen as a Multi-Head MLP Layer where each head has different, tunable, activation function d;. In this analogy, each head first applies its individual activation function Φi, then applies a linear layer X; = Ai: with output dimension 1. Finally, the output of each head concatenated together before being passed to the next layer.
Alternatively, the ActLayer can also be seen as a lighter version of KAN layer, where each inner function &gg has the form ( (x) ) (x)). Assuming all inner functions parametrized by

---

Published as a conference paper at ICLR 2025
the same set of basis functions, this is then equivalent employing a KAN layer using Low-Rank Adaptation (LoRA) from Hu et al. (202I). This comparison however, overlooks the fact that the basis functions used for KAN cubic B-splines, whereas ActNet employs a sinusoidal basis by default, as described in section 3.5
All three interpretations mathematically equivalent, and showcase potential benefits of the Act- Layer through different perspectives.
3.5 CHOICE OF BASIS FUNCTIONS AND INITIALIZATION
Although ActNet can be implemented with any choice of basis functions, empirically we have ob- served robust performance using basis functions b by IR + IR of the form
(C) = sin(w,t + p.) - plw,, pt)
(11)
(w,,P:)
where frequencies w; ~ N(0, 1) are initialized randomly at each layer from a standard normal distribution and phases ps initialized at O. We also add small & 0 to the denominator in for numerical stability when a (w,, Pt) is small. In the equation above, the constants u(ar, p.)
and ((),.p.) are defined as the mean and standard deviation, respectively, of the variable Y sin (w; + Pt), when X ~ N(0, 1). These values are expressed in closed form as
plw;, pt) E e - sin(p:),
(12)
1 2
aWw,p:) = V Y - E|y
V
2n (2p:) puwe,, ps)
(13)
This ensures that, if the inputs to the ActNet are normally distributed with mean 0 and variance (a common assumption in deep learning), then the value of these basis functions will also have mean O and variance 1. After properly initializing the 33 and À parameters (detailed in Appendix (D3). the central limit theorem then tells us that the output of an ActLayer will roughly follow a standard normal initialization, which results in stable scaling of the depth and width of the network.
This can be formally stated as the theorem below, which implies that the activations of an ActNet will remain stable depth and width increase. The proof can be seen in Appendix F
Theorem 3.4. At initialization, if the input x Rdd is distributed as (O, Ia). then each entry of the output Act Layer (x) has mean 0 and variance I. In the limit as either the basis size N - co or the input dimension d + we get that each output is distributed as a standard normal.
EXPERIMENTS
We believe that the framework of Physics Informed neural networks (PINNs) Raassi et al.. 2019) provides an ideal testbed for evaluating ActNet and other KST-inspired architectures. PINNs align well with the strengths of the Kolmogorov Superposition Theorem in several key aspects:
1. -ow-dimeniional domains: PINNs often deal with functions representing physical fields (e.g.. velocity, temperature, pressure) that typically live in low-dimensional domains, where KST's function approximation capabilities have been better studied.
2. Derivative information:
PINNs leverage derivative information through PDE residuals, unique aspect that aligns with KST's ability to represent functions and their derivatives. In particular, as detailed in Appendix C the derivative of an ActNet is another ActNet, which tells us the higher order derivatives of the network will not vanish, frequent problem with ReLU MLPs and many of its variants.
3. Complex functions: PDEs often involve highly nonlinear and complex solutions, chal- lenging traditional neural networks but potentially benefiting from KST-based approaches.
4. Lack of direct supervision: Unlike traditional supervised learning. PINNs often learn la- tent functions without direct measurements, instead relying on minimizing PDE residuals. This indirect learning scenario presents a unique challenge that has posed difficulties to ex- isting architectures (Wang al. 2024a) and may benefit from KST-inspired architectures.
7

---

Published as a conference paper at ICLR 2025
Table 2: Best relative L2 errors and residual losses obtained by ablation experiments, as described
in section 4.1 More details can be seen in Appendix G
Relative L2 Error (4)
Residual Loss (4)
MLP
MLP
Benchmark
ActNet
KAN
Siren
ActNet
KAN
Siren
(base)
(base)
Poisson (w 16)
6.3e-4
1.8c-1
9.6e-2
6.6e-2
4.6e-3
2.6c0
8,9e-1
4.8c0
Poisson (w 32)
6.3e-2
1.1e0
1.8e-1
1.0e0
8.0e-1
1.4e+3
8.4e+1
6.2e+4
Helmholtz (w 16)
1.3e-3
2.8e-1
1.2e-1
4.6e-2
5.0e-3
2.6e0
9.2e-1
6.1e0
Helmholtz (w 32)
1.le-1
1.1e0
1.3e-1
1.0e0
5.7e-1
1.5e+3
8.1e+1
6.1e+4
Allen-Cahn
5.6e-5
5.3e-4
2.1e-5
1.1e-4
1.2e-8
3.8e-8
1.9e-8
2.0e-8
By focusing our experiments on PINNs, we not only aim to demonstrate ActNet's capabilities in a challenging and practically relevant setting, but also explore how KST-based approaches can con- tribute to scientific computing and PDE simulation. This choice of experimental framework allows us evaluate ActNet's performance scenarios that closely align with the theoretical strengths of the KST, potentially revealing insights that could extend to broader applications in the future.
Motivated by these observations, several papers (Shukla et al., 2024, Wang et al., 2024c Howard et al. 2024 Shuai & Li] 2024 Koenig et al. 2024, Rigas et al. 2024 Patra et al. (024) have been published attempting specifically exploit KANS for physics informed training. with varying degrees of success. In the experiments that follow, we demonstrate that ActNet consistently outper- forms KANs, both when using conducting our own experiments and when comparing against the published literature. Additionally, we also show that ActNet can be competitive against strong MLP baselines such as Siren (Sitzmann et al.] 2020), as well as state-of-the-ar art architectures for PINNs such the modified MLP from Wang al. (2023) and PirateNets from Wang et al. (2024a)
4.1 ABLATIONS STUDIES
We compare the effectiveness of ActNet, KANs and MLPs in minimizing PDE residuals. In order to focus on PDE residual minimization, we enforce all initial and boundary conditions exactly, follow- ing Sukumar & Srivastava (2022), which removed the necessity for loss weight balancing schemes (Wang et 202220, and thereby simplifies the comparisons. For all experiments, we train networks of varying parameter counte] and for each parameter size, with 12 different hyperparemeter con- figurations for each architecture. Additionally, for the sake of robustness, for each hyperparameter configuration run experiments using 3 different seeds and take the median result. This means that for each PDE, 144 experiments were carried out per architecture type. We do this as our best attempt at providing careful and thorough comparison between the methods.
Additionally, for the Poisson and Helmholtz PDEs, we consider different forcing terms, with frequency parameters E 2,, 4, 16, 32}. This means that for the Poisson PDE alone, 3,456 networks were trained independently, and similarly for Helmholtz. As the frequency parameter increases, solving the PDE becomes harder, the model needs to adapt to very oscillatory PDE solutions. This is intended to showcase how ActNet's expressive power and training stability directly translate performance in approximating highly oscillatory functions.
Table 2 summarizes the results of these ablations, where we find that ActNet outperforms KANs across all benchmarks, often by several orders of magnitude, and also outperforms both traditional MLPs as and strong MLP baselines like Siren (Sitzmann et al. 2020) on most cases.
Poisson Equation. We consider the 2D Poisson PDE on [-1, 1]2 with Dirichlet boundary and forcing terms which yield exact solution u(x, y) sin (in()rr)) sin(Twy) for w in 2, 4, 8, 16, 32), which are shown in Appendix G.3 as Figure 6) along with full description of the PDE. ActNet consistently outperforms the other two types of architectures for all frequencies difference in performance between methods becomes starker as the frequency w increases. For the highest two frequencies of w = 16 and as two orders of magnitude. The full results comparing ActNet, KANs, Siren and traditional MLPs can be seen the table in Figure of of the Appendix.
In particular, the
32, ActNet can outperform KAN and Siren by as much
2 [110k, 20k, 40k, 80k) for Poisson and Helmholtz. PDEs, and {25k, 50k, 100k) for Allen-Cahn.
8

---

Published as a conference paper at ICLR 2025
ActNett Prediction
Siren Pred ction
KAN Prediction
1.05
346
9325
000
-34e
-21
-033 -025
-C.0
-025
666
6.2
ActNet Absolute Error
Siren Absolute Error
KAN Absolute Error
119
115
315
922
020
3082
+1.
Figure 2: Example predictions for the Helmholtz equation using w = 16. The relative L2 errors for the ActNet, Siren and KAN solutions above are 1.04e-03, 8.82e-2 and 2.64e-1, respectively.
Table 3: Relative 1.2 errors of ActNet versus state-of-the-art results for PINNs.
JaxPi
PirateNet
Benchmark
ActNet
(Wang et al. 2023)
(Wang al. 2024a
Advection (c= 80)
9.50e-5 1.34e-5
6.88e-4 1.42c-4
5.48e-4 1.23e-5
Kuramoto-Sivashinss Sivashinsky (first time window)
Kuramoto-Sivashie Sivassissky (full solution)
8.53e-2
1.61e-1
2.
Inhomogeneous Helmholtz Equation. We consider the inhomogeneous Helmholtz PDE on [|, with Dirichlet boundary condition, which is fully described in Appendix G.4 Similarly to what was done for the Poisson problem, we set forcing terms so this PDE has exact solution 1.
u(x,yy sin(ww) sin(wy), which can be seen in Figure for different values of w and Once again, ActNet outperforms both KANs and MLPs, across all frequencies (full details can be seen in Figure in the Appendix), with a starker discrepancy for high values of
Allen-Cahn Equation On the Allen-Cahn PDE (fully described in Appendix G.5). ActNet outper- forms KANs by around one order of magnitude across all network sizes. ActNet and Siren perform comparably, with ActNet yielding better results for smaller network sizes, and Siren performing bet- ter for larger network sizes, can seen in Figure [3] However, for larger networks, the relative L2 error is close to what can be achieved with single-precision computation for neural networks, so it is possible that the discrepancy arises due floating- point error accumulation during training. Despite yielding slightly larger eтror in some cases, ActNet achieves the lowest final PDE residual loss across all network sizes compared to both Siren and KAN, as can seen in Figure 14
4.2 COMPARISONS AGAINST CURRENT STATE-OF-THE-ART
The goal of this section is to compare the performance of ActNet against some of the state-of-the-an results such as the modified MLP from the JaxPi library (Wang et al., (023) and PirateNets (Wang et al. 2024a). As be seen in Table 3 ActNet is capable of improving the best results available in the literature for two very challenging PINN problems.
Advection Equation. We consider the ID advection equation with periodic boundary conditions, which has been extensively studied in Wang et al. (003)); ; Daw et al. (2022); Krishnapriyan et al. initial conditions g( = sin(x) and the high transport velocity constant e = 80. This yields a challenging problem for PINNs, with highly oscillatory solution function. We compare ActNet against results
(222)) and is further described in Appendix G.6 Following Wang et al. (2023),

---

Published as a conference paper at ICLR 2025
R.WASS
Figure 3: ActNet predictions for the advection equation (c 80). The relative L2 error is 9.50e-5.
whereas the best result found in the literature is 6.88e-4 (Wang et al. 2023
Figure 4: ActNet predictions for the chaotic Kuramoto Juramoto-Sivashinsky PDE. The relative L2 error is 8.53e-2, whereas the best result found in the literature is 1.61e-1 (Wang et al. 2023).
from Wang et al. 2003). obtaining a relative L,2 error of 9.50 x 10-6, compared to their 6.88 x 10 result, see Figure On this same benchmark, PirateNet obtains a relative L2 error of 5.48 10-4,
Kuramoto-Sivashinsky Sivashinsky Equation. The SitamotooSSvvshinste equation (fully described in Ap- pendix G.7 is chaotic fourth-order non-linear PDE that models laminar flames. Due to its chaotic nature, slight inaccuracies in solutions quickly lead to diverging solutions. Training PINNs for this PDE usually requires splitting the time domain into smaller windows, sequentially solving dow at a time. Considering only the first window E [0, 0.1). ActNet obtains a relative L2 error an order of magnitude lower than what is reported in Wang
win-
(2023). For the full solution, ActNet also improves on the current best results from Wang et al. (002) and PirateNet, see Figure
CONCLUSION
Summary. We explore alternative formulations of the Kolmogorov Superposition Theorem (KST) develop neural network architectures, and introduce ActNet. This architecture is inspired by Laczkovich's 's version of KST and is faster and simpler than Kolmogorov-Amold Networks (KANs). We prove ActNet's universality for approximating multivariate functions and propose an initializa- tion scheme to maintain stable activations. In physics-informed experiments, ActNet consistently outperforms KANs and is competitive with leading MLP architectures, even surpassing state-of-the- art models such as as PirateNet and the modified MLP of Wang et al. (2023) in some benchmarks.
Limitations. The primary limitation of ActNet and other KST-based approaches to deep learning the slower computational speed compared to plain MIPs, although ActNet mitigates this to a con- siderable degree compared to KANs. Additionally, further optimizations may be possible, including hardware-level implementations tailored to ActNet's forward pass, similar to recent advances like FlashAutention for Transformers (Dao et al. 2022).
Future Work. Potential future directions in this field are determining the performance of ActNet and other KST-based architectures on data-driven problems, as well as studying the effect of replacing MLPs with these architectures inside larger neural networks such transformers and U-Nets. We believe further study into KST-based architectures and specifically ActNet have the potential to advance not only scientific machine learning applications, but deep learning whole.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
Our contributions enable advances deep learning architectures. This has the potential to signifi- cantly impact wide range of downstream applications. While we do not anticipate specific negative impacts from this work, as with any powerful predictive tool, there is potential misuse. We en- courage the research community to consider the ethical implications and potential dual-use scenarios when applying these technologies in sensitive domains.
REPRODUCIBILITY STATEMENT
Our entire code-base is publicly available online on GitHub. The code used to carry out ablation experiments can be found the GitHub repository https: 7/github. ctmc Predd www.eeed.ccveeennaaaaa.eeeeeeeee /ctt/tttNtt Our ActNet implementa- tion is also available in the open-source JaxPi framework at https: /ttps::/github com/ htwpliiggnneaaaaaaa/i/eeeeee/e comparisons against current state-of-the-art.
which we used to carry out the
AUTHOR CONTRIBUTIONS
L.F.G. conceived the methodology and conducted the experiments. P.P. provided funding and super- vised this study. All authors helped conceptualize experiments and reviewed the manuscript.
ACKNOWLEDGMENTS
We would like to acknowledge support from the US Department of Energy under the Advanced Scientific Computing Research program (grant DE-SC0024563) and the US National Science Foun- dation (NSF) Soft AE Research Traineeship (NRT) Program (NSF grant 2152205). We also thank Maryl Harris for helpful feedback when reviewing the writing of the manuscript and the develop- of software that enabled this research, including JAX (Bradbury et al. 2018). Flax (Heek al. 2023 Matplotlib (Hunter 2007) and NumPy (Harris et al. 2020).
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Rafal Jozefowicz, Yangqing Jia, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Mike Schuster, Rajat Monga, Sherry Moore, Derek Chris Olah, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaogiang Zheng. TensorFlow, Large-scale machine learning on heterogeneous systems, November 2015.
Murray,
Jason Ansel, Edward Yang. Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary De Elias Ellison, Will Feng, Jiong Gong, Michael Gschwind, Brian Hirsh, Sherlock Huang, Kshiteej Kalambarkar, Laurent Kirsch, Michael Lazos, Mario Lezcano, Yanbo Liang, Jason Liang. Yinghai Lu, CK Luk, Bert Maher, Yunjie Pan, Chris- tian Puhrsch, Matthias Reso, Mark Saroufim. Marcos Yukio Siraichi, Helen Suk, Michael Suo, Phil Tillet. Eikan Wang, Xiaodong Wang. William Wen, Shunting Zhang, Xu Zhao, Keren Zhou, Richard Zou, Ajit Mathews, Gregory Chanan, Peng Wu, and Soumith Chintala. Py PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation. In 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume (ASPLOS 24). ACM, April 2024. doi: 011145/366206553664033 URL https: /ttp::://yttcccchoo wwtp://tt..gaaaeaaeooooooa
assets httpettssppyocccccce 2. pd
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman- Milne, and Qiao Zhang. JAX: composable transformations Python+NumPy programs, 2018. URL http:
7 teu.thu...... cwm //oa////ooll/jaa/a/a/
11

---

Published as a conference paper at ICLR 2025
Andrew Brock, Soham De, Samuel L. Smith, and Karen Simonyan. Sigh-performance large- scale image recogmition without normalization. CoRR, abs/2102.06171, 2021. URL hetps:
0617
George Cybenko. Approximation by superpositions of sigmoidal function. Mathematics control, signals and systems, 2(4):303-314, 1989.
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory--fficient exact attention with io-awareness, 2022. URL https://arxiv..ogg/ /bs/ 2205.14153
Arka Daw, Jie Bu, Sifan Wang, Paris Perdikaris, and Anuj Karpatne. Rethinking the importance of sampling physics-informe informedd neural networks. arXiv preprint arXiv:2207.00 02338, 2022.
Craig Gin, Bethany Lusch, Steven L Brunton, and Nathan Kutz. Deep learning models for global coordinate transformations that linearise pdes. European Journal of Applied Mathematics, 32(3): 515-539, 2021.
Leonardo Ferreira Guilhoto and Paris Perdikaris. Composite bayesian optimization in function spaces using neon neural epistemic operator networks. Scientific Reports, 14(1):29199 2024.
Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Maraadd,arraaat,, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Ar- ray programming with NumPy. Nature, 585(7825):357-362, September 2020. doi: 10. 10310 /5558///411555 41486 020 2649-2
641586-020-2649-22 URL
Robert Hecht-Nielsen. Kolmogorov's mapping neural network existence theorem. In Proceedings of the international conference on Neural Networks, volume 3, pp. 11-14. IEEE press New York, NY, USA, 1987.
Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Marc van Zee. Flax: A neural network library and ecosystem for JAX, 2023. URL
http::/githe ctt/ http:/oooleeeEee flax
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus), 2020.
David Hilbert. Mathematical problems. Bulletin of the American Mathematical Society, 37(4): 407-436, 2000.
Amanda A. Howard, Bruno Jacob, Sarah H. Murphy, Alexander Heinlein, and Panos Stinis. Finite basis kolmogorov-amold networks: domain decomposition for data-driven and physics-informed problems, 2024. URL ltttps://axxii
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint
N11::2206.09685. 2021.
J. D. Hunter. Matplotib: A 2d graphics environment. Computing in Science & Engineering, 9()): 90-95, 2007. doi: 10.1 1..10///...2200.55.
George Em Karniadakis, Ioannis Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. Physics-informed machine learning. Nature Reviews Physics, 3(6):422-440, 2021.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
Georgios Kissas, Jacob H Seidman, Leonardo Ferreira Guilhoto, Victor M Preciado, George J Pap- pas, and Paris Perdikaris. Learning operators with coupled attention. The Journal of Machine Learning Research, 23(1):9636-966 9698, 2022.
12

---

Published as a conference paper at ICLR 2025
Benjamin C. Koenig, Suyong Kim, and Sili Deng. Kan-odes: Kolmogorov- arnol network ordinary differential equations for learning dynamical systems and hidden physics. Computer Methods in Applied Mechanics and Engineering, 432. 132:11722 117397, December 2024. ISSN 0045-7825. doi: 10. (016/j.cma.2024.. 1179.7 17397. StRl.ttt::/ / dx doi org 101 cma 1
Andrei Nikolaevich Kolmogorov. On the representation of continuous functions of several variables by superpositions of continuous functions of smaller number of variables. American Mathemat- ical Society, 1961.
Mario Köppen. On the training of kolmogorov network. In José R. Dorronsoro (ed.), Artificial Neural Networks ICANN 2002, pp. 474-479, Berlin, Heidelberg, 2002. Springer Berlin Hei- delberg. ISBN 978-3-540-46084-..
Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney. Char- acterizing possible failure modes in physics-informed neural networks, Advances in neural infor- mation processing systems, 34:26548-26566 2021.
Věra Kürková. Kolmogorov's theorem and multilayer neural networks. Neural networks, 5(3):
501-506, 1992.
Miklós Laczkovich. A superposition theorem of kolmogorov type for bounded continuous func- tions. Journal of Approximation Theory, 269:105609, 2021. ISSN 0021-9045. doi: httpss//doi. ja0.1001fija.2222..155//////// art 2xICCee/sL//002254552000777
URL httpp::/wwwt www.weeeieincceie.tccccc www/:ciicn/aaaeaeeee..c.c...e.ea
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Andrew Stuart, Kaushik Bhat- tacharya, and Anima Anandkumar. Multipole graph neural operator for parametric partial differ- ential equations. Advances in Neural Information Processing Systems, 33:6755-6766 2020.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations, 2021a.
Zongyi Li, Hongkai Zheng, Nikola Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Kamyar Arizzadenesheli, and Anima Anandkumar. Physics-informed neural operator for learning partial differential equations. ACM/JMS Journal of Data Science, 2021b.
Ji-Nan Lin and Rolf Unbehauen. On the Realization of Kolmogorov Network. Neural Com- 0ttp//eco..193.5.1.1 1993.5.1.18.
putation, 5(1):18-20, 01 1993. ISSN 0899-7667. doi: 10.1162/neco.
URL
orwp:/......1.... 1162/neco. 1993.5.1.
Dong C Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimization. Mathematical programming. 45(1):503-522, 1989.
Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruehle, James Halverson, Marin Soljačić, Thomas Y. Hou, and Max Tegmark. Kan: Kolmogorov-arnold networks, 2024.
Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3(3):218-22)) 229, mar 2021. doi: 10.1031-4425--0000000022 URL 021 00302-5
htttps://doi wtt.:/o..ogg....o. wwt.:r///............. 10w.10..113444F222222
Bethany Lusch, Nathan Kutz, and Steven I. Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nature communications, (1):4950, 2018.
Subhajit Patra, Sonali Panda, Bikram Keshari Parida, Mahima Arya, Kurt Jacobs, Denys I. Bondar, and Abhijit Sen. Physics informed kolmogorov- neural networks for dynamical analysis via efficent-kan and wan-kan, 2024. https:: wwwp:iivvvrrrgaaoo wwwp://gg/bbbbo......o... ab2 18373
M. Raissi, P. Perdikaris, and G.E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equa- tions. Journal of Computational Physics, 778:686-707,2011 ISSN 0021-999) doi: https://doi.
org/ 10/110.0161....p221100 10.045. URL https: wttp:://wwwwwwwweeeeeeeee..ee... com/ wtt.eecccc.ccccceceeeeeeeeeeecee science//////eece /ttccle//s11/50022999991111
13

---

Published as a conference paper at ICLR 2025
Spyros Rigas, Michalis Papachristou, Theofilos Papadopoulos, Fotios Anagmostopoulos, and Geor- gios Alexandridis. Adaptive training of gridddependent physics- informmed kolmogorov-armold net- works, 2024. /tttps:::// http:a::/aaxvvv.v.oo wttpa/.ww../gb///////0// .7611
Shyam Sankaran, Hanwen Wang, Leonardo Ferreira Guilhoto, and Paris Perdikaris. On the impact of larger batch size in the training of physics informed neural networks. In The Symbiosis of Deep Learning and Difserential Equations II, 2022. URL,
formmmmm?
id=THCvahg1RY
Hang Shuai and Fangxing Li. Physics-infomedd kolmogorov-amold networks for power system dynamics, 2024. URL
Khemraj Shukla, Juan Diego Toscano, Zhicheng Wang, Zongren Zou, and George Em Karniadakis. A comprehensive and fair comparison between mlp and kan representations for differential equa- tions and operator networks. Computer Methods in Applied Mechanics and Engineering. 431: 117290, 2024. ISSN 0045-7825. doi:
17290. URL https: //www. sci encedi rect. com htt.://cc.ccccceeeeeeaaaaee/////
Vincent Sitzmann, Julien N. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wet- zstein. Implicit neural representationn with periodic activation functions. CoRR, abs/2006.096611 2020.
09661
David Sprecher.
Kolmogorov superpositions: A new computational algorithm. In Efficiency and scalability methods for computational intellect, pp- 219-245. IGI Global, 2013.
David A Sprecher. A numerical implementation of kolmogorov's superpositions. Neural networks, 9()):655-772, 1996.
David A Sprecher. From Algebra to Computational Algorithms: Algorttt 13. Docent Press, 2017.
Kolmogorov and Hilbert's 's Problem
N. Sukumar and Ankit Srivastava. Exact imposition of boundary conditions with distance functions in physics-informed deep neural networks. Computer Methods in Applied Mechanics and Engi- neering, 389:1 14333, February 2022. ISSN 0045-7825, doi: 00.1016/j.cma.222) 2021. URL
http:///aa doi.
cma. 214333
Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Kein Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let net- works learn high frequency functions in low dimensional domains. NeurlPS, 2020.
A Vaswani. Attention all you need. Advances in Neural Information Processing Systems, 2017.
Anatoliy Georgievich Vitushkin. A proof of the existence of analytic functions of several variables not representable by linear superpositions of continuously differentiable functions of fewer vari- ables. In Dokl. Akad. Nauk SSSR, volume 356-1258-1268 1261, pp. 3, 1964.
Sifan Wang. Hanwen Wang, and Paris Perdikaris. Learning the solution operator of parametric par- tial differential equations with physics-informed deeponets. Science advances, e(40)). 2021.
Sifan Wang, Hanwen Wang, and Paris Perdikaris. Improved architectures and training algorithms for deep operator networks. Journal of Scientific Computing, 92(2):35, 2022a.
Sifan Wang, Xinling Yu, and Paris Perdikaris. When and why pinns fail to train: A neural tangent kernel perspective. Journal of Computational Physics, 449:110768, 2022b. ISSN 0021-9991. doi: http://w...rr/o................. http://ccennessrrrcccee/////////
URL hattps://wwe sciencedirect..
Sifan Wang, Shyam Sankaran, Hanwen Wang, and Paris Perdikaris. An expert's guide to training physics-informed neural networks. arXiv preprint 0886688, 2023.
Sifan Wang, Bowen Li, Yuhan Chen, and Paris Perdikaris. Piratenets: Physics- informed deep learn- ing with residual adaptive networks. arXiv preprint arXiv:2402.003326, 2024a.
14

---

Published as a conference paper at ICLR 2025
Sifan Wang, Shyam Sankaran, and Paris Perdikaris. Respecting causality for training physics-
informed neural networks. Computer Methods in Applied Mechanics and Engineering, 421: 116813, 2024b.
Yizheng Wang, Jia Sun, Jinshuai Bai, Cosmin Anitescu, Mohammad Sadegh Eshaghi, Xiaoying Zhuang, Timon Rabezuk, and Yinghua Liu. Kolmogorov amold informed neural network: A physics-informed deep learning framework for solving forward and inverse problems based on kolmogorov arnold networks, 2024c. URL httt:tts:://ett wrt.vv.iivv......i....... www.voogg/aaa..
11045
Runpeng Yu, Weihao Yu, and Xinchao Wang. Kan or mip: A fairer comparison, 2024. URL
https://arxiv wttp:iiww.gg///////////// 16674
15

---

Published as a conference paper at ICLR 2025
Appendix
A
MATHEMATICAL NOTATION
Table summarizes the symbols and notation used in this work.
Table 4: Summary of the symbols and notation used in this paper.
Symbol
Meaning
A vector x (Z1. ..)) a in Rd
((2)
The set of continuous functions from a set Ω to IR Outer function implied by a superposition theorem (see table 1 Inner functions implied by superposition theorem (see table Inner function expansion matrix with ij entry equal to @, (x,) Basis function from R to R
g
((x)
13
b,
B
Basis expansion matrix with ij entry equal to (x,)) E Matrix with ij entry equal to Xij E R Function that returns row sums of a matrix, or the sum of entries of a vector Hadamard (element-wise) product between vectors/matrices Parameters of a neural network model The identity matrix of size d Expectation operand
A
S
Ia
B COMPARISON ON PARAMETERS & FLOPS
From equation each of the = 1.... m outputs an ActLayer with input dimension d, output dimension m and N basis functions is defined as
(z
This entails (ddN operations per output. Since there outputs total, the computational complexity of this ActLayer is O(mdN). This means that this algorithmic complexity is a little slower than that of the traditional dense layer of an MLP (which is O(dm)), but around the same order of magnitude for low values of N. In our experiments, we have observed that the basis size N usually does not need to be large, with N usually yielding the best results. In particular, in all experiments where ActNet beats state-of-the-ar results, we do so by setting N = 4 basis function, as can seen in section 4.2 This setting makes an ActNet around =2-3 times slower than an MLP of the same size. To see this phenomena in practical terms, we direct the reader to table 9 where computational times for the Allen-Cahn PDE are compared for setting N {4, 8, 16).
In the case of a KAN layer, the algorithmic complexity is O( + K)) Yu et al. 2024 where is the grid size hyper-parameter, and K the spline order (usually taken to Unlike ActNet, however, typical values of G can range from around 5 to several hundred, with the original KAN paper (Liu et al. 2024p using experiments with as high as G 1000, which makes the architecture impractical for realistic values of widths and depths. Once again, phenomena practical terms, we direct the reader to table 9 where we set = N in {4, 8, 16). This comparison showcases another advantage of ActNets over KANs.
3).
this
summary of the hyperparameters of ActNet, KAN and MLP, along with their implied parameter count and flops of forward pass can be seen in table
16

---

Published as a conference paper at ICLR 2025
Table Table comparing parameter and algorithmic complexity of ActNet, KAN and MLP. Since the typical value of N is low, ActNet performs at slower, but comparable speed to MLPs, whereas KANs become very slow as the value of G increases.
Architecture
Hyperparameters Parameter Count
FLOPs
depth L
ActNet
hidden dim. m basis size N
O(Lm(m + +
((L
depth I.
hidden dim.
KAN
O[Lm² K))
OLLm2 + K))
grid size G
spline order K depth L
MLP
O(Lm²)
O(Lm²)
hidden dim. m
C
THE DERIVATIVE OF AN ACTNET IS ANOTHER ACTNET
From equation
we get that the partial derivative of the kth output of an Actl with respect to
Ty is
N
a
Σ Aki
Σ
Bagby
(2r))
dm
dxi
i=1
j=1
Aka
Bkj Oxl
by(11)
N
Ap
B2)
(a))
where dt()) = (2)) is the derivative of the kth inner function @x - This formula then tells us that the Jacobian JAcitayer,.A (ar) of ActLayer is
j== (x)))
JActLayer, (x) = A @ d'(x),
(14)
where ('(x) E Rxd the matrix defined by (1(x); = (xy)) This formulation is precisely what compute in the forward pass ActLayer,, (x) S(A @ ((x)). with the caveat that this
Jacobian now matrix instead single vector.
If the basis functions b.,...,by by picked that their derivatives do not deteriorate, as is the with the sinusoidal basis proposed in section [3.]] then the universality result of theorem [3.3 tells us that a sufficiently large ActNet will be able to approximate derivatives up any desired precision e > O. This statement once again makes the case for the potential of ActNet for Physics Informed Machine Learning, where a network needs learn to approximate derivatives of function, instead of point values.
D ACTNET IMPLEMENTATION DETAILS
Our implementation of ActNet is carried out in Python using JAX (Bradbury et al., 2018) and Flax (Heek et al., 2023). although the architecture is also casily compatible with other deep learning frameworks such as PyTorch (Ansel et al. 2024) and TensorFlow Abadi et al. (2015). The code is available at
com wtt.aa/www..eeeeeeeeeeeeeeeeeeee
ActNet
D.1 FREQUENCY PARAMETER
In a manner analogous to what is done in Siren (Sitzmann et al. 2020) and random Fourier features Taneik al. 2020), have observed that ActNet benefits from having wo 0 parameter that
17

---

Published as a conference paper at ICLR 2025
Table 6: Initialization schemes for an ActLayer with input dimension d, output dimension m and N basis functions. Each entry of a parameter is sampled in an i... manner.
Trainable Parameter
Mean
Std. Dev. of Entries
Dimension
Distribution
of Entries
/ R=xN
Gaussian/uniform
A
Rwxd
0
Gaussian/uniform
va
3
RV
0
Gaussian
p
RW
0
0
constant
bias (optional)
I m
0
0
constant
multiplies the original input to the network. Using such a parameter better allows the architecture to create solutions that attain to specific frequency content, using higher values of wi for approx- imating highly oscillatory functions and smaller values for smoother ones. This parameter can be trainable, but we generally found it best to set it to fixed value, which may be problem dependent, much like the case with Siren and Fourier Features.
D.2 TRAINING FREQUENCIES AND PHASES
Additionally, it is possible to allow the frequencies wi and phases pt of that basis functions b; from equation (IT) to be trainable parameters themselves, making the basis functions each layer cus- tomizable to the problem at hand. However, the parameters wi and p; of the basis functions appear be very sensitive during training, and optimization via stochastic gradient-based methods some- times lead to unstable behavior. This issue, however, can be mitigated by using Adaptive Gradient Clipping (AGC) proposed in Brock et al. (202T). where each parameter unit has a different clip de- pending on the magnitude of the parameter vector. We empirically observe that hyperparameter of le works well for training basis functions of an ActNet.
D.3 INITIALIZATION SCHEME
The parameters of an ActLayer are initialized according table 6 As will be shown later in the proof of theorem 3.4 using these initializations and applying the central limit theorem then tells us that the output an ActLayer, as defined in (6), will be normally distributed with mean 0 and variance 1, as long as either N or d are sufficiently large. This same argument can then be applied inductively on the composed layers of the ActNet to get a statement on the stability of the statistics of activations throughout the entire network.
The 3 and A parameters can be initialized using either a Gaussian or uniform distribution. Having said that, for the sake of consistency use uniform distributions for initializing these parameters for all experiments in section 4
D.4 CHOICE OF HYPERPARAMETERS
Results from the ablation studies from section can also be used to gain information over reasonable hyperparameter choices for the ActNet architecture. A representative plot of the choice of hyper- parameters for the Allen-Cahn PDE can be seen in figure Overall, our the main "rulc-of-thumb" heuristics have learned are:
The basis size N does not need to be large for accurate results. In fact, except for small parameter counts and shallow depths, we don't 't see accuracy gains for increasing the value of N beyond 4 or so. At the same time, as detailed in appendix B. the computational com- plexity of the model increases for larger N (even when network size is fixed). Therefore, we recommend first setting the basis size to around N 4, and increasing this value only if needed.
As often the case for deep learning. we find that composing more layers and increasing network width generally improves performance by increasing network capacity. Under
18

---

Published as a conference paper at ICLR 2025
fixed network size, choosing to increase depth implies decrease in width, and vice versa. While choosing the right combination of depth/width is likely problem dependent (as the case with MLPs),
recommend setting the number of ActLayers in an ActNet at least 2 2. Not only do we empirically observe a large performance boost over using a single Lyee,, this is also the minimum required depth for the theoretical guarantee of universal approximation, as described by theorem 3.3
ActNict on Alle- Nien-Cann FDE
ActNict on Allon-Cahn PCE bass functions
Acthet on Allcn Cahnn PDE
N-4
N-16
119
10a2
10
depthe
depth-]
depth
10"4
depth..
dept.
depth
depth
depii
deptiin
depthwb
2014
10
10
40300
09200
00200
100000
42000
62002
82000 100000
40000
ED0CD
EDOCO 100000
Paramitee Counc
Forameter Count
Paramettr Court
Figure 5: ActNet performance (relative L2 eπrоr) the Allen-Cahn PDE under different hyperpa- rameter settings. After selecting network depth L and number of basis functions N, the width m of networks computed in order to satisfy the required parameter size. As such, for given network size, larger values of deph imply smaller widths, and vice-versa. The values plotted for each hyper- parameter configuration is the median from 3 runs using different seeds.
E PROOF OF THEOREM 3.3
Proof of theorem 3.3 By KST (theorem 3.1 we know that for any (2 + v2) 1), the function f IRd R has a representation of the form
f(x)
Σ g
Σ
Apppia (opp(((8))))
where g R. -> TR and Ø1,..., pm lR - IR are continuous univariate functions and Apg 0 are positive scalars.
Now, let Y : {Ex ( E x 1, ,,...,,77
,m 1.m..,,.., be the set of possible inputs to g. Since X is compact and each of the dy is continuous, this means that must be compact as well.
Now let e' e/m. Since g is continuous and its inputs confined to compact domain Y. it must be uniformly continuous. That is, there exists some ó > 0 such that for all 2 E Y, if |z y| 8 then [g(=) g(v)l <'/2.. Additionally, due to Weierstrass theorem we know that there exists a polynomial poly, R + R of degree Ng such that for any E Y we have that
(9(y) polyy(()) e//2. Therefore, we get that
lg(=) - poly(v)) = 1g(z) - g(v) + g(t) polyy (v)
s [g(2) g(v)l (g(u) poly, (v) < < e//2 e'.
WWeierstrass theorem states that polynomials are dense in C(X) for any compact X C R.
19

---

Published as a conference paper at ICLR 2025
Which implies that if 21, 21...., 2m and V1..... ym scalars such that 12g Mel 8 for all Leee m. then we have
m
m - _ 9=1
m
Σ 8=1
g(2g)
polyy (yg)
-
Σ (g(2)) 9=1
-polyg(yu)
ms' = E.
Therefore, if we can approximate the inner functions (t) using polynomials poly, (t) and make sure that Sp=1 (Tp) is at most o far from the approximation Σp Apppoly, (xpp) for all q 1,..., m, we can finish the proof of universality for ActNet.
To prove this final statement, first set M max Apglp 1....,,;; note that by Weierstrass' Theorem, for each 6, there exists a polynomial poly, of degree N₂ such that (0) polyy(z)] < for all 2 E Y. Thus, we have that
d: 9
1, 1..... m) and then
ZM
Σ Ap (2ip)
Σ p
(xpolyy((p))
S
App 10g(1p)) - (poly((1pp)
p
8
App a.
d
which completes the proof.
F PROOF OF STATISTICAL STABILITY OF ACTIVATIONS THROUGH LAYERS
Before proving theorem 3.4 we prove the following lemma, where we compute the formulas stated
in equations 112 13
Lemma F1. wet E R be fixed real numbers and X ~ N0, 1) be a random variable distributed as a standard normal. Then the random variable Y sin(w X p) has mean and variance as follows:
sin(p),
(15)
Var[Y] = 2
2w cos(2p) - -w² sin(p)²
(16)
Proof. First, note that for any > 0 and b E C we have that
000
az + bar
22
4n
(17)
20

---

Published as a conference paper at ICLR 2025
Now, using this fact and the complex representation sin(0) -i"
have that
21
(
Y
sin(wr + p)
dr
2T
ellwxt))
((wxtt)))
6 dr.
V2T
2i
-00
((wx+t)= - 4 E ((wnt))) de
2m
00
1 goe+hwotipgl dx
c -f-² pot--wa dx
21v 2m
1
e 1 +inz dx -
f 22² iwo da
2
2T
-00
!² 2T e - 2n
e e
2m
1
eee
me 4
2i
exp
sp
2i
sin(p),
which proves the claim for the expectation Y. As for the variance, we first compute
1
22 2
E
sin (wer + p)²
dx
22T
21(wx+t)) + 21(wx+p)
2 -22
E dx
2m
1
(2(ww+p)) e =4²
+ 2(wxw+p)) -4² -2--22 de
=²²
2
1
21(wx+ dx +
E 21lww++)) E - dx
2e - dx
4v
-00
1 12+22w++2pp dx
e -= -inx 22p dx. 22 2T
2
1
25p
000
e - -- dx + E
2ip
000
e
222--
2iw dx
+ 2
2
-00
1
2ip e
2T + c
2ip e
2 Vt
+
2
2
2ip + e
-2p
26p
- 2n²
+ 2
4
2nd (2s
c
+
2
2
2n²
cos (2p),
2
which means that
Var = 2| E2
e 2w
cos(2p) -u² sin(p)²
2 2
thus completing the proof.
21

---

Published as a conference paper at ICLR 2025
We now ready to prove the main result of this section.
Proof of theorem 3.4 The kth output of an ActLayer is computed as
N
(ActLayer,,, A (a))
==1
Therefore, by linearity of expectation and then lemma
we have
E (( (A
Aut Be (x,()
A SusBEj.
Since A and | are independent and all entries have mean O, this means that the kth output has mean As for the variance of we now compute
o = E (ActLayer, (
(18)
27
E
Aks (k)
(19)
AusBezby by (cra) Maa BassBabb ]
E
(20)
N
E (1t (us
(21)
1=1
E by
(22)
E [
E. [Gha]] EENN(O) [b,(s:)
(23)
=1
N
1
Σ =1
(24)
d
N
(25)
In the deduction above, to go from line 21 to line (2 we use fact that the expectation of products of independent variables is the product of expectations, then recall that E [b,(,)]] 0.
which allows us to cancel the terms where * a and j * b.
This proves the first part of the theorem (outputs have mean 0 and variance at initialization). Now to prove the second part (outputs are distributed normally as either or go to infinity), we note that from the deduction above, for any fixed = 1,...,d. d, the random variable X E R defined by
x
(z.
has mean 0 and variance 4. Therefore, by the central limit theorem, we have that
(ActLayer, a. (x))
Σ x
1=1
will be distributed as a standard normal N 1) as d + swapping the role of and N, which finishes the proof for the theorem.
co. An analogous argument can be made,
22

---

Published as a conference paper at ICLR 2025
G EXPERIMENTAL DETAILS
We dedicate the remaining of the appendix for detailing the experimental setup used to produce the results indicated in section 4 this has been shown to improve training in PINNs (Sankaran et al. (2022), and enforced boundary conditions exactly when possible, following the strategy in Sukumar & Srivastava (2022).
As a general rule, used large batch sizes whenever possible, as
We would also like to highlight that while we focus our experiments on the data-free paradigm of Physics Informed Neural Networks, other methods exist for learning solutions of PDEs from data
(Gin ct al. 2021 Lusch et al. 2018). Notably, methods in the field of operator learning such as (Li al. 202 Lu al. 2021 Kissas al.] 2022] Li al., 2020 Wang al. 2022a Guilhoto & Perdikaris 2024) allow for predicting entire families of PDEs using single model, and may even be trained with the aid of physics informed loss (Li al. 2022b] Wang et al. [02T]. While we
used PINNs as a first testbed for ActNet, we believe the architecture has potential application to
data-driven methods as well, and leave further exploration as future research.
G.1 KAN IMPLEMENTATION DETAILS
All experiments were conducted in Python using JAX (Bradbury et al. 2018). Since the original implementation of KANs is not GPU-enabled and therefore very slow, we instead reference the "Efficient KAN" code found in intttps:
cww. weweeeaaaaneeeaiiiiiaaeeeeeeaaae
qit
Since Efficient KANs were implemented originally in PyTorch (Ansel et al. 20024), we translated the code to Flax (Heek et al. 2023). This implementation can be found at https: hpps://go h/ggtt/gtttttb/.
G.2 HYPERPARAMETER ABLATION FOR POISSON, HELMHOLTZ AND ALLEN-CAHN PDES
For every architecture, we disregard the parameters used in the first and last layers, as the input di- mension of 2 and output dimension of I make them negligible compared to the number of parameters used in the intermediate layers. For all architectures and problems, we consider {1, 2, 4, 6} inter- mediate layers (not considering first and last layers), well as three different architecture-speciff hyperparameters, as detailed below. All model widths were then inferred to satisty the model size at hand. This results in each architecture having 12 possible hyperparameter configurations for each parameter size (4 for depth, times 3 for architecture-specffi byperparameter).
Poisson & Helmholtz: For ActNet, use {8, 16, 32} values of N and for KAN grid resolutions of {3, 10, 30}. For Siren we consider wp in { kw, 3TW). For MLP we consider activations in (tanh, i gmo id GELU), where GELU the Gaussian Error Linear Unit from (Hendrycks & Gimpel 2020).
84,8, 16) values of N and for KAN grid resolutions of (4, 8, 16). For Siren we consider wo in {10, 30, 903. For MLP we consider activations in (tanh, si sigmiiid GELUJ, where GELU is the Gaussian Error Linear Unit from (Hendrycks & Gim- pel 2020
Allen-Cahn: For ActNet,
G.3 POISSON
We consider the 2D Poisson PDE with zero Dirichlet boundary condition, defined by
Au(z, y) = f(r v), u( v) = 0, We use forcing terms of the form f(x y) 2x sin((wx)) sin(Twyy) for values of w in {1, 2, 4, 8, 16). Each of these PDEs has exact solution u(x, sin(Twr)) sin(wwyy), which are shown in 6
(x. v) E [-1, 1]2,
(x. 81-1,11
Each model was trained using Adam Kingma & Ba, 2017 for 30,000 iterations. then fine tuned using LBFGS Liu & Nocedal (1889) for 100 iterations. We use batch size of 5,000 points uni- 0, 1 at each step. For training using Adam, use learning rate warmup from 10 to 5 x 10-3 over 1,000 iterations, then exponential decay with rate 0.75 every 1,000 steps, and adaptive gradient clipping with parameter 0.01 descrihed in Brock et al. 2021
formly sampled at random
23

---

Published as a conference paper at ICLR 2025
Furthermore, avoid unfairness from different scaling between the residual loss and the boundary loss, we enforce the boundary conditions exactly for all problems by multiplying the output of the neural network by the factor (1 x1 y²). in strategy similar to what is outlined in Sukumar & Srivastava (022)..
The final relative L.2 errors be in figure while the final residual losses be seen in figure An example solution is plotted in figure and sample computational times are reported in table
Target Function
y)x, = sin(swo)ssin((mme)))
300
180
8.5
06
525
0/0
-1.5
-0.5
-0.0
-08
-255
-10
65
-25
-.6
1.0
8.59
6M
-1.5
-000
Figure 6: Visualization of the different target functions for the Poisson and Helmholtz PDEs. As the frequency us increases, the magnitude of gradients become larger and the laplacian Au(a, y) = 2x² sin (x)) sin(wy) = 2n 2 y) scales x w thus creating challenging problem for PINNs.
Siren Prediction
60.
530
400
-205
Figure 7: Example predictions for the Poisson equation using 32. The relative L2 errors for the ActNet, and Siren solutions above are 6.42e-02, 1.91e-1, respectively. Predictions for KANs are not plotted here, they did not converge for this example.
G.4 HELMHOLTZ.
The inhomogeneous Helmholtz. PDE with zero Dirichlet boundary condition is defined by Au(z, y) + u2x( v) = f(x,y.)
(z.v) E [-1,
u(z, v) O,
we set 6 and the forcing term to be f v) = (R 2 sin sin(rWy), then this PDE has exact solution u(x; y) sin(war) sin sin(wy), which is pictured in figure
24

---

Published as a conference paper at ICLR 2025
Poisson 2D At Different Frequencies (Relative L2 Error Lower is betier)
3.05
5.8e-
2x 2r.2.355
1ibe4
3
5
9.165
23665
3.4e.
23e4
1il4
1805
3.6e
9.2e55
1.5e5
5.2e5
52c-4
3.te4
3.5e4
1.1e4
3.2-4
2
7.
.400
7.le.5
10-1
4.6e-4
1.9e2
8.703
1.2o2
1.7e4
5.36-3
K.764
5.5e-1
1.4e4
2.5e-3
2.6e-4
2.5e-3
Lile4
20c-3
2.0-4
I.le3
10"
26- 2.4e2
100+0
1-4-1
10K+00
6.7m3
1:3+00
9ibe-2
1.0+00
1.0e-3
3.0e-1
1-5-1
4.0c-1
1.5e1
1:le+0
Iettl
3.ee00
6.9e1
1,0400
2.tr1
2
6.5e22
Li---
1.0++t
15-
bann toreo
Nown Boos
Nap 1o/00
Arter trera 20a8
Siee 20.e0
MB MEr
Arter dwary Miv
Simo 00000
Antie frem Reldo
Jesa Rouee
E-st
EdM
Mi
Rat
Archteetur
Figure 8: Results for the 2D Poisson problem using PINNs. For each hyperparameter configuration, 3 different seeds were used for initialization, and the median result was used. For each square, best hyperparameter configuration (according to the median) reported. The best performing method for each frequency and each number of parameters is underlined.
Poisson 2D At Different Prequencies (Final Residual Loss Lower is better)
6.2e
9.7c
5.9e
.33eM
2.4e
6.0e
3-4e 9 2lce
1.8c
5.1e
Life
..Se.
10
3.76-8
1.le
7.2e
1.2-7
2.066
1
4 2:466
5.3e-6
Lite-
Lics
1.Je44
300
3.2e5
4.te7
2.1e-
1.5e-6
8.3e-6
2.8e=7
1.4e-1
8.5--7
3.8c
5.0c-4
Lie-1
3.20-3
2.764
1.4-2
3.36.
3.0:-2
1.5e-2
3
1.0r-1
1:2++2
124e-2
2.he=1
3.3+40
2.8e+1
1.4e22
1,9660
2.20+2
4.te3
KMh-1
Ne+4
32 7.0+1
4.0c14
1.3133
1.5e14 L.eel
2lee4
3.3e2
6.76r4
A.v10
5 5
1.3x2
6.erd
5.fel
LAc+3
8.4e1
622c4
Avoiel tren 1tut0
RAAN maad
siow laaee
MLR 10.00
Arter feoss 20a0
Kes Saoee
Nim 2un
Autia (eary Ataee
Stea daom
Mt.2 40.000
Autiet $500
KiN aute
siews Hudeo
Mix sapp
Adds
Aedhtettre 4 panmeners
Figure 9: Final residual loss for the Poisson problem using PINNs. For each hyperparameter figuration, different seeds were used for initialization, and the median result was used. For each square, the best hyperparameter configuration (according the median) is reported. The best per- forming method for each number of parameters is underlined.
Each model was trained using Adam (Kingma & Ba, 2017) for 30,000 iterations, then fine tuned using LBFGS Liu & Nocedal (1989) for 100 iterations. We use batch size of 5,000 points uni- formly sampled at random on 0, 1J each step. For training using Adam, we use learning rate warmup from 10-7 to 5 x 10-3 over 1,000 iterations, then exponential decay with rate 0.75 every
Table 7: Average computational time per Adam training iteration for the Poisson problem on Nvidía RTX A6000 GPU. All times are reported in milliseconds. Each cell averages across three seeds and four different widths and depths hyperparameters as reported on G.2
Model Size
ActNet
ActNet
ActNet (N
KAN
KAN (G
KAN
Siren
(N =8) 8)
(N = 16)
-2)
(G' = 3)
10)
(G
30)
10k
5.10
7.17
9.59
6.87
10.7
13.0
1.32
20k
7.37
10.8
15.3
10.3
16.3
18.5
1.66
40k
10.7
16.7
26.2
17.1
25.1
25.6
2.49
80k
16.2
26.5
43.4
23.6
40.0
37.4
3.60
25

---

Published as a conference paper at ICLR 2025
Helmholtz 2D At Different Frequencies (Relative L2 Error Lower better)
2.5e5
Lited
2.0e
Ales
231-8
1.le4
4.0-5
12od
1-b44
2.0-5
13c.
Lioss
1.6ed
1.5c4
1.1e4
1805
39p-5
respary
7.3-4
1.2e4
4.7e-
M44-
7.0e5
2.7e41
..de-
3.0e4
7.9e-
2.1e-4
J.les
2.0e-4
0.6e
Lie-4
5.1c-4
3.5e-2
1.2e2
Liba
1.9-3
1.5e3
1.8e-
1ifc4
26- 2.2m22
1.5e1
1,0+0
4.2e3
1ite00
L.tel
1ie+00
1.6e-8
4.8e-1
1.90-1
2.8o-1
1:3e-1
4A--2
3.5el
7.le1
25c1
3.4-11
L00+00
Liel
2.0+00
IJee0
1.3-
Arthal tveny Huat
KAs Hoads
Mloa stzee
Mts Iean
RAs 3000
Jobia teeny Atae
Skrw miam
Aothl comes Staao
RiN aonn
sta
Anchitectree pammetees
Figure 10: Results for the 2D Helmholtz problem using PINNs, For each hyperparameter configura- tion, 3 different seeds were used for initialization, and the median result was used. For each square, the best hyperparameter configuration (according to the median) is reported. The best performing method for each frequency w and each number of parameters is underlined.
Table 8: Average computational time per Adam training iteration for the Helmholtz problem Nvidia RTX A6000 GPU, All times are reported milliseconds. Each cell averages across three seeds and four different widths and depths hyperparameters reported G.2
Model
ActNet (N 8)
ActNet
ActNet
KAN
KAN
KAN
Siren
Size
(N 16)
(N
32)
(G' 3)
(G= 10)
(G = 30)
10k
5.03
7.12
9.53
6.84
10.7
13.0
1.31
20k
7.33
10.8
15.2
10.2
16.2
18.5
1.67
40k
10.6
16.6
26.0
17.1
25.0
25.7
2.50
80k
16.2
26.3
43.1
23.6
40.0
37.4
3.57
1,000 steps, and adaptive gradient clipping with parameter 0.01 described in Brock et (2021) Furthermore, to avoid unfairness from different scaling between the residual loss and the boundary loss, we enforce the boundary conditions exactly for all problems by multiplying the output of the neural network by the factor (1 - x² x²)(1 - y²). in strategy similar to what is outlined in Sukumar Srivastaval 2022).
The final relative L2 errors can be seen in figure 10 while the final residual losses can be seen in figure 11
An example solution is plotted in figure 2 and sample computational times are reported in
table 8
G.5 ALLEN-CAHN
We consider the Allen-Cahn equation with Dirichlet boundary and initial condition u(x;0) = x² cos(TT)), defined as
du
82 Ox
(a, t) (x, 1) + 5(") u(x, t)), u(z, 0) = =x ==2cos(mm) cos(TTT)) u(-1,1) u(1,() -1,
(x, E [-1,1]] x [0,1].
E [-1,]]
LE [0,
Each model was trained using Adam (Kingma & Ba 2017 for 100,000 iterations and batch size of 10,000 points uniformly sampled at random each step on [-1, x [0, 1). We use learning rate warmup from 1 to 5 x 10 1 over 1,000 iterations, then exponential decay with rate 0.9 every 1,000 steps, stopping the decay once the learning rate hits 10 10-6 We also use adaptive gradient clipping with parameter 0.01 described in Brock al. (202). Furthermore, to avoid unfairness
26

---

Published as a conference paper at ICLR 2025
Helmholtz 2D At Different Frequencies (Final Residual Loss Lower better)
.2.eet
5.9-10
-.508
0.1m-6
1.207
276-7
Lic4
4.9e5
4.0-8
1.4-7
3466
3.26-8
Liee
2.166
3.6-8
100-3
5.0e6
Lite-
1.3e-4
2866
25e-
2.te4
1.566
8.2e6
.27e22
1.fe-1
7.8-7
3.9c
3.Ac-4
3.5-4
3.1e-
3.3
1.1e5
4.0c
1.7c1
1-40+2
3,0++0
Me+S
2.4e+1
2.7--1
3.1+4)
200++1
1.fe2
6.34+1
1.6e0
7,9++2
5.8ma
9.3e1
6.l4
5.
3.5044
1.4+3
13ce1
7.3++4
6.0c3
6.4e1
Mtl1
Aethal tveny Humt
KAs Haad
Mlea stzee
Iean
RAs 3100
Johia reery Atom
Skrw miam
Authu comes Suaao
KiN aozn
Mt MnPann
Anchitectree pammeters
Figure 11: Final residual loss for the Helmholtz problem using PINNs. For each hyperparameter configuration, different seeds were used for initialization, and the median result was used. For each square, the best hyperparameter configuration (according to the median) is reported. The best performing method for each number parameters underlined.
Table Average computational time per Adam training iteration for Allen-Cahn problem on Nvidia RTX A6000 GPU. All times reported in milliseconds. Each cell averages across three seeds and four different widths and depths hyperparameters as reported on G.2
Model Size
ActNet
ActNet
ActNet
KAN
KAN
KAN
Siren
(N = 4)
(N = 8)
(N = 16)
(G 4)
(G = 8)
(G
16)
25k
4.16
6.29
10.8
9.01
24.5
56.5
2.04
50k
5.94
10.2
18.2
12.3
35.5
84.1
2.76
100k
9.65
16.9
30.1
17.1
48.8
122.9
4.30
from different scaling between the residual loss and the boundary loss, we enforce the initial and boundary conditions exactly by setting the output of the model to (1 - (2 cos(TTr) + t[(1 x² (ug I) - 1]. where ug is output the neural network, strategy similar to what outlined in Sukumar & Srivastava (2022).
Finally, we also employ the causal learning strategy from Wang et al. (2024b), updating the causal parameter every 10,000 steps according to the schedule [10 10 10 10a. 103, 10 10a, 10a. 10 10*].
The final relative L2 errors can be seen in figure 13 while the final residual losses can be seen in figure [4 An example solution is plotted in figure 12 and sample computational times are reported in table
Prediction
Reference Solution
Absolute Error
077
0038
035
031
035
20046
(004)
025
000
653
015
322
405
-20
-058
4.55
4075
-015
Figure 12: ActNet predictions for the Allen-Cahn equation. The relative L.2 error 4.51e-05.
27

---

Published as a conference paper at ICLR 2025
Allen-Cahn PINN (Relative L2 Error
Lower is beller)
1.64
2033
3.604
2.0c44
5.6e-5
Mes
2.4e-4
6.80-5
21e5
1.1c4
Addior coarn 25300
KAN25000
Stom 25000
MLP MLP25.00
Autiet rane) stozo
RAN sugon
Stem sor80
ML 30,000
Authir Ieeue
KAN 100s00
Sims niuoo
MLF 10000
Architecture purimeters
E0
Figure 13: Results for the Allen-Can Cahn problem using PINNs. For each hyperparameter configura- tion, 3 different seeds were used for initialization, and the median result was used. For each square, the best hyperparameter configuration (according the median) is reported. The best performing method for each number of parameters is underlined.
Allen-Cahn PINN (Final Loss
Lower is hetter)
2.5e-8
AdNa couns :3.00
2.00250000
Sion 23,0
2500
AutNai (mans Stoog
KAN s0,00
Siee 30.000
MLD 30.00
Auttet 1o0o0
RAN tuouaoi
Siens tun.a0
MLF tuRuta"
Architecture parmettes
Figure 14: Final residual loss for the Allen-Cahn problem using PINNs. For each hyperparameter configuration, different seeds were used for initialization, and the median result was used. For each square, the best hyperparameter configuration (according to the median) is reported. The best performing method for each number parameters underlined.
28

---

Published as a conference paper at ICLR 2025
G.6 ADVECTION
The ID advection equation is defined by
Tu du σt + c
Or 0,
E [0, 1], x E (0, 2x),
u(0,z) = g(x),
E (0, 2T),
with initial condition g(x) = sin(z) and transport velocity constant c 80.
For our implementation, configuration of their sota. py config file, along with their learnable pertodic embedding, changing ActNet with 5 layers, embedding dimension of 256 and wi value of We train using Adam for 300,00 iterations with adaptive gradient clipping (Brock et al. (021)) with parameter 0.01. We use starting learning rate of 10-3 and exponential rate decay of 0.9 every 5,000 steps. Each step took on average 40.0ms on a Nvidia RTX A6000 GPU.
the JaxPi library from Wang al. (023 2024a We copy over the
the architecture
For the PirateNet implementation, we used the code from the public GitHub repository https: //github. com www.eeeeeeeeeeeeeeeeeeeeeeeeeeee
and once again adapted the sota py config file from (Wang et al., 2023). ) as was done with ActNet. For a com- parable network size to the ActNet and modified MLP considered in (Wang et al. (003)) we set the network with two blocks of 3 layers each, totaling layers, all with width 200. Each step took on average 1 11.9ms on a Nvidia RTX A6000 GPU.
G.7 SURRAMOTO-SIVASHINGE
The Suramoto-Sivashinsky equation we considered is defined by
du
du
+ 3 yu
atu
01
aul
dr
Ox
dz
0
t E [0, 1]. E [m]
w00,0 x) u0(x)
E [0, 2m]
with periodic boundary conditions and constants 100/16 /6, 3 100/16 and 7 100 16" along with initial condition uo(z) = cos(2)( + sin())).
For our implementation, configuration of their sota. aotappy config file, changing the architecture to an ActNet with 5 layers, embedding dimension of 256 and Wo value of 5. We train using Adam (this time without adaptive gradient clipping) using a starting learning rate 10 and exponential rate decay of 0.8 every 3,500 steps.
use the JaxPi library from Wang al. 2023 2024a). We copy over the
We split the time domain [o, 1] into 10 windows of equal length. Since the precision of the solution matters most in the initial time steps (most of the error from the latter windows is due to propagated error in the initial conditions), we train the first window for 250k steps, the second and third for 200k steps, and all other for 150k steps. This results 15% less steps than what was done JaxPi, where each window is trained for 200k steps. Each step took on average 66.7ms on a Nvidia RTX A6000 GPU.
For the PirateNet implementation, we used the code from the public GitHub repository https: 7/qithub. www. www.:ea.........eeeeeeeeeeeaaaaa htwpaaaa//pppppeeeeeeeeeaaaeaaae httpe//www..iieoooaaaaa/////e///
and once again adapted the sota. - py config file from (Wang et al. 2023), as was done with ActNet. For a com- parable network size to the ActNet and modified ML. considered in (Wang et al. 2023), we set the network with two blocks of 3 layers each, totaling 6 layers, all with width 200. Fach step took on average 22.2ms on a Nvidia RTX A6000 GPU.
29


Reviews:
[{"review_id": "x9dUljBeQg", "reviewer": "Reviewer_cQ8K", "summary": "The paper presents an alternative formulation for a trainable network based on the KST. \n\nThe paper presents the theoretical property of the model and experimentally validates the performance as an approximation function of PINN, on 3 PDEs.", "questions": "I would like to see, even in the annex, the behavior against Neural Operators.\n\nis there other choices of the b(t) functions that works well?\n\nHow do you choose the hyper-parameters (N,m)?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The paper is clear and justify its contribution, and put in context with the current state of art\n\nThe paper justifies the change in the KAN architecture and its relationship. The connection with the multi-head transformer is a bit stretched tho. \n\nThe paper provides some experiments to show the potential of changing the representation of the KST.", "weaknesses": "The exposition is very good, the experiments show the advantage with respect to other KST architecture.\n\nThere is only the evaluation in the PINN context. It would be nice to have more experiments, for example against some neural operators and training from data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tQAkBKJwaF", "reviewer": "Reviewer_JpDB", "summary": "Based on a new formulation of the Kolmogorov Superposition Theorem (KST), this paper introduces a neural network ActNet to solve PDEs. ActNet is proposed as a global approximation in solving PDEs, which is similar to PINN. The authors demonstrate ActNet's theoretical properties, including universality, and validate it on a range of PDE benchmarks, showing improved performance over KAN and SIREN.", "questions": "1. This paper presents a novel approach for solving PDEs, similar to PINN. However, since the PDEs are known, why not use traditional numerical methods, such as FEM or FVM, on GPUs? A fair comparison is needed, focusing on accuracy, efficiency, and implementation complexity.\n2. How does the proposed method address high-dimensional input problems?\n3. Line 161: Why is the pathological behavior of the inner function relevant to the exact representation rather than the approximation?\n4. Table 2: A relative L2 error should suffice for accuracy comparison, so why include the residual loss? Additionally, why does the Allen-Cahn example in Table 2 lack a consistently best-performing model?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The authors provide a solid theoretical foundation for ActNet, detailing its universal approximation capabilities and presenting a well-motivated formulation based on KST. This paper compares the complexity of different formulations well.\n2. Three 1D examples and two 2D examples are included to demonstrate that ActNet can achieve better performance. \n3. This paper compares the model performance with other open-source JaxPi frameworks to enhance reproducibility and fairness.", "weaknesses": "1. The selected examples primarily use sinusoidal forcing terms, including equations like Poisson, Helmholtz, and Allen-Cahn. The proposed model uses sine functions as basis functions, which justifies its improved performance over the spline basis functions used in KAN. In Figure 12, we can see that SIREN can have the best performance. Therefore, comprehensive benchmarks should be included for challenging 2D and 3D problems, including the Navier–Stokes equations and turbulence cases.\n2. As this paper emphasizes the comparison between MLP and KAN, it is essential to include a general MLP model for performance evaluation alongside SIREN.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V5oZR5PJqf", "reviewer": "Reviewer_3BDJ", "summary": "This paper investigates alternative approaches to the Kolmogorov Superposition Theorem (KST) for neural network design, addressing practical limitations of the original KST formulation, such as its complexity and lack of structural insights. Kolmogorov-Arnold Networks (KANs) utilize KST for function approximation but have shown inconsistent performance compared to traditional multilayer perceptrons (MLPs). The authors introduce ActNet, a scalable model that builds on KST while mitigating some of its original limitations. Evaluated within the Physics-Informed Neural Networks (PINNs) framework for PDE simulation, ActNet consistently outperforms KANs and competes well with leading MLP-based methods, marking it as a promising direction for KST-based deep learning in scientific computing.", "questions": "1. As you mentioned about JAXPI, Why you didn't compare with their latest result which is Piratenet [1] that has been pubilshed Feb 2024 rather than Causal PINN that from 2022. Also I saw that you have cited this paper as well.\n2. What happen if you apply ActNet into more chaotic system, for example Navier–Stokes equations. is ActNet still performs better?\n3.\n\n[1] PirateNets: Physics-informed Deep Learning with Residual Adaptive Networks", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. **Originality**: The paper brings a fresh perspective to neural network design by leveraging alternative formulations of the Kolmogorov Superposition Theorem (KST). Introducing ActNet, based on Laczkovich's theorem, reflects a novel approach to overcoming the limitations of Kolmogorov-Arnold Networks (KANs), making KST more applicable to practical deep learning tasks, particularly within Physics-Informed Neural Networks (PINNs).\n\n2. **Quality**: The research is thorough, with ActNet being tested across multiple benchmarks against established models like KANs and MLPs, demonstrating its advantages in function approximation and handling PDE simulations. Theoretical foundations are strong, with proofs of ActNet’s universal approximation properties, and empirical results consistently show ActNet’s improved performance in accuracy and stability.\n\n3. **Clarity**: The paper is well-structured, presenting a clear motivation for the need for alternative KST formulations, followed by detailed descriptions of ActNet’s architecture and its theoretical underpinnings. The explanations of mathematical concepts and the positioning of ActNet within current scientific computing challenges are accessible and well-supported with illustrative figures and tables.\n\n4. **Significance**: ActNet addresses critical limitations in applying KST to neural networks, opening new possibilities for KST-based models in scientific computing and PDE simulation. The model's competitive performance against leading MLP-based approaches highlights its potential impact on advancing scientific machine learning, particularly in low-dimensional function approximation and complex simulations, where existing architectures struggle.", "weaknesses": "1.  Although ActNet performs well on PINNs, its comparisons are limited to specific benchmarks and do not consistently compare against the latest models for PINNs. \n2. The paper lacks detailed ablation studies on critical design choices within ActNet, such as the basis functions used or the impact of ActLayer depth. Including these analyses would clarify the sensitivity of ActNet’s performance to these parameters and help guide future implementations or adaptations of the model.\n3. While the paper offers rigorous theoretical grounding, some sections—particularly those detailing the inner workings of the ActLayer—may be dense for readers unfamiliar with KST.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uujfc37fKd", "reviewer": "Reviewer_CpiV", "summary": "The paper proposes ActNet, a novel neural network architecture based on Kolmogorov Superposition Theorem (KST), as an alternative to Kolmogorov-Arnold Networks (KANs). The authors use the Laczkovich version of KST to develop their architecture instead of the original KST emplyoed in KANs. They prove universal approximation properties and propose an initialization scheme for their method which they assures stability of network activations. \n\nEmpirically, they evaluate their model on PDE simulation tasks using a PINN framework. They compare their model to KANs and MLPs on a range of PDE equations such as Poisson, Helmholtz, Allen-Cahn and Kuramoto-Sivashinsky. Their method has comparable accuracy to the benchmarks chosen for comparison.", "questions": "* The original KAN paper boasts interpretability of the learnt network. Does your method offer any interesting results in terms of interpretability?\n* Do you have any experiments out-of-distribution (OOD) regime? It would be interesting to see how the model reacts to change in initial conditions or boundary conditions for example.\n* Why didn't you test on the function approximation tasks as presented in the original KAN paper? Given your method is a direct competitor to the KAN architecture, such results would be interesting to have.\n* How does your method perform given a different function basis?\n* Are $p$ and $\\omega$ trainable parameters?\n* My understanding of the proof of Theorem 3.3 is that you use Weierstrass theorem to create polynomial bases which approximate both i) the sum of $g$s ii) the weighted sum of $\\phi$s up to the desired precision $\\epsilon$. However, the link between this and the architecture is not immediately clear to me.\n\n    From what I gather, each polynomial approximation can be represented by an ActLayer with a polynomial basis of size $N_g$/$N_q$ and well chosen $\\beta$ values. Is my reasoning correct here? I think it would improve clarity if you mentioned how your proof relates to the proposed architecture.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. I find the paper to be well-written. The exposition of KST and of the ActNet architecture are clearly presented. The contributions made by this work is also clearly outlined.\n2. As far as I am aware, the technical contributions in this paper are novel. The proposed architecture outperforms other similar models (KST variants such as KANs and MLPs) on the chosen tasks, has better parameter efficiency and seems to benefit from good theoretical guarantees (universal approximation properties for a two layer network).\n3. I found the experimental section of the paper to be strong. Information about implementation and the considered PDEs is clearly given either in the main paper or in the appendix. Multiple ablations are conducted to validate the model. The choice of experimental framework is well motivated and the code is publicly available on Github.\n\nAll in all, I find this paper presents an interesting alternative to KANs.", "weaknesses": "1. Although interesting when compared to KANs and other MLPs, this method is not competitive when compared to SoTA ML informed solvers which use more sophisticated inductive biases such as [1,2,3,4] for example.\n2. I find the theoretical contribution of the paper to be a bit weak. Although an interesting result, it seems trivial to me that a method based on Superposition Theorems for representing multivariate functions would benefit from universal approximation properties. Moreover, it is not clear to me why the property presented in Theorem 3.4 is a selling point of your method. Even if this property didn't hold, we could simply use some form of LayerNorm to enforce this. Also, in the context of modelling physical equations, it is not clear to me that inputs would be distributed following a $(0,1)$ Normal distribution in the first place.\n3. There are a few typos/minor mistakes in the proofs. I will list them by point below.\n\nComments about the proof of Theorem 3.3:\n* Line 868: typo, I believe you meant $poly_g$ not $pol$.\n* \"by making $\\sum \\lambda \\phi (x)$ sure that is at most $δ$ far from the approximation $\\sum \\lambda \\hat{\\phi} (x)$\". You do not introduce/define this approximation before mentioning it here, I think it would make this step clearer if you defined what this approximation is.\n\nComments about the proof of Theorem 3.4\n* \"After properly initializing the $β$ and $λ$ parameters (detailed in Appendix F)\" this is not detailed in the appendix. Thus, when defining the expectation and second moment of these variables, it is not clear where the values you obtain come from (around lines 981 and 1006 )\n* Do you consider the that $p=0$ and $\\omega \\sim \\mathcal{N}(0,1)$ for this proof? If so, the eq. at line 984 is not clear to me. Same for the step between lines 1006 and 1009.\n\nAlso here are some minor comments you may want to address for the final version: \n* \"Table 1\" not \"table 1\" (line 67 for instance).\n* $m$ in Table 1 is not described until section 3. It would be nice to know (at least in big O) what this value is in terms of $d$ when looking at this table.\n* $\\beta$ should be bolded since it is a matrix, not a scalar (line 237).\n* Inconsistent use of $\\epsilon$ and $\\varepsilon$ (around line 295 theorem 3.3 def 3.2).\n* The grid size considered for the experiments is not reported in main paper and generally hard to find.\n* The \"Discussion\" section appears to be more of a conclusion to me than a discussion of results.\n\n[1] Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A., & Anandkumar, A. (2020). Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895.\n\n[2] Gin, C., Lusch, B., Brunton, S. L., & Kutz, J. N. (2021). Deep learning models for global coordinate transformations that linearise PDEs. European Journal of Applied Mathematics, 32(3), 515-539.\n\n[3] Lusch, B., Kutz, J. N., & Brunton, S. L. (2018). Deep learning for universal linear embeddings of nonlinear dynamics. Nature communications, 9(1), 4950.\n\n[4] Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Stuart, A., Bhattacharya, K., & Anandkumar, A. (2020). Multipole graph neural operator for parametric partial differential equations. Advances in Neural Information Processing Systems, 33, 6755-6766.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)