Deep Learning Alternatives Of The Kolmogorov Superposition Theorem

Authors: Leonardo Ferreira Guilhoto, Paris Perdikaris
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: kolmogorov-arnold representation theorem, function approximation, physics informed neural networks, ai4science

Abstract:
This paper explores alternative formulations of the Kolmogorov Superposition Theorem (KST) as a foundation for neural network design. The original KST formulation, while mathematically elegant, presents practical challenges due to its limited insight into the structure of inner and outer functions and the large number of unknown variables it introduces. Kolmogorov-Arnold Networks (KANs) leverage KST for function approximation, but they have faced scrutiny due to mixed results compared to traditional multilayer perceptrons (MLPs) and practical limitations imposed by the original KST formulation. To address these issues, we introduce ActNet, a scalable deep learning model that builds on the KST and overcomes some of the drawbacks of Kolmogorov's original formulation. We evaluate ActNet in the context of Physics-Informed Neural Networks (PINNs), a framework well-suited for leveraging KST's strengths in low-dimensional function approximation, particularly for simulating partial differential equations (PDEs). In this challenging setting, where models must learn latent functions without direct measurements, ActNet consistently outperforms KANs across multiple benchmarks and is competitive against the current best MLP-based approaches. These results present ActNet as a promising new direction for KST-based deep learning applications, particularly in scientific computing and PDE simulation tasks.

Decision: accept (spotlight)
Reviewer scores: 8, 6, 8, 8

Reviews and discussion:
<thread>
creation date:
1730482825393

thread_type:
official_review

<thread_object>
creator: reviewer_cpiv

<content>
summary:
The paper proposes ActNet, a novel neural network architecture based on Kolmogorov Superposition Theorem (KST), as an alternative to Kolmogorov-Arnold Networks (KANs). The authors use the Laczkovich version of KST to develop their architecture instead of the original KST emplyoed in KANs. They prove universal approximation properties and propose an initialization scheme for their method which they assures stability of network activations. 

Empirically, they evaluate their model on PDE simulation tasks using a PINN framework. They compare their model to KANs and MLPs on a range of PDE equations such as Poisson, Helmholtz, Allen-Cahn and Kuramoto-Sivashinsky. Their method has comparable accuracy to the benchmarks chosen for comparison.

strengths:
1. I find the paper to be well-written. The exposition of KST and of the ActNet architecture are clearly presented. The contributions made by this work is also clearly outlined.
2. As far as I am aware, the technical contributions in this paper are novel. The proposed architecture outperforms other similar models (KST variants such as KANs and MLPs) on the chosen tasks, has better parameter efficiency and seems to benefit from good theoretical guarantees (universal approximation properties for a two layer network).
3. I found the experimental section of the paper to be strong. Information about implementation and the considered PDEs is clearly given either in the main paper or in the appendix. Multiple ablations are conducted to validate the model. The choice of experimental framework is well motivated and the code is publicly available on Github.

All in all, I find this paper presents an interesting alternative to KANs.

weaknesses:
1. Although interesting when compared to KANs and other MLPs, this method is not competitive when compared to SoTA ML informed solvers which use more sophisticated inductive biases such as [1,2,3,4] for example.
2. I find the theoretical contribution of the paper to be a bit weak. Although an interesting result, it seems trivial to me that a method based on Superposition Theorems for representing multivariate functions would benefit from universal approximation properties. Moreover, it is not clear to me why the property presented in Theorem 3.4 is a selling point of your method. Even if this property didn't hold, we could simply use some form of LayerNorm to enforce this. Also, in the context of modelling physical equations, it is not clear to me that inputs would be distributed following a $(0,1)$ Normal distribution in the first place.
3. There are a few typos/minor mistakes in the proofs. I will list them by point below.

Comments about the proof of Theorem 3.3:
* Line 868: typo, I believe you meant $poly_g$ not $pol$.
* "by making $\sum \lambda \phi (x)$ sure that is at most $δ$ far from the approximation $\sum \lambda \hat{\phi} (x)$". You do not introduce/define this approximation before mentioning it here, I think it would make this step clearer if you defined what this approximation is.

Comments about the proof of Theorem 3.4
* "After properly initializing the $β$ and $λ$ parameters (detailed in Appendix F)" this is not detailed in the appendix. Thus, when defining the expectation and second moment of these variables, it is not clear where the values you obtain come from (around lines 981 and 1006 )
* Do you consider the that $p=0$ and $\omega \sim \mathcal{N}(0,1)$ for this proof? If so, the eq. at line 984 is not clear to me. Same for the step between lines 1006 and 1009.

Also here are some minor comments you may want to address for the final version: 
* "Table 1" not "table 1" (line 67 for instance).
* $m$ in Table 1 is not described until section 3. It would be nice to know (at least in big O) what this value is in terms of $d$ when looking at this table.
* $\beta$ should be bolded since it is a matrix, not a scalar (line 237).
* Inconsistent use of $\epsilon$ and $\varepsilon$ (around line 295 theorem 3.3 def 3.2).
* The grid size considered for the experiments is not reported in main paper and generally hard to find.
* The "Discussion" section appears to be more of a conclusion to me than a discussion of results.

[1] Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A., & Anandkumar, A. (2020). Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895.

[2] Gin, C., Lusch, B., Brunton, S. L., & Kutz, J. N. (2021). Deep learning models for global coordinate transformations that linearise PDEs. European Journal of Applied Mathematics, 32(3), 515-539.

[3] Lusch, B., Kutz, J. N., & Brunton, S. L. (2018). Deep learning for universal linear embeddings of nonlinear dynamics. Nature communications, 9(1), 4950.

[4] Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Stuart, A., Bhattacharya, K., & Anandkumar, A. (2020). Multipole graph neural operator for parametric partial differential equations. Advances in Neural Information Processing Systems, 33, 6755-6766.

questions:
* The original KAN paper boasts interpretability of the learnt network. Does your method offer any interesting results in terms of interpretability?
* Do you have any experiments out-of-distribution (OOD) regime? It would be interesting to see how the model reacts to change in initial conditions or boundary conditions for example.
* Why didn't you test on the function approximation tasks as presented in the original KAN paper? Given your method is a direct competitor to the KAN architecture, such results would be interesting to have.
* How does your method perform given a different function basis?
* Are $p$ and $\omega$ trainable parameters?
* My understanding of the proof of Theorem 3.3 is that you use Weierstrass theorem to create polynomial bases which approximate both i) the sum of $g$s ii) the weighted sum of $\phi$s up to the desired precision $\epsilon$. However, the link between this and the architecture is not immediately clear to me.

    From what I gather, each polynomial approximation can be represented by an ActLayer with a polynomial basis of size $N_g$/$N_q$ and well chosen $\beta$ values. Is my reasoning correct here? I think it would improve clarity if you mentioned how your proof relates to the proposed architecture.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730674609266

thread_type:
official_review

<thread_object>
creator: reviewer_3bdj

<content>
summary:
This paper investigates alternative approaches to the Kolmogorov Superposition Theorem (KST) for neural network design, addressing practical limitations of the original KST formulation, such as its complexity and lack of structural insights. Kolmogorov-Arnold Networks (KANs) utilize KST for function approximation but have shown inconsistent performance compared to traditional multilayer perceptrons (MLPs). The authors introduce ActNet, a scalable model that builds on KST while mitigating some of its original limitations. Evaluated within the Physics-Informed Neural Networks (PINNs) framework for PDE simulation, ActNet consistently outperforms KANs and competes well with leading MLP-based methods, marking it as a promising direction for KST-based deep learning in scientific computing.

strengths:
1. **Originality**: The paper brings a fresh perspective to neural network design by leveraging alternative formulations of the Kolmogorov Superposition Theorem (KST). Introducing ActNet, based on Laczkovich's theorem, reflects a novel approach to overcoming the limitations of Kolmogorov-Arnold Networks (KANs), making KST more applicable to practical deep learning tasks, particularly within Physics-Informed Neural Networks (PINNs).

2. **Quality**: The research is thorough, with ActNet being tested across multiple benchmarks against established models like KANs and MLPs, demonstrating its advantages in function approximation and handling PDE simulations. Theoretical foundations are strong, with proofs of ActNet’s universal approximation properties, and empirical results consistently show ActNet’s improved performance in accuracy and stability.

3. **Clarity**: The paper is well-structured, presenting a clear motivation for the need for alternative KST formulations, followed by detailed descriptions of ActNet’s architecture and its theoretical underpinnings. The explanations of mathematical concepts and the positioning of ActNet within current scientific computing challenges are accessible and well-supported with illustrative figures and tables.

4. **Significance**: ActNet addresses critical limitations in applying KST to neural networks, opening new possibilities for KST-based models in scientific computing and PDE simulation. The model's competitive performance against leading MLP-based approaches highlights its potential impact on advancing scientific machine learning, particularly in low-dimensional function approximation and complex simulations, where existing architectures struggle.

weaknesses:
1.  Although ActNet performs well on PINNs, its comparisons are limited to specific benchmarks and do not consistently compare against the latest models for PINNs. 
2. The paper lacks detailed ablation studies on critical design choices within ActNet, such as the basis functions used or the impact of ActLayer depth. Including these analyses would clarify the sensitivity of ActNet’s performance to these parameters and help guide future implementations or adaptations of the model.
3. While the paper offers rigorous theoretical grounding, some sections—particularly those detailing the inner workings of the ActLayer—may be dense for readers unfamiliar with KST.

questions:
1. As you mentioned about JAXPI, Why you didn't compare with their latest result which is Piratenet [1] that has been pubilshed Feb 2024 rather than Causal PINN that from 2022. Also I saw that you have cited this paper as well.
2. What happen if you apply ActNet into more chaotic system, for example Navier–Stokes equations. is ActNet still performs better?
3.

[1] PirateNets: Physics-informed Deep Learning with Residual Adaptive Networks
</content>
</thread_object>

</thread>
<thread>
creation date:
1730684071708

thread_type:
official_review

<thread_object>
creator: reviewer_jpdb

<content>
summary:
Based on a new formulation of the Kolmogorov Superposition Theorem (KST), this paper introduces a neural network ActNet to solve PDEs. ActNet is proposed as a global approximation in solving PDEs, which is similar to PINN. The authors demonstrate ActNet's theoretical properties, including universality, and validate it on a range of PDE benchmarks, showing improved performance over KAN and SIREN.

strengths:
1. The authors provide a solid theoretical foundation for ActNet, detailing its universal approximation capabilities and presenting a well-motivated formulation based on KST. This paper compares the complexity of different formulations well.
2. Three 1D examples and two 2D examples are included to demonstrate that ActNet can achieve better performance. 
3. This paper compares the model performance with other open-source JaxPi frameworks to enhance reproducibility and fairness.

weaknesses:
1. The selected examples primarily use sinusoidal forcing terms, including equations like Poisson, Helmholtz, and Allen-Cahn. The proposed model uses sine functions as basis functions, which justifies its improved performance over the spline basis functions used in KAN. In Figure 12, we can see that SIREN can have the best performance. Therefore, comprehensive benchmarks should be included for challenging 2D and 3D problems, including the Navier–Stokes equations and turbulence cases.
2. As this paper emphasizes the comparison between MLP and KAN, it is essential to include a general MLP model for performance evaluation alongside SIREN.

questions:
1. This paper presents a novel approach for solving PDEs, similar to PINN. However, since the PDEs are known, why not use traditional numerical methods, such as FEM or FVM, on GPUs? A fair comparison is needed, focusing on accuracy, efficiency, and implementation complexity.
2. How does the proposed method address high-dimensional input problems?
3. Line 161: Why is the pathological behavior of the inner function relevant to the exact representation rather than the approximation?
4. Table 2: A relative L2 error should suffice for accuracy comparison, so why include the residual loss? Additionally, why does the Allen-Cahn example in Table 2 lack a consistently best-performing model?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730716002694

thread_type:
official_review

<thread_object>
creator: reviewer_cq8k

<content>
summary:
The paper presents an alternative formulation for a trainable network based on the KST. 

The paper presents the theoretical property of the model and experimentally validates the performance as an approximation function of PINN, on 3 PDEs.

strengths:
The paper is clear and justify its contribution, and put in context with the current state of art

The paper justifies the change in the KAN architecture and its relationship. The connection with the multi-head transformer is a bit stretched tho. 

The paper provides some experiments to show the potential of changing the representation of the KST.

weaknesses:
The exposition is very good, the experiments show the advantage with respect to other KST architecture.

There is only the evaluation in the PINN context. It would be nice to have more experiments, for example against some neural operators and training from data.

questions:
I would like to see, even in the annex, the behavior against Neural Operators.

is there other choices of the b(t) functions that works well?

How do you choose the hyper-parameters (N,m)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732222307331

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Improvements

comment:
## **General Post To Reviewers**
First of all, we would like to thank our reviewers and area chairs for their thoughtful feedback and insights. Based on the reviewers' comments, we have made several significant improvements to our paper, summarized as follows. The new version of the PDF file includes changes highlighted in blue text.

### **Additional Experiments and Comparisons**
**Expanded Comparisons with State-of-the-Art**: We now include comprehensive comparisons against PirateNet, a leading PINN architecture with sophisticated inductive biases. Our results show ActNet outperforms PirateNet on both the Advection equation and full-time solution of the Kuramoto-Sivashinsky equation. Notably, ActNet achieves this superior performance without using PirateNet's physics-informed initialization and adaptive residual connections (features that could potentially further improve ActNet's performance in future work).

**Broader MLP Baselines**: We have added traditional MLPs with tanh, sigmoid and GELU activations as additional baselines. After running over 1,700 new experiments across different architectures and hyperparameter settings, we find ActNet consistently outperforms these baselines, particularly on challenging problems with high-frequency content. In total, our evaluation now encompasses over 7,000 independent training runs, demonstrating a rigorous empirical validation of our approach.

### **New Hyperparameter Analysis**
We have added a detailed analysis of ActNet's hyperparameter choices in Appendix D.4, providing practical guidance for implementing our method. Key findings include:
- The basis size N=4 is typically sufficient, with limited gains from larger values.
- Network width and depth trade-offs follow expected deep learning patterns.
- Minimum of two ActLayers is recommended, aligning with our theoretical analysis.

### **Enhanced Theoretical Clarity**
We have improved our theoretical exposition by:
- Correcting notation and typos in proofs.
- Adding explanatory details to make derivations more accessible.
- Clarifying connections between theoretical guarantees and practical implementation.

We also plan to make all experimental data, including loss curves and training runs, publicly available through WeightsAndBiases (currently withheld for anonymity).
</content>
</thread_object>

</thread>
<thread>
creation date:
1732564303694

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Reminder About Discussion Period Deadline

comment:
First of all, we would like to thank the reviewers for their time. We would also like to remind you that last week we published improvements to our paper (including additional experiments and ablations) and responded to the questions from your reviews. Given that **tomorrow, November 26th**, is the last day for reviewers to post comments, we would like to remind you of this deadline, and encourage you to ask any more clarifying questions you might have. So far, only one of the reviewers has responded to our replies, and improved their score significantly, which we believe is indicative of the significance of our new additions to the paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733268435577

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Paper And Discussion Period

comment:
# Summary of Paper And Discussion Period

## Paper Summary
Our  paper explores alternative formulations of the Kolmogorov Superposition Theorem (KST) for deep learning applications and introduces **ActNet**, a scalable neural network architecture that addresses many of the practical limitations of KANs and other KST-inspired methods. Evaluated in the context of Physics-Informed Neural Networks (PINNs) for solving partial differential equations, a well-suited testbed for approaches based on the KST, ActNet consistently outperforms KANs and competes with state-of-the-art MLP-based models such as PirateNets, highlighting its potential for scientific computing and PDE simulations. Overall, we believe our paper successfully proposes an interesting and novel theoretical perspective on applications of the KST to deep learning, and supports this new direction with several rigorous experiments and ablations that showcase ActNet’s promise in physics-informed machine learning. We hope this work leads other researchers to use ActNet in challenging PINN tasks, and to consider their own alternatives to our method, by potentially adapting further formulations of the KST.

## Discussion Period Summary
We would like to begin by thanking all the reviewers and area chairs for facilitating the discussion period and helping us strengthen our paper. After receiving initial reviews we took the time to carefully consider each of the reviewer's questions and concerns, and based on this feedback we carried out new experiments and provided additional information in the manuscript. We believe this lead to a considerable improvement to our submission by helping highlight the advantages of our method over existing architectures, in addition to providing better theoretical clarity.

In the rebuttal version of our paper we:
- **Expanded comparisons against state-of-the-art architectures** by including results for PirateNet, which currently offer some of the best performance for PINNs in challenging settings. We find that a plain ActNet achieves superior performance without even using PirateNet's physics-informed initialization and adaptive residual connections (features that could potentially further improve ActNet's performance in future work).
- **Included broader MLP baselines** in ablation experiments by running MLPs with tanh, sigmoid and GELU activations as additional baselines. After running over 1,700 new experiments across different architectures and hyperparameter settings, we find ActNet consistently outperforms these baselines, particularly on challenging problems with high-frequency content. In total, our evaluation now encompasses over 7,000 independent training runs, demonstrating a rigorous empirical validation of our approach, in addition to the strong theoretical component of our work.
- **Added a new appendix analysing hyperparameter choice**, which clarifies the influence of architecture hyperparameters on performance. Notably, we find that the basis size $N$ does not need to be large in order to yield good results (with $N=4$ typically sufficing), and that there is significant improvement when using at least two ActLayers, a phenomena which is directly related to the theoretical guarantees provided by the Laczkovich version of the KST as described in theorem 3.3.
- **Improved the exposition of theoretical elements** by introducing new explanations to steps in proofs and correcting typos and notation mistakes.

We believe these improvements were possible due in no small part to the quality of the reviews, and we deeply thank reviewers and area chairs for taking the time to make the discussion period productive. Reviewers appeared to be supportive of the changes we included in our rebuttal, **with the average reviewer score increasing by 1.5 points**, which we believe is indicative of the hard work we put into the submission, as well as the quality of our paper.
</content>
</thread_object>

</thread>


Metareview:
metareview:
Based on a variant of Kolmogorov superposition theorem given by Miklos Laczkovich in 2021, the authors propose to use it as a deep neural network design. It used d inner functions, and one outer function. The ActNet based on this has d adaptive activation functions, different for each neuron. The numerical experiments are quite sparse in this case: 5 problems vs 4 methods However, the experiments themselves seem to be done correctly, for different number of parameters/seeds. 
Overall, although: 
1) No theoretical results (i.e. are the 'true' functions smooth, so they can be approximated well with simple basis functions?)
2) Numerical experiments are not 100% convincing that this approach is better

all the reviewers are quite positive in the end.