Sinc Kolmogorov-Arnold Network and Its Applications on Physics-informed Neural Networks

Authors: Tianchi Yu, JINGWEI QIU, Jiang Yang, Ivan Oseledets
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: Physics-informed neural networks, Kolmogorov-Arnold Networks, Partial differential equations, Computational physics.

Abstract:
In this paper, we propose to use Sinc interpolation in the context of Kolmogorov-Arnold Networks, neural networks with learnable activation functions, which recently gained attention as alternatives to multilayer perceptron. Many different function representations have already been tried, but we show that Sinc interpolation proposes a viable alternative, since it is known in numerical analysis to represent well both smooth functions and functions with singularities. This is important not only for function approximation but also for the solutions of partial differential equations with physics-informed neural networks. Through a series of experiments, we show that SincKANs provide better results in almost all of the examples we have considered.

Decision: reject
Reviewer scores: 5, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1730456883432

thread_type:
official_review

<thread_object>
creator: reviewer_gzak

<content>
summary:
The work proposes using Sinc interpolations replace the splines used within recently popularised Kolomogorov Arnold Networks (KANs). The work outlines foundational background required of Sinc interpolation and demonstrates some nice properties of Sinc interpolation on some specific benchmarks. It then outlines certain properties that must hold for effective Sinc interpolation and incorporates these within its KAN framework. Benchmarks are performed against against baselines comprising of two different multilayer perceptron architectures, and two other KAN architectures. These experiments encompass a range of function approximation tasks for manufactured functions, and physics-informed tasks.

strengths:
1. KANs are recent and might have much scope for optimisation. The interpolation scheme in KANs seems to be an important point to consider in practical use, so it is good that the authors draw attention to this.
2. Given the recency of KANs, the study provides more independent comparisons between KANs and MLPs which might be of interest to the wider community.
3. The authors do not straightforwardly substitute Sinc interpolants into the KAN architecture, but properly consider its principled application in terms of optimal step sizes, required exponential decay of the approximated function and how to circumvent this etc.

weaknesses:
1. The manufactured solutions used in the learning for approximation secion (3.1) are arbitrary and seemingly not very relevant to real world applications. They are both manufactured, and all are one dimensional.
2. While the performance of SincKANs is good in physics-informed settings, it feels like some scrutiny of the results is missing. There are many reasons why a physics-informed problem might fail to converge. It might not be that the reason for better performance in a physics-informed problem is a better architecture.

questions:
Question:
1. Can't one use a Sinc interpolator directly for all the function approximation tasks?

Sugestions:
1. The experiments of section 3.1 do not seem so relevant for practical applications. It seems that what might be most interesting would be to take a selection of complicated PDE/ODE data (using any off the shelf differential equation solver) and use these instead. This would demonstrate the value of SincKANs in practical circumstances, and in situations of more than 1 dimension. As it stands, these benchmarks feel artificial and irrelevant for real world application.
2. More scrutiny of the physics-informed losses would be beneficial. Some plots of solutions and errors across the poorer performing methods might help understand why they are performing badly. Is it that boundary conditions are not being adhered to? Maybe there are regions of high PDE loss in the resulting solution? Perhaps small changes (e.g. weighing boundary conditions effectively) might lead to improved performance.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730667372662

thread_type:
official_review

<thread_object>
creator: reviewer_wjox

<content>
summary:
The paper builds upon recently proposed architecture KANs that are meant to be more parameter efficient than MLPs and provide additional benefits like interpretability. The authors replace the splines in KANs with sinc to propose SincKAN. Furthermore, the paper presents experiments in PINNs (using KANs) to demonstrate potential benefits of SincKAN.

strengths:
1. Applying sinc to replace splines in KAN is a novel approach. Additionally, the authors address the practical challenges of selecting step sizes using multiple step sizes.

weaknesses:
1. The PDEs selected for the experiments are generally unsuitable to demonstrate the benefits of PIKANs. The chosen problems are extremely small and these problems can be solved using any numerical PDE solver. Since PINNs / PIKANs don't provide any practical advantage in small problems, it would be interesting to see if the benefits proposed here can be demonstrated in real problems.

questions:
1. For MLP architectures, how were the architectures chosen? Since the problems are small and toy-ish, over-parameterization may cause a higher final loss.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730682614560

thread_type:
official_review

<thread_object>
creator: reviewer_fyyo

<content>
summary:
This paper proposes KANs with sinc function as basis to represent univariate function, to which they refer as sincKan. They first discuss about the sinc function and its convergence properties and then extend it such that it becomes a viable and practical workhorse in the context of KANs. Equation 14 clarifies their contributions, namely, 1) choosing step size without the knowledge of function f (by using different h values, larger and smaller than the optimal one), 2) coordinate transformation (a composition of transformation from finite to infinite domain and a normalizing transformation), and 3) enforcing exponential decay (by considering g-f instead of f with a linear g). First property manifests itself in use of ci,j and hj, second one is hidden in $\gamma^{-1}$, and the third property is $c1x+c2$. They use both function approximation and solution of PDEs (similar to PINNs, which enforces the solution as soft constraint in the loss function) as numerical experiments and show lower error in most cases compared to relevant baselines.

strengths:
The paper and idea are well-written. It is clear what authors try to implement and they are also rigor in deriving equation 14, which is a foundation for their sincKAN network. 
The choice of boundary layer example is smart and shows the good performance of sincKAN in the presence of singularities (specially pronounced in large epsilon (lower width of boundary layer or larger Reynolds, which is more complicated).

weaknesses:
The contribution of this paper seems limited to me and I hope authors clarify this in the rebuttal period. 
Using more advanced basis functions for KANs is a meaningful research direction but currently it seems to me no single method can be seen as the best solution for all PDEs. In fact sincKAN may have better performance in the presence of singularities but may not be the best option otherwise. In the current version, I can't derive a concrete conclusion as which version of KANs to use for what type of problems (within PDE domain or even beyond). In fact, in Table 2, sincKAN is not the best option in some cases.
In an analogy to more conventional deep learning papers, the use of Relu activation functions with CNN enable them to achieve very good performance: it helped with vanishing gradient problem, it helped with efficiency of implementations, and also empirical success (for instance AlexNet). sincKAN does not seem to be such an abrupt idea. The theorems provided do not really guarantee the performance (specially considering the fact that authors use a range of step sizes instead of the optimal value due to practical reasons). 
In this vein, authors discuss how sincKAN can help with spectral bias, but not much discussion is done on other issues for PINNs e.g. issue for causality (which makes them problematic in chaotic PDEs--which by the way is good to add to examples), and many other issues (like plateauing loss, etc.).
Finally, in "vanilla" KANs paper there are some arguments on the advantage of using B-splines, which enable them the use of grid related tricks including grid extension and grid update tricks. In other variants also there are some interesting merits; it's hard to say sincKAN still has all this merits and in addition to them provide more robustness in the case of singularities.

questions:
Can sincKAN help with some pathological issues of PINNs? Would be interesting to explore; see for example Wang, S., Teng, Y., and Perdikaris, P. Understanding and Mitigating Gradient Flow Pathologies in Physics- Informed Neural Networks. SIAM Journal on Scientific Computing, 43(5):A3055–A3081, 2021a. or Wang, S., Sankaran, S., and Perdikaris, P. Respecting causal- ity is all you need for training physics-informed neural networks. arXiv preprint arXiv:2203.07404, 2022a. or the more recent paper Challenges in Training PINNs: A Loss Landscape Perspective by Rathore et al.

I still don't see the real advantages of theorems 1 and 2, since the optimal value of step is not uses in sincKAN. Also, I advise authors formally add another theorem to Coordinate transformation section to prove for the composition, the theory still holds (it should be straightforward). Also, if a function needs to be Hardy, does that mean the guarantees are only valid for differentiable functions f? It would be good to discuss the implementations.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730686382166

thread_type:
official_review

<thread_object>
creator: reviewer_mr4i

<content>
summary:
This paper presents a new model that applies sinc interpolation to the previously proposed KAN model. It discusses the advantages of the proposed model for function fitting and solving PDEs in a Physics-informed manner.

strengths:
The study explores how the recently developed KAN model can better handle singular problems from the perspective of Physics-Informed Neural Networks (PINNs).

weaknesses:
I believe the main goal of this paper is to improve existing PINN-based models through the development of sincKAN. However, as shown in Table 2, it's unclear if sincKAN demonstrates convincingly better performance than other existing models across various PDE problems. Clearly, as mentioned in the PI-KAN model, there are specific advantages and disadvantages to using KAN for solving PDEs, such as computational cost or training speed. If sincKAN does not consistently outperform across all datasets, as seen in Table 2, what advantages does sincKAN offer over using standard MLP or modified MLP to train Physics-informed Loss? While sincKAN appears to perform better on boundary layer problems, as shown in Table 3, the paper’s title and focus could have been more aligned with boundary layer issues if that is its primary strength. Additionally, are there recent studies related to PINNs specifically aimed at solving boundary layer problems? What would happen if epsilon increased to a larger value, such as 10^8? It might also help to include a basic explanation of KAN’s underlying mechanism for readers unfamiliar with the model, as certain concepts could be difficult to grasp without this context.

questions:
* What exactly is f(jh) in equation (6)? Is it simply the value of f at jh? Is equation (6) interpolating f(x) using those values? If so, do we need to know those values precisely to approximate f(x) with this formulation?
* Could other problems with singularities be tested? Does the proposed method also show significant advantages and accurately approximate solutions in all cases where singular phenomena appear? Could it be applied to other boundary layer problems or higher-dimensional problems beyond two dimensions?
* For line 140, it might be better to use the \paragraph function for terms like “Convergence theorem” or “On a general interval (a, b)” to save space.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732367066969

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Many reviewers doubted that :

# 1) the relationship between Theorem 1 and SincKAN.

Theorem 1 and 2 are for Sinc numerical methods, especially the Sinc interpolations. Those two theorems inspired us to use Sinc interpolation in KANs. However, because directly applying the Sinc interpolation is not a good idea as we discuss in our manuscript, we unfold this approach to meet the demands of machine learning. Thus, this theorem is merely our fundamental theorem for replacing the cubic interpolation in KANs. Furthermore, interpolation is one of the most important compositions in KANs but not all of KANs. Thus, Theorem 1 and 2 can't dominate SincKAN totally.

# 2) SincKAN is not the best in every experiment.
Our research explores the implementations of KANs in PINNs and proposes a powerful network that is helpful in PINNs. As Sinc interpolation is not as good as Chebyshev and splines when dealing with smooth functions, it is acceptable that SincKAN is not the best when solving smooth PDEs. However, because SincKAN excels in handling singular problems and the results of smooth cases are good enough, we think SincKAN is a competitive network for PINNs. 

Additionally, some reviewers argued that why we put the experiments that SincKAN is not the best. In Table 1, the approximation experiments including the singular case and smooth cases show that SincKAN has the ability to be the best network. However, due to the limitations discussed in the Conclusion section, SincKAN can't be applied in PINNs for every case. Thus, we want to determine the extent to which those restrictions impair SincKAN's performance in complex but non-single cases.  That is the reason we have results in nonlinear functions, Burgers' equations, and Navier-Stokes equations: we want to show that although SincKAN has those limitations, SincKAN is still good enough for general cases. 

# 3) the performance of SincKAN on more complex problems.
We added two extra experiments to show the performance on complex problems:
## Fractional PDEs
We consider the fractional spacial order problem:
\begin{equation}
    -\Delta^su+\gamma u=f, \quad \boldsymbol{x}\in (-1,1)^d
\end{equation}
if $f=1+\gamma u$, the exact solution is
\begin{equation}
    u(x)=\frac{2^{-2s}\Gamma(d/2)}{\Gamma(d/2+s)\Gamma(1+s)}(1-\|x\|^2)^s,
\end{equation}
where $s\in \mathbb{R}_+$ is a hyperparameter. We consider $d=1$ case, the results are:
|model|$s=0.85$|$s=0.95$|
|----|----|---|
MLP  | 8.84e-03  | 7.32e-02 |
SincKAN | 6.57e-04 |7.28e-03 |

## Poisson equations
The second one is high-dimensional Poisson equations:

\begin{equation}
    \Delta u =f, \quad \boldsymbol{x}\in [0,1]^d.
\end{equation}
where $f=2\alpha(2\alpha ||\boldsymbol{x}||^2_2-d) e^{-\alpha ||\boldsymbol{x}||_2^2}$, the exact solution is $u(\boldsymbol{x})=e^{-\alpha ||\boldsymbol{x}||^2_2}$. In our experiments, $d=100$, $\alpha=10/d$. The results are:
| Model   | Relative error|
|---------|----------------|
|  MLP     | 7.58e-02      |
| SincKAN | 3.49e-02     |

The main changes in our paper are:

1) add $\epsilon=10^4$ boundary layer problems, fractional PDEs, and high-dimensional Poisson as extra experiments.

2) added a theoretical proof of coordinate transformation.

3) added a thoughtful introduction to KAN.

4) changed the title to more focus on singular problems.

5) added a discussion of the grid extension.

6) added more experiments on approximation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732424409646

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear AC,

We kindly ask to additionally encourage reviewer wJoX interact with us during the discussion period. It is our belief that we addressed concerns of wJoX and we would like to receive additional comments or reevaluation of the score.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes the use of sinc interpolation in Kolmogorov-Arnold networks (KANs) with the goal of achieving better function interpolation and for the solution of PDEs in the context of physics-informed neural networks (PINNs). I thank the authors and the reviewers for their discussions. From these, I believe improving KANs is an interesting research direction, in particular, within the context of solving PDEs in PINNs. 

The major criticism of this paper is in terms of the experimental evaluation, with regards to, e.g., the focus of the paper (should it be on boundary layer problems?), the lack of robustness, more visualizations, better explanations of the KANs underlying mechanism, and the unsuitability of the benchmarks problems used for evaluation. 

I strongly disagree with the belief of some reviewers (Reviewer mr4i, other reviewers at ICLR and in other top ML venues) that an ML paper must be shown to be the best across all datasets and (worse) that those problem where the method does not do well should not be reported. In fact, as a community, we need to commend authors to do otherwise. However, I believe the point on the unsuitability of the benchmarks used is an important one. Although the authors have  added new results, some of the reviewers remained unconvinced. Because of this and the additional issues outlined above, I recommend rejection.