Sinc Kolmogorov-Arnold Network and Its Applications on Physics-informed Neural Networks

## Abstract

Abstract In this paper, we propose to use Sinc interpolation in the context of Kolmogorov-Arnold Networks, neural networks with learnable activation functions, which recently gained attention as alternatives to multilayer perceptron. Many different function representations have already been tried, but we show that Sinc interpolation proposes a viable alternative, since it is known in numerical analysis to represent well both smooth functions and functions with singularities. This is important not only for function approximation but also for the solutions of partial differential equations with physics-informed neural networks. Through a series of experiments, we show that SincKANs provide better results in almost all of the examples we have considered.

## 1 Introduction

Multilayer perceptron (MLP) is a classical neural network consisting of fully connected layers with a chosen nonlinear activation function, which is a superposition of simple functions. The classical Kolmogorov-Arnold representation theorem Kolmogorov (); Arnol’d () states that every function can be represented as a superposition of function of at most $2$ variables, motivating the research for learnable activation functions.

Kolmogorov’s Spline Network (KSN) Igelnik & Parikh () is a two-layer framework using splines as the learnable activation functions. Recently, Kolmogorov-Arnold Networks (KANs) Liu et al. () sparkled a new wave of attention to those approaches, by proposing a multilayer variant of KSN. Basically, any successful basis to represent univariate functions can provide a new variant of KAN. Many well-known methods have already been investigated including wavelet Bozorgasl & Chen (); Seydi (), Fourier series Xu et al. (), finite basis Howard et al. (), Jacobi basis functions Aghaei (), polynomial basis functions Seydi (), rational functions Aghaei () and Chebyshev polynomials SS (); Shukla et al. ().

We propose to use Sinc interpolation (the Sinc function is defined in Eq. 4) which is a very efficient and well-studied method for function interpolation, especially 1D problems Stenger (). To our knowledge, it has not been studied in the context of KANs. We argue that the the cubic spline interpolation used in KANs should be replaced by the Sinc interpolation, because splines are particularly good for the approximation of analytic functions without singularities which MLP is also good at, while Sinc methods excel for problems with singularities, for boundary-layer problems, and for problems over infinite or semi-infinite range Stenger (). Herein, utilizing Sinc functions can improve the accuracy and generalization of KANs, and make KANs distinguishing and competitive, especially in solving aforementioned mathematical problems in machine learning. We will confirm our hypothesis by numerical experiments.

Physics-informed neural networks (PINNs) Lagaris et al. (); Raissi et al. () are a method used to solve partial differential equations (PDEs) by integrating physical laws with neural networks in machine learning. The use of Kolmogorov-Arnold Networks (KANs) in PINNs has been explored and is referred to as Physics-Informed Kolmogorov-Arnold Networks (PIKANs) Rigas et al. (); Wang et al. (). Due to the high similarity between KAN and MLP, PIKANs inherit several advantages of PINNs, such as overcoming the curse of dimensionality (CoD) Wojtowytsch & Weinan (); Han et al. (), handling imperfect data Karniadakis et al. (), and performing interpolation Sliwinski & Rigas (). PINNs have diverse applications, including fluid dynamics Raissi et al. (); Jin et al. (); Kashefi & Mukerji (), quantum mechanical systems Jin et al. (), surface physics Fang & Zhan (), electric power systems Nellikkath & Chatzivasileiadis (), and biological systems Yazdani et al. (). However, they also face challenges such as spectral bias Xu et al. (); Wang et al. (), error estimation Fanaskov et al. (), and scalability issues Yao et al. ().

In this paper, we introduce a novel network architecture called Sinc Kolmogorov-Arnold Networks (SincKANs). This approach leverages Sinc interpolation, which is particularly adept at approximating functions with singularities, to replace cubic interpolation in the learnable activation functions of KANs. The ability to handle singularities enables SincKAN to mitigate the spectral bias observed in PIKANs, thereby making PIKANs more robust and capable of solving PDEs that traditional PINNs may struggle with. Additionally, we conducted a series of experiments to validate SincKAN’s interpolation capabilities and assess their performance as a replacement for MLP and KANs in PINNs.
Our specific contributions can be summarized as follows:

- 1.
We propose the Sinc Kolmogorov-Arnold Networks, a novel network that excels in handling singularities.
- 2.
We propose several approaches based on classical techniques of Sinc methods that can enhance the robustness and performance of SincKAN.
- 3.
We conducted a series of experiments to demonstrate the performance of SincKAN in approximating a function and PIKANs.

The paper is structured as follows:
In Section 2, we briefly introduce the PINNs, discuss Sinc numerical methods, and provide a detailed explanation of SincKAN. In Section 3 we compare our SincKAN with several networks including MLP, modified MLP Wang et al. (), KAN, ChebyKAN in several diverse benchmarks including smooth functions, discontinuous functions, and boundary layer problems. In Section 4, we conclude the paper and discuss the remaining limitations and directions for future research.

## 2 Methods

### 2.1 Physics-informed neural networks (PINNs)

We briefly review the physics-informed neural networks (PINNs) Raissi et al. () in the context of
inferring the solutions of PDEs. Generally, we consider time-dependent PDEs for $\bm{u}$ taking the form

$$ $\displaystyle\partial_{t}\bm{u}+\mathcal{N}[\bm{u}]=0,\quad t\in[0,T],\ \bm{x} \in\Omega,$ (1) $\displaystyle\bm{u}(0,\bm{x})=\bm{g}(\bm{x}),\quad\bm{x}\in\Omega,$ $\displaystyle\mathcal{B}[\bm{u}]=0,\quad t\in[0,T],\ \bm{x}\in\partial\Omega,$ $$

where $\mathcal{N}$ is the differential operator, $\Omega$ is the domain of grid points, and $\mathcal{B}$ is the boundary operator. When considering time-independent PDEs, $\partial_{t}\bm{u}\equiv 0$.

The ambition of PINNs is to approximate the unknown solution $\bm{u}$ to the PDE system Eq. 1, by optimizing a neural network $\bm{u}^{\theta}$, where $\theta$ denotes the trainable parameters of the neural network. The constructed loss function is:

$$ $\mathcal{L}(\theta)=\mathcal{L}_{ic}(\theta)+\mathcal{L}_{bc}(\theta)+\mathcal {L}_{r}(\theta),$ (2) $$

where

$$ $\displaystyle\mathcal{L}_{r}(\theta)=\frac{1}{N_{r}}\sum_{i=1}^{N_{r}}\left| \partial_{t}\bm{u}^{\theta}\left(t_{r}^{i},\bm{x}_{r}^{i}\right)+\mathcal{N} \left[\bm{u}^{\theta}\right]\left(t_{r}^{i},\bm{x}_{r}^{i}\right)\right|^{2},$ (3) $\displaystyle\mathcal{L}_{ic}(\theta)=\frac{1}{N_{ic}}\sum_{i=1}^{N_{ic}}\left |\bm{u}^{\theta}\left(0,\bm{x}_{ic}^{i}\right)-\bm{g}\left(\bm{x}_{ic}^{i} \right)\right|^{2},$ $\displaystyle\mathcal{L}_{bc}(\theta)=\frac{1}{N_{bc}}\sum_{i=1}^{N_{bc}}\left |\mathcal{B}\left[\bm{u}^{\theta}\right]\left(t_{bc}^{i},\bm{x}_{bc}^{i}\right )\right|^{2},$ $$

corresponds to the three equations in Eq. 1 individually; $\bm{x}_{ic}^{i},\bm{x}_{bc}^{i},\bm{x}_{r}^{i}$ are the sampled points from the initial constraint, boundary constraint, and residual constraint, respectively; $N_{ic},N_{bc},N_{r}$ are the total number of sampled points for each constraint, correspondingly. Note that in Raissi et al. (), $u^{\theta}\left(x\right)=\mathbf{MLP}\left(x\right)$.

### 2.2 Sinc numerical methods

The Sinc function is defined as(^1^11In engineering, they define Sinc function as $\mathrm{Sinc}(x)=\frac{\sin(\pi x)}{\pi x}$)

$$ $\mathrm{Sinc}(x)=\frac{\sin(x)}{x},$ (4) $$

the Sinc series $S(j,h)(x)$ used in Sinc numerical methods is defined by:

$$ $S(j,h)(x)=\frac{\sin[(\pi/h)(x-jh)]}{(\pi/h)(x-jh)},$ (5) $$

then the Sinc approximation for a function $f$ defined on the real line $\mathbb{R}$ is given by

$$ $f(x)\approx\sum_{j=-N}^{N}f(jh)S(j,h)(x),\quad x\in\mathbb{R},$ (6) $$

where $h$ is the step size with the optimal value $\sqrt{\pi d/\beta N}$ provided in Theorem 1, and $2N+1$ is the degree of Sinc series.

Thanks to Sinc function’s beautiful properties including the equivalence of semidiscrete Fourier
transform Trefethen (), its approximation as a nascent delta function, etc., Sinc numerical methods have become a technique for solving a wide range of linear and nonlinear problems arising from scientific and engineering applications including heat transfer Lippke (), fluid mechanics Abdella (), and solid mechanics Abdella et al. (). But Sinc series are the orthogonal basis defined on $\left(-\infty,\infty\right)$ which is impractical for numerical methods. To use Sinc numerical methods, one should choose a proper coordinate transformation based on the computing domain $\left(a,b\right)$ and an optimal step size based on the target function $f$. However, manually changing the network to meet every specific problem is impractical and wasteful. In the following of this section, we will introduce current techniques used in Sinc numerical methods. Then in Section 2.3, we will unfold Sinc numerical methods to meet machine learning.

Convergence theorem

###### Theorem 1

Sugihara & Matsuo ()

Assume $\alpha,\beta,d>0$, that

(1) $f$ belongs to $H^{1}\left(\mathcal{D}_{d}\right)$, where $H^{1}$ is the Hardy space and $\mathcal{D}_{d}=\{z\in\mathbb{C}\ |\ \left|\Im z\right|<d\}$;

(2) $f$ decays exponentially on the real line, that is, $|f(x)|\leq\alpha\exp(-\beta|x|),\ \forall x\in\mathbb{R}$.

Then we have

$$ $\sup_{-\infty<x<\infty}\left|f(x)-\sum_{j=-N}^{N}f(jh)S(j,h)(x)\right|\leq CN^ {1/2}\exp\left[-(\pi d\beta N)^{1/2}\right]$ (7) $$

for some constant $C$, where the step size $h$ is taken as

$$ $h=\left(\frac{\pi d}{\beta N}\right)^{1/2}.$ (8) $$

Theorem 1 indicates that the exponential convergence of Sinc approximation on the real line depends on the parameters $d,\beta,N$ which are determined by the target function $f$. Thus, in Sinc numerical methods, researchers set specific parameters for specific function $f$ Sugihara & Matsuo (); Mohsen () or set them by bisection Richardson & Trefethen (). Note that both approaches require the target function $f$, but in machine learning, $f$ is usually unknown.

On a general interval $(a,b)$

Practical problems generally require approximating on an interval $(a,b)$ instead of the entire real line $\mathbb{R}$. To implement Sinc methods on general functions, we have to transform the interval $(a,b)$ to $\mathbb{R}$ with a properly selected coordinate transformation, i.e. we define a transformation $x=\psi\left(\xi\right)$ such that $\psi:\left(-\infty,+\infty\right)\rightarrow\left(a,b\right)$. Then Eq. 6 is replaced by

$$ $f(\psi(\xi))\approx\sum_{j=-N}^{N}f(\psi(jh))S(j,h)(\xi),\quad-\infty<\xi<\infty,$ (9) $$

where $h$ is the step size with the optimal value $\sqrt{\pi d^{\prime}/\beta^{\prime}N}$ provided in Theorem 2.
The following theorem states that Theorem 1 still holds with some different $\alpha,\beta$, and $d$ after the coordinate transformation.

###### Theorem 2

Sugihara & Matsuo ()

Assume that, for a variable transformation $x=\psi(\xi)$, the transformed function $f(\psi(\xi))$ satisfies assumptions 1 and 2 in Theorem 1 with some $\alpha^{\prime},\beta^{\prime}$ and $d^{\prime}$. Then we have

$$ $\sup_{a\leq x\leq b}\left|f(x)-\sum_{j=-N}^{N}f(\psi(jh))S(j,h)\left(\psi^{-1} (x)\right)\right|\leq CN^{1/2}\exp\left[-(\pi d^{\prime}\beta^{\prime}N)^{1/2}\right]$ $$

for some $C$, where the step size $h$ is taken as

$$ $h=\left(\frac{\pi d^{\prime}}{\beta^{\prime}N}\right)^{1/2}$ $$

This theorem suggests the possibility that even a function $f$ with an end-point singularity can be approximated successfully by Eq. 9 with a suitable choice of transformation.

Furthermore, we empirically demonstrate the merits of Sinc methods in Fig. 1 via numerical results generated by Chebfun Driscoll et al. () for Chebyshev and cubic spline interpolation and Sincfun Richardson & Trefethen () for Sinc interpolation.

Figure: (a) $f(x)=\sqrt{x}$
Refer to caption: x1.png

###### Theorem 1

###### Theorem 2

### 2.3 Sinc Kolmogorov-Arnold Network (SincKAN)

Suppose $\mathbf{\Phi}=\{\phi_{p,q}\}$ is the matrix of univariate functions where $p=1,2,\dots n_{in},q=1,2,\dots n_{out}$. Then the L-layers Kolmogorov-Arnold Networks can be defined by:

$$ $\mathrm{KAN}({\bm{x}})=\left(\mathbf{\Phi}_{L-1}\circ\mathbf{\Phi}_{L-2}\circ \cdots\circ\mathbf{\Phi}_{1}\circ\mathbf{\Phi}_{0}\right){\bm{x}},\quad{\bm{x} }\in\mathbb{R}^{d}.$ (10) $$

In vanilla KAN Liu et al. (), every univariate function $\phi$ is approximated via a summation with cubic spline:

$$ $\phi_{\mathrm{spline}}\left(x\right)=w_{b}\mathrm{silu}\left(x\right)+w_{s} \left(\sum_{i}c_{i}B_{i}(x)\right),$ (11) $$

where $c_{i},w_{b},w_{s}$ are trainable parameters, and $B_{i}$ is the spline. Intuitively, to replace cubic interpolation with Sinc, we can define:

$$ $\phi_{\mathrm{single}}\left(x\right)=\sum_{i=-N}^{N}c_{i}S(i,h)(x),$ (12) $$

where $c_{i}$ is trainable parameters, if $h$ is set to the optimal value Eq. 8, the optimal approximation of $f(x)$ in Eq. 6 is $c_{i}^{*}=f(ih),\forall i=-N,\cdots,N$. However, to replace the interpolation method successfully, the aforementioned techniques require further investigations:

##### Optimal h ℎ h italic_h .

As we discussed in Section 2.2, it is impractical to set a single optimal $h$ in machine learning frameworks. Thus in SincKAN, we propose an extension to the Sinc approximation with a mixture of different step sizes $h_{j}$:

$$ $\phi_{\mathrm{multi}}\left(x\right)=\sum_{j=1}^{M}\sum_{i=-N}^{N}c_{i,j}S(i,h_ {j})(x),$ (13) $$

where $c_{i,j}$ are trainable parameters. Stenger () states that, if the chosen $h$ is larger than the optimal value predicted by Eq. 8, the interpolation is less accurate near the origin and more accurate farther away from the origin; if the chosen $h$ is smaller than the optimal value predicted by Eq. 8, the interpolation is more accurate near the origin and less accurate farther away from the origin. Herein, combining different $h$ with adaptive weights can result in a more accurate approximation than the optimal $h$ and doesn’t need to calculate the optimal $h$. Thus, compared to Eq. 12, expanding the approximation by a summation of several different $h_{j}$ can not only avoid determining $h$ for every specific function but also improve the accuracy.

##### Coordinate transformation.

Another challenge is the choice of coordinate transformation which is also problem-specific Stenger (). Let’s inherent the notation of Section 2.2, and suppose $\mathcal{X}=\{x_{i}\}_{i=1}^{N}$ is the ordered set of input points with $x_{1}\leq x_{2}\leq\cdots\leq x_{N}$, then we can define the open interval $(a,b)$ by $a=x_{1}-\epsilon,b=x_{N}+\epsilon$, where $\epsilon$ is a chosen number, and $\xi_{1}=\psi^{-1}(x_{1}),\xi_{N}=\psi^{-1}(x_{N})$. Thus, the interval of input points changes to $\left[\xi_{1},\xi_{N}\right]$ from $\left[x_{1},x_{N}\right]$. However, if we perform such a transformation for every sub-layer, the scale of the input becomes larger and inconsistent, making the network converge slower Ioffe (). Herein, we argue that the normalization for the input of every layer is necessary, and Bozorgasl & Chen () already utilizes the batch normalization Ioffe () on every layer to enhance the performance of KANs. In our SincKAN, a normalizing transformation, $\phi(x)=\frac{x-\mu}{\sigma}$ is introduced, where $\sigma$ is the scaling factor and $\mu$ is the shifting factor. Composing $\phi$ and $\psi$ still meets the condition of $\psi$ in Theorem 2 and the transformed function $f(\psi\circ\phi^{-1}(\xi))$ also satisfies assumptions 1 and 2 in Theorem 1 with $\alpha^{\star},\beta^{\star}$ and $d^{\star}$. Consequently, the optimal value of the step size $h$ is changed to $\sqrt{\pi d^{\star}/\beta^{\star}N}$.

Let us define the normalized coordinate transformation $\gamma^{-1}(x):=\phi\circ\psi^{-1}(x)$ such that $\gamma^{-1}:\left(a,b\right)\rightarrow\left(-\infty,\infty\right)$ and $\left[x_{1},x_{N}\right]\rightarrow\left[\frac{\xi_{1}-\mu}{\sigma},\frac{\xi_
{N}-\mu}{\sigma}\right]$, where $\sigma,\mu$ satisfies $\left[\frac{\xi_{1}-\mu}{\sigma},\frac{\xi_{N}-\mu}{\sigma}\right]\subset\left
[-1,1\right]$. Herein, instead of coordinate transformation $\psi$, we use normalized coordinate transformation $\gamma$ in SincKAN. As for the changing of optimal $h$ with different $\sigma,\mu$, the summation of different $h_{j}$ implemented in SincKAN makes it easy to meet the fixed scale $\left[-1,1\right]$ i.e. we can depend the set $\{h_{j}\}$ on the domain $\left[-1,1\right]$ regardless of $\sigma,\mu$.

3) Exponential decay

In Theorem 2, $f$ should satisfy the condition of exponential decay which constrains that $f(-\infty)=f(+\infty)=0$. To utilize the Sinc methods on general functions, Richardson & Trefethen () interpolates the subtraction $g-f$ instead of $f$, where $g$ is the linear function that has the same value as $f$ at the endpoints. In our SincKAN, we introduce a learnable linear function as a skip-connection to approximate the subtraction.

Finally, combining the three aforementioned approaches, we can define our learnable activation function in SincKAN:

$$ $\phi_{\mathrm{sinc}}\left(x\right)=c_{1}x+c_{2}+\sum_{j=1}^{M}\sum_{i=-N}^{N}c _{i,j}S(i,h_{j})(\gamma^{-1}(x)),$ (14) $$

where $c_{1},c_{2},c_{i,j}$ are the learnable parameters $S$ is the Sinc function and $\gamma$ is the normalized transformation.

## 3 Experiments

In this section, we will demonstrate the performance of SincKANs through experiments including approximating functions and solving PDEs, compared with several other representative networks: Multilayer perceptron (MLP) which is the classical and most common network used in PINNs, Modified MLP which is proposed to project the inputs to a high-dimensional feature space to enhance the hidden layers’ capability, KAN which is proposed to replace MLP in AI for Science, and ChebyKAN which is proposed to improve the performance by combining KAN with the known approximation capabilities of Chebyshev polynomials and has already been examined in Shukla et al. (). In this paper, we choose to implement the normalized transformation $\gamma(x)=\tanh(x)$, and the linear skip connection $w_{1}\in\mathbb{R}^{n_{in}\times n_{out}},w_{2}\in\mathbb{R}^{n_{out}}$. Note that, we also observed the instability of ChebyKAN highlighted in Shukla et al. (), and the ChebyKAN used in our experiments is actually the modified ChebyKAN proposed by Shukla et al. () which has $\tanh$ activation function between each layers. The rest details of the used networks are provided in Appendix F. The other details including hyperparameters can be found in Appendix C. All code and data-sets accompanying this manuscript are available on GitHub at [https://github.com/DUCH714/SincKAN](https://github.com/DUCH714/SincKAN).

### 3.1 Learning for approximation

Approximating a function by given data is the main objective of KANs with applications in identifying relevant
features, revealing modular structures, and discovering symbolic formulas Liu et al. (). Additionally, in deep learning, the training process of a network can be regarded as approximating the map between complex functional spaces, thus the accuracy of approximation directly indicates the capability of a network. Therefore, we start with experiments on approximation to show the capability of SincKAN and verify whether SincKAN is a competitive network. In this section, to have consistent results with KAN, we inherit the metric RMSE which is used in KAN.

Sinc numerical methods are recognized theoretically and empirically as a powerful tool when dealing with singularities. However, in machine learning instead of numerical methods, we argue that SincKAN can be implemented in general cases. To demonstrate that SincKAN is robust, we conducted a series of experiments on both smooth functions and singularity functions: in Table 1 the first four functions show the results of analytic functions including functions in which cubic splines interpolation is good at, and the last four functions show the results of functions with singularities which Sinc interpolation is good at. The details of the used functions can be found in Appendix A.

Table: Table 1: RMSE of functions for approximation

Figure: (a)
Refer to caption: x4.png

The results in Table 1 show that SincKAN achieves impressive performance on low-frequency functions (sin-low), high-frequency functions (sin-high), continuous but non-differentiable functions (multi-sqrt) and discontinuous functions (piece-wise). Furthermore, the last function (spectral-bias) is designed to evaluate the ability to address the prevalent phenomenon of spectral bias by Rahaman et al. (), and the corresponding result in Table 1 indicates that SincKAN maximally alleviates the spectral bias. Additionally, we also evaluate every network on the finer grid to test their generalization, we put the results on Appendix E. The comparison of cost is in Appendix G. Furthermore, as an important aspect of understanding SincKANs, we plot some interior $\phi$ in Appendix D.

#### 3.1.1 Selecting h

Utilizing a set of $\{h_{i}\}$ instead of a single step size $h$ is a novel approach that we developed specifically for SincKANs. To evaluate the effectiveness of this approach, in this section, we design a comprehensive experiment. Suppose $h_{min}=\min\{h_{i}\},h_{max}=\max\{h_{i}\}$, based on the discussion of Section 2.3, the ideal case is the optimal $h^{*}=\sqrt{\pi d^{\star}/\beta^{\star}N}\in\left(h_{min},h_{max}\right)$. In the experiments, we provide two types of the set:

- 1.
inverse decay $\{h_{i}\}_{i=1}^{M}$: $h_{i}=1/ih_{0}$,
- 2.
exponential decay $\{h_{i}\}_{i=1}^{M}$: $h_{i}=1/h_{0}^{i}$.

Thus the hyperparameters of the set $\{h_{i}\}$ are the base number $h_{0}$ and the number of the set $M$.

We train SincKAN on sin-low and sin-high functions with $M=1,6,12,24$ for inverse decay and $M=1,2,3$ for exponential decay and $h_{0}=2.0,\pi,6.0,10.0$. Besides, the number of discretized points $N_{points}$ and the degree $N_{degree}$ ($N_{degree}=2N+1$, where $N$ is the notation in Eq. 14) also influence the performance of SincKAN for different $\{h_{i}\}_{i=1}^{M}$, we empirically set $N_{degree}=100$, and $N_{points}=5000$ in this experiment.

The results are illustrated in Fig. 3 for inverse decay and Fig. 8 for exponential decay, and the details including the corresponding error bars are shown in Appendix I. For sin-low function, the best RMSE $1.49e\mbox{-}4\pm 8.74e\mbox{-}5$ is observed with $h_{0}=10.0$ and $M=1$; for sin-high function, the best RMSE $4.60e\mbox{-}3\pm 3.70e\mbox{-}4$ is observed in inverse decay with $h_{0}=10.0$ and $M=24$.

The experiments use two divergent Fourier spectra ($4\pi$, and $400\pi$), and get extremely different optimal hyperparameters $M$. The results conclude that: to obtain an accurate result, one can use small $M$ with large $h_{0}$ for a low-frequency function and can use large $M$ with large $h_{0}$ for a high-frequency function. Although the experiments show that the SincKAN is sensitive to the approximated function, the RMSE is accurate enough for both sin-low and sin-high in inverse decay with $M=6$ and $h_{0}=10.0$.

Figure: (a)
Refer to caption: x8.png

#### 3.1.2 Relationship between degree and size of data

In Sinc numerical methods, the number of the sampled points is equal to the degree because each degree requires a corresponding value $f(jh)$ at the point $jh$ i.e. $N_{degree}=N_{points}$. However, in SincKAN, $N_{degree}=N_{points}$ is impractical, and it is also not necessary because Sinc numerical methods can be regarded as a single-layer representation while our SincKAN is a multi-layer representation where the multi-layer representation has an exponentially increasing capability with depth Yarotsky (). To explore the relationship between degree and size of data, we train our SincKAN with different $N_{degree}$ and $N_{points}$. The results are shown in Appendix H and reveal that it is unnecessary to maintain $N_{degree}=N_{points}$ in SincKANs.

### 3.2 Learning for PIKANs

Solving PDEs is the main part of scientific computing, and PINNs are the representative framework for solving PDEs by neural networks. In this section, we solve a series of challenging PDEs to show the performance of SincKAN. At first, we select several classical PDEs to verify the robustness of SincKAN, the results are shown in Table 2, and the details of the PDEs can be found in Appendix B.

Table: Table 2: Relative L2 error for chosen PDE problems

#### 3.2.1 Boundary layer problem

To intuitively show the performance of SincKAN compared with other networks, we conducted additional experiments on the boundary layer problem:

$$ $u_{xx}/\epsilon+u_{x}=0,x\in[0,1]$ (15) $$

with the exact solution $u(x)=\exp(-\epsilon x)$. As $\epsilon$ increases, the width of the boundary layer (left) decreases, and the complexity of learning increases. The results shown in Table 3 and Fig. 4 reveal that SincKANs can handle the boundary layer effectively, while other networks struggle when $\epsilon$ is large.

Table: Table 3: Relative L2 error for different $\epsilon$ in Eq. 15

Figure: (a)
Refer to caption: x10.png

#### 3.2.2 Ablation study

Compared with Sinc numerical methods, SincKANs have a normalized transformation; compared to KANs, SincKANs have a skip connection with linear functions. However, the Sinc numerical methods also have some choices of coordinate transformations and KANs also have a skip connection with SiLU functions. In this section, we conduct an ablation study on SincKANs with non-normalized transformation and the SiLU skip connection to verify the effect of the two proposed modules. This experiment uses Burger’s equation Eq. 28 and time-dependent nonlinear equation Section B.5.
Table 4 shows the results of the ablation study where $\psi(x)=\log(\frac{x-a}{b-x})$, $\gamma(x)=\tanh(x)$, $\mathrm{Linear}(x)=w_{1}x+w_{2}+\phi(x)$, and $\mathrm{SiLU}(x)=w_{b}\mathrm{silu}(x)+w_{s}\phi(x)$. The non-normalized transformation performs poorly, even compared to the cases without transformations. Although the linear skip connection is not the best for both equations, it is the most stable approach for SincKANs.

**Table 4: Relative L2 error for ablation study**
| $\psi$ | $\gamma$ | Linear | SiLU | T-nonlinear | Burgers’ equation |
| --- | --- | --- | --- | --- | --- |
| ✘ | ✘ | ✘ | ✘ | $9.20e\mbox{-}4\pm 4.31e\mbox{-}4$ | $1.57e\mbox{-}2\pm 4.31e\mbox{-}3$ |
| ✘ | ✔ | ✘ | ✘ | $5.80e\mbox{-}4\pm 1.89e\mbox{-}4$ | $6.21e\mbox{-}4\pm 1.96e\mbox{-}4$ |
| ✘ | ✔ | ✔ | ✘ | $2.44e\mbox{-}4\pm 6.08e\mbox{-}5$ | $3.12e\mbox{-}3\pm 2.48e\mbox{-}3$ |
| ✘ | ✔ | ✘ | ✔ | $1.60e\mbox{-}4\pm 3.17e\mbox{-}5$ | $8.90e\mbox{-}3\pm 6.76e\mbox{-}3$ |
| ✔ | ✘ | ✔ | ✘ | $1.11e\mbox{-}2\pm 1.17e\mbox{-}3$ | $7.36e\mbox{-}2\pm 2.02e\mbox{-}2$ |
| ✔ | ✘ | ✘ | ✔ | $1.30e\mbox{-}2\pm 4.00e\mbox{-}4$ | $1.12e\mbox{-}1\pm 6.80e\mbox{-}3$ |

## 4 Conclusion

In this paper, we propose a novel network called Sinc Kolmogorov-Arnold Networks (SincKANs). Inspired by KANs, SincKANs leverage the Sinc functions to interpolate the activation function and successfully inherit the capability of handling singularities. To set the optimal $h$, we propose the multi-$h$ interpolation, and the corresponding experiments indicate that this novel approach is the main reason for SincKANs’ superior ability in approximating complex smooth functions; to choose a proper coordinate transformation for machine learning, we propose the normalized transformation which prevents slow convergence.; to satisfy the decay condition, we introduce the skip-connection with learnable linear functions. After tackling the aforementioned challenges, SincKANs become a competitive network that can replace the current networks used for PINNs.

We begin with training on approximation problems to demonstrate the capability of SincKANs. The results reveal that SincKANs excel in most experiments with other networks. However, directly approximating the target function is an impractical objective for almost all machine learning tasks. After verifying the capability, we turn to solving PDEs in the PINNs framework. Although the SincKANs achieve impressive performance in approximation tasks for solving all chosen PDEs, SincKANs merely have the best accuracy on boundary layer problems, due to the oscillations caused by the inaccuracy of derivatives.

##### Limitations

Approximating derivative by Sinc numerical methods is always inaccurate in the neighborhood of the Sinc end-points. To address this problem, Stenger () suggested using Lagrange polynomial to approximate the derivative instead of straightforwardly calculating the derivative of Sinc polynomials, Wu et al. () used several discrete functions to replace the derivative of Sinc polynomials, etc. Unfortunately, to the best of our knowledge, there isn’t an approach that can be implemented in our SincKAN when we demand the derivatives of SincKAN in PIKANs. Herein, to alleviate the inaccuracy, we choose small $h_{0}$, small $M$, and small $N$ so that SincKAN can solve PDEs, otherwise, the solution will have oscillations (see Appendix J). Such kind of setting limits the capability of SincKAN and we argue that this is the main reason that SincKAN can obtain good results but not the best results for some cases. Furthermore, the inaccuracy limits SincKAN in solving high-order problems such as Korteweg–De Vries equations, and Kuramoto–Sivashinsky equations.

##### Future

As the accuracy of approximating the derivative decreases with the order of derivative increases if the PDE merely requires the first derivatives, then the SincKANs will release the limitation to have larger enough $h_{0}$, $M$, and $N$ and improve the performance. In literature, to avoid calculating the high-order derivatives, MIM Lyu et al. (); Li et al. () is proposed to use the mixed residual method which transforms a high-order PDE into a first-order PDE system. SincKANs can implement this approach to calculate several first-order derivatives instead of the high-order derivatives so that SincKANs can have accurate estimations for the residual loss. Furthermore, replacing the automatic differentiation Cen & Zou (); Yu et al. () by other operators is also an expected research.

#### Acknowledgments

T. Yu is supported by China Scholarship Council No. 202308090008. J. Yang is supported by the National Natural Science Foundation of China (NSFC) Grant No. 12271240, the NSFC/Hong Kong RGC Joint Research Scheme (NSFC/RGC 11961160718), and the Shenzhen Natural Science Fund (No. RCJC20210609103819018). We thank Vladimir Fanakov for fruitful discussion and constructive suggestions.

## Appendix A Explicit expression of functions

The following functions are used in Table 1.

- 1.
sin-low
$f(x)=\sin(4\pi x),\quad x\in[-1,1]$
(16)
- 2.
sin-high
$f(x)=\sin(400\pi x),\quad x\in[-1,1]$
(17)
- 3.
bl
$f(x)=e^{-100x},\quad x\in[0,1]$
(18)
- 4.
sqrt
$f(x)=\sqrt{x},\quad x\in[0,1]$
(19)
- 5.
double-exponential
$f(x)=\frac{x(1-x)\mathrm{e}^{-x}}{(1/2)^{2}+(x-1/2)^{2}},\quad x\in[0,1]$
(20)
- 6.
multi-sqrt
$f(x)=x^{1/2}(1-x)^{3/4},\quad x\in[0,1]$
(21)
- 7.
piece-wise
$f(x)=\left\{\begin{aligned} &\sin(20\pi x)+x^{2},&\quad x\in[0,0.5]\\
&0.5xe^{-x}+|\sin(5\pi x)|,&\quad x\in[0.5,1.5]\\
&\log(x-1)/\log(2)-\cos(2\pi x),&\quad x\in[1.5,2]\end{aligned}\right.$
(22)
- 8.
spectral-bias
$f(x)=\left\{\begin{aligned} &\sum_{k=1}^{4}\sin(kx)+5,&\quad x\in[-1,0]\\
&\cos(10x),&\quad x\in[0,1]\\
\end{aligned}\right.$
(23)

## Appendix B Details of PDEs

### B.1 1D problems

### B.2 Perturbed boundary value problem

We consider the singularly perturbed second-order boundary value problem (perturbed in Table 2):

$$ $\epsilon u_{xx}-u_{x}=f(x),\quad x\in[-1,1].$ (24) $$

In specific cases, the problem has exact solutions, in this paper, we choose $f(x)=-1$, and the exact solution is

$$ $u(x)=1+x+\frac{e^{\frac{x}{\epsilon}}-1}{e^{\frac{1}{\epsilon}}-1},$ (25) $$

where $\epsilon=0.01$ in our experiments.

### B.3 Nonlinear problem

We consider the nonlinear boundary value problem (nonlinear in Table 2):

$$ $\displaystyle-u_{xx}+\frac{u_{x}}{x}+\frac{u}{x^{2}}$ $\displaystyle=\frac{\left(-41x^{2}+34x-1\right)\sqrt{x}}{4}-2x+\frac{1}{x^{2}} ,\quad x\in[0,1]$ (26) $\displaystyle u(0)-2u_{x}(0)$ $\displaystyle=1,$ $\displaystyle 3u(1)+u_{x}(1)$ $\displaystyle=9,$ $$

with the exact solutions

$$ $u(x)=x^{5/2}(1-x)^{2}+x^{3}+1.$ (27) $$

### B.4 Burgers

We consider the Burgers’ equation (Burgers’ equation in Table 4):

$$ $\frac{\partial u}{\partial t}+u\frac{\partial u}{\partial x}-\nu\frac{\partial ^{2}u}{\partial x^{2}}=0,\quad x\in[-1,1],t\in[0,0.1].$ (28) $$

with Dirichlet boundary condition, and the exact solution is

$$ $u=\frac{a}{2}-\frac{a\tanh\left(\frac{a(x-at/2)}{4\nu}\right)}{2},$ (29) $$

where $a=0.5,\nu=0.01$ in our experiments.

### B.5 T-nonlinear problem

We consider the time-dependent nonlinear problem (T-nonlinear in Table 4):

$$ $\begin{gathered}u_{t}=\frac{x+2}{t+1}u_{x},\quad x\in[-1,1],t\in[0,0.1].\\ u(x,0)=\cos(x+2),\\ u(1,t)=\cos(3(t+1)),\end{gathered}$ (30) $$

with the exact solution:

$$ $u(x,t)=\cos((t+1)(x+2)).$ (31) $$

### B.6 Convection-diffusion

We consider the 1-D convection-diffusion equation with periodic boundary conditions (used in Appendix J):

$$ $\begin{gathered}u_{t}+au_{x}-\epsilon u_{xx}=0,x\in\left[-1,1\right],t\in\left [0,0.1\right],\\ u(x,0)=\sum_{k=0}^{5}\sin\left(k\pi x\right),\end{gathered}$ (32) $$

with the analytic solution

$$ $u(x,t)=\sum_{k=0}^{5}\sin\left(k\pi x-ka\pi t\right)e^{-\epsilon k^{2}\pi^{2}t},$ (33) $$

where $\epsilon=0.01$, and $a=0.1$in our experiments.

### B.7 2D problems

#### B.7.1 Boundary layer

We consider the 2-D boundary layer problem (bl-2d in Table 2):

$$ $u_{xx}/\alpha_{1}+u_{x}+u_{yy}/\alpha_{2}+u_{y}=0,$ (34) $$

with the exact solution

$$ $u(x,y)=\exp(-\alpha_{1}x)+\exp(-\alpha_{2}y),$ (35) $$

where $\alpha_{1}=\alpha_{2}=100$ in our experiments.

Figure: (a)
Refer to caption: x13.png

#### B.7.2 Navier stokes equations

We consider the Taylor–Green vortex (ns-tg-u and ns-tg-v in Table 2):

$$ $\begin{gathered}\nabla\cdot\bm{u}=0,\quad t\in[0,T],\ \bm{x}\in\Omega,\\ \partial_{t}\bm{u}+\bm{u}\cdot\nabla\bm{u}=-\nabla p+\nu\triangle\bm{u},\quad t \in[0,T],\ \bm{x}\in\Omega,\\ \end{gathered}$ (36) $$

where $\bm{u}=(u,v)$,with the exact solution

$$ $\begin{gathered}u=-\cos(x)\sin(y)\exp(-2\nu t)\\ v=\sin(x)\cos(y)\exp(-2\nu t)\\ p=-\left(\cos(2x)+\sin(2y)\right)\exp(-4\nu t)/4\end{gathered}$ (37) $$

with $T=1$, $\nu=1/400$ in our experiments. After dimensionless, $\bm{x}\in[0,1]^{2}$.

## Appendix C Experiment details

Totally, in our experiments, the Adam Kingma & Ba () optimizer is used with the exponential decay learning rate. The MLP and modified MLP are equipped with the tanh activations and Xavier initialization inherited from Raissi et al. ().

### C.1 Approximation

The hyperparameters of used networks are shown in Table 6.

- •
For the 1-D problem, we generate the training dataset by uniformly discretizing the input interval to 5000 points and train the network with 3000 points randomly sampled from the training dataset for each iteration. In total, We train every network with $10^{5}$ iterations. Additionally, to evaluate the generalization, we generate the testing (fine) dataset by uniformly discretizing the input interval to 10000 points.

### C.2 PIKANs

The hyperparameters of used networks are shown in Table 7.

- •
For time-independent 1-D problems, we generate the training dataset by uniformly discretizing the input interval to 1000 points, then train the network with 500 points randomly sampled from the training dataset for each iteration. In total, We train every network with $1.5\times 10^{6}$ iterations.
- •
For time-dependent 1-D problems, we generate the training dataset by uniformly discretizing the spatial dimension to 1000 points and the temporal dimension to 11 points, then train the network with 5000 points randomly sampled from the training dataset for each iteration. In total, We train every network with $1.5\times 10^{6}$ iterations.
- •
For time-independent 2-D problems, we generate the training dataset by uniformly discretizing every dimension to 100 points, then train the network with 5000 points randomly sampled from the training dataset for each iteration. In total, We train every network with $1.5\times 10^{6}$ iterations.
- •
For time-dependent 2-D problems, we generate the training dataset by uniformly discretizing every spatial dimension to 100 points and the temporal dimension to 11 points, then train the network with 50000 points randomly sampled from the training dataset for each iteration. In total, We train every network with $1.5\times 10^{6}$ iterations.

## Appendix D Interior approximation of SincKAN

Figure: (a) Eq. 34 for approximation
Refer to caption: x16.png

## Appendix E Approximation on fine grids

As we discussed in Appendix C, we additionally evaluate every network on fine grids. And due to the oscillations discussed in Appendix J, Table 5 reveals the weak generalization of SincKANs demands further research, although the applications of approximating a function don’t strongly require this capability.

Table: Table 5: RMSE evaluated on fine grids

## Appendix F Details of other networks

### F.1 MLP

Multilayer Perceptron (MLP) is the neural network consisting of fully connected neurons with a nonlinear activation function, and can be represented simply by:

$$ $\operatorname{MLP}(x)=\left(W^{L-1}\circ\sigma\circ W^{L-2}\circ\sigma\circ \cdots\circ W^{1}\circ\sigma\circ W^{0}\right){\bm{x}}$ (38) $$

where $W^{i}(x)=W_{i}x+b_{i}$, $W_{i}\in\mathbb{R}^{m_{i}\times n_{i}}$ is a learnable matrix, $b_{i}\in\mathbb{R}^{m_{i}}$ is a learnable bias, $\sigma$ is the chosen nonlinear activation function, and $L$ is the depth of MLP.

### F.2 Modified MLP

Modified MLP is an upgraded network of MLP inspired by the transformer networks. It introduces two extra features and has a skip connection with them:

$$ $\begin{gathered}U=\sigma(W^{L+1}{\bm{x}}),\quad,V=\sigma(W^{L+2}{\bm{x}}), \quad H^{1}=\sigma(W^{0}{\bm{x}}),\\ H^{i+1}=(1-\sigma(W^{i}H^{i}))*U+\sigma(W^{i}H^{i})*V,\quad i=1,\cdots,L,\\ \operatorname{ModifiedMLP}({\bm{x}})=W^{L+3}H^{L+1},\end{gathered}$ (39) $$

where $*$ is the element-wise multiplication.

### F.3 KAN

Kolmogorov-Arnold Network (KAN) is a novel network proposed recently that aims to be more accurate and interpretable than
MLP. The main difference is KAN’s activation functions are learnable: suppose $\mathbf{\Phi}=\{\phi_{p,q}\}$ is the matrix of univariable functions where $p=1,2,\dots n_{in},q=1,2,\dots n_{out}$ and $\theta$ represents the trainable parameters. The KAN can be defined by:

$$ $\operatorname{KAN}({\bm{x}})=\left(\mathbf{\Phi}_{L-1}\circ\mathbf{\Phi}_{L-2} \circ\cdots\circ\mathbf{\Phi}_{1}\circ\mathbf{\Phi}_{0}\right){\bm{x}},\quad{ \bm{x}}\in\mathbb{R}^{d},$ (40) $$

where

$$ $\mathbf{\Phi}_{l}({\bm{x}}^{(l)})=\left\{\sum_{i=1}^{n^{(l)}_{in}}\phi_{j,i} \left(x_{i}^{(l)}\right)\right\}_{j=1}^{n^{(l)}_{out}},\quad\forall l=0,1, \cdots,L-1$ (41) $$

where ${\bm{x}}\in\mathbb{R}^{n_{in}^{(l)}},\mathbf{\Phi}_{l}({\bm{x}}^{(l)})\in
\mathbb{R}^{n_{out}^{(l)}}$.
To approximate every single activation function $\phi$, KAN utilizes the summation of basis function and spline interpolation:

$$ $\phi\left(x\right)=w_{b}\mathrm{silu}\left(x\right)+w_{s}\left(\sum_{i}c_{i}B_ {i}(x)\right),$ (42) $$

where $c_{i},w_{s},w_{b}$ are learnable, and $B_{i}$ is the spline.

### F.4 ChebyKAN

ChebyKAN utilizes the Chebyshev polynomials to construct the learnable activation function $\phi$ in KAN. And the modified ChebyKAN embeds the $\tanh$ activation function between every layer. Thus the ChebyKAN used in our experiments can be defined by:

$$ $\operatorname{ChebyKAN}({\bm{x}})=\left(\mathbf{\Phi}_{L-1}\circ\tanh\circ \mathbf{\Phi}_{L-2}\circ\cdots\circ\mathbf{\Phi}_{1}\circ\tanh\circ\mathbf{ \Phi}_{0}\circ\tanh\circ\right){\bm{x}},\quad{\bm{x}}\in\mathbb{R}^{d},$ (43) $$

where $\mathbf{\Phi}$ has the same definition of Eq. 41 with different univariable function

$$ $\phi=\sum_{i}c_{i}T_{i}(x),$ (44) $$

where $T_{i}$ is the $i$th Chebyshev polynomial.

## Appendix G Computational Cost

### G.1 Training

In Eq. 14, SincKAN has an additional summation on several $h$, so the trainable coefficients $c$ are $M$ times larger than KAN and ChebyKAN. However, the training time is not only dependent on the number of total parameters, thus, we demonstrate the cost of training for approximation in Table 6, and demonstrate the cost of training for PDEs in Table 7. In Table 6 and Table 7, we use ’$\text{depth}\times\text{width}$’ to represent the size for MLP and modified MLP; ’$\text{width}\times\text{degree}$’ to represent the size for KAN and ChebyKAN; and ’$\text{width}\times\text{degree}\times M$’ to represent the size for SincKAN. Note that we train the network in two environments distinguished by two superscripts:

†: training on single NVIDIA A100-SXM4-80GB with CUDA version: 12.4.

‡: training on single NVIDIA A40-48GB with CUDA version: 12.4.

**Table 6: Computational cost for approximation^†**
| Network | Size | Training rate (iter/sec) | Referencing time (ms) | Parameters |
| --- | --- | --- | --- | --- |
| MLP | $10\times 100$ | $9.89\times 10^{2}$ | $1.62\times 10^{1}$ | $81101$ |
| modified MLP | $10\times 100$ | $9.13\times 10^{2}$ | $2.92\times 10^{1}$ | $81501$ |
| KAN | $8\times 8$ | $1.15\times 10^{3}$ | $1.01\times 10^{2}$ | $160$ |
| ChebyKAN | $40\times 40$ | $1.29\times 10^{3}$ | $3.11\times 10^{1}$ | $3280$ |
| SincKAN | $8\times 100\times 6$ | $1.29\times 10^{3}$ | $2.06\times 10^{1}$ | $9696$ |

**Table 7: Computational cost for train PIKANs**
| Function name | Network | Size | Training rate (iter/sec) | Parameters |
| --- | --- | --- | --- | --- |
| boundary layer^‡ | MLP | $10\times 100$ | $6.47\times 10^{2}$ | 81101 |
| modified MLP | $10\times 100$ | $3.55\times 10^{2}$ | 81501 |  |
| KAN | $8\times 8$ | $1.33\times 10^{3}$ | 160 |  |
| ChebyKAN | $40\times 40$ | $1.89\times 10^{3}$ | 3280 |  |
| SindcKAN | $8\times 8\times 1$ | $1.27\times 10^{3}$ | 194 |  |
| perturbed^‡ | MLP | $10\times 100$ | $6.85\times 10^{2}$ | 81101 |
| modified MLP | $10\times 100$ | $3.59\times 10^{2}$ | 81501 |  |
| KAN | $8\times 8$ | $1.27\times 10^{3}$ | 160 |  |
| ChebyKAN | $40\times 40$ | $1.07\times 10^{3}$ | 3280 |  |
| SincKAN | $8\times 8\times 1$ | $1.25\times 10^{3}$ | 194 |  |
| nonlinear^‡ | MLP | $10\times 100$ | $4.62\times 10^{2}$ | 81101 |
| modified MLP | $10\times 100$ | $3.07\times 10^{2}$ | 81501 |  |
| KAN | $8\times 8$ | $1.56\times 10^{3}$ | 160 |  |
| ChebyKAN | $40\times 40$ | $1.53\times 10^{3}$ | 3280 |  |
| SincKAN | $8\times 4\times 1$ | $1.54\times 10^{3}$ | 130 |  |
| bl-2d^‡ | MLP | $10\times 100$ | $2.39\times 10^{2}$ | 81201 |
| modified MLP | $10\times 100$ | $1.96\times 10^{2}$ | 81801 |  |
| KAN | $8\times 8$ | $3.46\times 10^{2}$ | 240 |  |
| ChebyKAN | $40\times 40$ | $4.95\times 10^{2}$ | 4920 |  |
| SincKAN | $8\times 20\times 1$ | $2.97\times 10^{2}$ | 570 |  |
| ns-tg^† | MLP | $10\times 100$ | $1.71\times 10^{2}$ | 81503 |
| modified MLP | $10\times 100$ | $1.51\times 10^{2}$ | 82303 |  |
| KAN | $8\times 8$ | $2.55\times 10^{2}$ | 480 |  |
| ChebyKAN | $40\times 40$ | $3.83\times 10^{3}$ | 9840 |  |
| SincKAN | $8\times 8\times 1$ | $2.77\times 10^{2}$ | 550 |  |

### G.2 Referencing

As the referencing cost doesn’t depend on the task i.e. the loss function, the results are evaluated on the model trained by approximation task and the results can be found in Table 6. The results reveal that although SincKAN has much more parameters than KAN and ChebyKAN, SincKAN is faster when referencing. Note that the referencing is slower than training because we compile the training procedure by JAX Bradbury et al. ().

## Appendix H Results of different degree

In this experiment, we train our SincKAN on spectral-bias function on $N=8,16,32,64,100,300$ and $N_{points}=100,500,1000,5000,10000$ with the inverse decay $\{h_{i}\}_{i=1}^{M}$ in $M=6$ and $h_{0}=7.0$.
Moreover, we set the batch size $N_{batch}=N_{points}/4$ to adapt to the changing of $N_{points}$. The results are shown in Table 8 and Fig. 7. Additionally, Fig. 7(a) shows that our neural scaling law is $\mathrm{RMSE}\propto G^{-4}$ compared to the best scaling law $\mathrm{RMSE}\propto G^{-3}$ claimed in KAN Liu et al. ().

Table: Table 8: RMSE for different degree and $N_{points}$

Figure: (a) $N_{points}=100$
Refer to caption: x18.png

## Appendix I Results of selected h

Table 9 and Table 10 show the results of selected $\{h_{i}\}$ in details. However, there are so many hyperparameters that may be adjusted when $h_{i}$ is larger. For example, for the large $h$ on fine grids, we argue that $N_{degree}=100$ may not exploit the capability fully. Thus, we conducted an extra experiment with 5000 grid points, $\{h_{i}\}=\{1/10,1/100,1/1000\}$, and $N_{degree}=500$. For sin-low, the RMSE is $7.92e\mbox{-}4\pm 4.21e\mbox{-}4$, and for the sin-high, the RMSE is $2.32e\mbox{-}3\pm 2.74e\mbox{-}4$. It shows that for sin-high, the SincKAN can obtain a more accurate result if we further tune the hyperparameters.

Figure: (a)
Refer to caption: x23.png

Table: Table 9: RMSE for different $\{h_{i}\}$ with inverse decay

Table: Table 10: RMSE for different $\{h_{i}\}$ with exponential decay

## Appendix J Oscillations of SincKAN

We conducted the experiments on convection-diffusion equations (Eq. 32) with $h_{0}=2.0,10.0$, $N=8,100$, and $M=1,6$. Except $h_{0}=2.0$ and $N=8$, the inaccuracy of derivatives makes SincKAN unstable with the loss diverging. We choose some figures plotted in Fig. 9 to show the oscillations that limit the improvement of SincKANs.

Figure: (a) $h_{0}=2.0,N=8,M=1$
Refer to caption: x25.png

## Appendix K Metrics

In this paper, we use two metrics. For interpolation, we inherit the RMSE metric from KAN Liu et al. (), the formula is :

$$ $\mathrm{RMSE}=\sqrt{\frac{1}{N}\sum_{i=1}^{N}(y_{i}-\hat{y}_{i})^{2}};$ (45) $$

for PIKANs, we utilize the relative L2 error which is the most common metric used in PINNs:

$$ $\mathrm{RelativeL2}=\frac{\|\bm{y}-\hat{\bm{y}}\|_{2}}{\|\bm{y}\|_{2}},$ (46) $$

where $y$ is the target value, and $\hat{y}$ is the predicted value

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 5
presentation: 3
questions: * What exactly is f(jh) in equation (6)? Is it simply the value of f at jh? Is equation (6) interpolating f(x) using those values? If so, do we need to know those values precisely to approximate f(x) with this formulation?
* Could other problems with singularities be tested? Does the proposed method also show significant advantages and accurately approximate solutions in all cases where singular phenomena appear? Could it be applied to other boundary layer problems or higher-dimensional problems beyond two dimensions?
* For line 140, it might be better to use the \paragraph function for terms like “Convergence theorem” or “On a general interval (a, b)” to save space.
soundness: 3
strengths: The study explores how the recently developed KAN model can better handle singular problems from the perspective of Physics-Informed Neural Networks (PINNs).
summary: This paper presents a new model that applies sinc interpolation to the previously proposed KAN model. It discusses the advantages of the proposed model for function fitting and solving PDEs in a Physics-informed manner.
weaknesses: I believe the main goal of this paper is to improve existing PINN-based models through the development of sincKAN. However, as shown in Table 2, it's unclear if sincKAN demonstrates convincingly better performance than other existing models across various PDE problems. Clearly, as mentioned in the PI-KAN model, there are specific advantages and disadvantages to using KAN for solving PDEs, such as computational cost or training speed. If sincKAN does not consistently outperform across all datasets, as seen in Table 2, what advantages does sincKAN offer over using standard MLP or modified MLP to train Physics-informed Loss? While sincKAN appears to perform better on boundary layer problems, as shown in Table 3, the paper’s title and focus could have been more aligned with boundary layer issues if that is its primary strength. Additionally, are there recent studies related to PINNs specifically aimed at solving boundary layer problems? What would happen if epsilon increased to a larger value, such as 10^8? It might also help to include a basic explanation of KAN’s underlying mechanism for readers unfamiliar with the model, as certain concepts could be difficult to grasp without this context.

---

confidence: 5
contribution: 1
initial_rating: 5
presentation: 3
questions: Can sincKAN help with some pathological issues of PINNs? Would be interesting to explore; see for example Wang, S., Teng, Y., and Perdikaris, P. Understanding and Mitigating Gradient Flow Pathologies in Physics- Informed Neural Networks. SIAM Journal on Scientific Computing, 43(5):A3055–A3081, 2021a. or Wang, S., Sankaran, S., and Perdikaris, P. Respecting causal- ity is all you need for training physics-informed neural networks. arXiv preprint arXiv:2203.07404, 2022a. or the more recent paper Challenges in Training PINNs: A Loss Landscape Perspective by Rathore et al.

I still don't see the real advantages of theorems 1 and 2, since the optimal value of step is not uses in sincKAN. Also, I advise authors formally add another theorem to Coordinate transformation section to prove for the composition, the theory still holds (it should be straightforward). Also, if a function needs to be Hardy, does that mean the guarantees are only valid for differentiable functions f? It would be good to discuss the implementations.
soundness: 3
strengths: The paper and idea are well-written. It is clear what authors try to implement and they are also rigor in deriving equation 14, which is a foundation for their sincKAN network. 
The choice of boundary layer example is smart and shows the good performance of sincKAN in the presence of singularities (specially pronounced in large epsilon (lower width of boundary layer or larger Reynolds, which is more complicated).
summary: This paper proposes KANs with sinc function as basis to represent univariate function, to which they refer as sincKan. They first discuss about the sinc function and its convergence properties and then extend it such that it becomes a viable and practical workhorse in the context of KANs. Equation 14 clarifies their contributions, namely, 1) choosing step size without the knowledge of function f (by using different h values, larger and smaller than the optimal one), 2) coordinate transformation (a composition of transformation from finite to infinite domain and a normalizing transformation), and 3) enforcing exponential decay (by considering g-f instead of f with a linear g). First property manifests itself in use of ci,j and hj, second one is hidden in $\gamma^{-1}$, and the third property is $c1x+c2$. They use both function approximation and solution of PDEs (similar to PINNs, which enforces the solution as soft constraint in the loss function) as numerical experiments and show lower error in most cases compared to relevant baselines.
weaknesses: The contribution of this paper seems limited to me and I hope authors clarify this in the rebuttal period. 
Using more advanced basis functions for KANs is a meaningful research direction but currently it seems to me no single method can be seen as the best solution for all PDEs. In fact sincKAN may have better performance in the presence of singularities but may not be the best option otherwise. In the current version, I can't derive a concrete conclusion as which version of KANs to use for what type of problems (within PDE domain or even beyond). In fact, in Table 2, sincKAN is not the best option in some cases.
In an analogy to more conventional deep learning papers, the use of Relu activation functions with CNN enable them to achieve very good performance: it helped with vanishing gradient problem, it helped with efficiency of implementations, and also empirical success (for instance AlexNet). sincKAN does not seem to be such an abrupt idea. The theorems provided do not really guarantee the performance (specially considering the fact that authors use a range of step sizes instead of the optimal value due to practical reasons). 
In this vein, authors discuss how sincKAN can help with spectral bias, but not much discussion is done on other issues for PINNs e.g. issue for causality (which makes them problematic in chaotic PDEs--which by the way is good to add to examples), and many other issues (like plateauing loss, etc.).
Finally, in "vanilla" KANs paper there are some arguments on the advantage of using B-splines, which enable them the use of grid related tricks including grid extension and grid update tricks. In other variants also there are some interesting merits; it's hard to say sincKAN still has all this merits and in addition to them provide more robustness in the case of singularities.

---

confidence: 2
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. For MLP architectures, how were the architectures chosen? Since the problems are small and toy-ish, over-parameterization may cause a higher final loss.
soundness: 3
strengths: 1. Applying sinc to replace splines in KAN is a novel approach. Additionally, the authors address the practical challenges of selecting step sizes using multiple step sizes.
summary: The paper builds upon recently proposed architecture KANs that are meant to be more parameter efficient than MLPs and provide additional benefits like interpretability. The authors replace the splines in KANs with sinc to propose SincKAN. Furthermore, the paper presents experiments in PINNs (using KANs) to demonstrate potential benefits of SincKAN.
weaknesses: 1. The PDEs selected for the experiments are generally unsuitable to demonstrate the benefits of PIKANs. The chosen problems are extremely small and these problems can be solved using any numerical PDE solver. Since PINNs / PIKANs don't provide any practical advantage in small problems, it would be interesting to see if the benefits proposed here can be demonstrated in real problems.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: Question:
1. Can't one use a Sinc interpolator directly for all the function approximation tasks?

Sugestions:
1. The experiments of section 3.1 do not seem so relevant for practical applications. It seems that what might be most interesting would be to take a selection of complicated PDE/ODE data (using any off the shelf differential equation solver) and use these instead. This would demonstrate the value of SincKANs in practical circumstances, and in situations of more than 1 dimension. As it stands, these benchmarks feel artificial and irrelevant for real world application.
2. More scrutiny of the physics-informed losses would be beneficial. Some plots of solutions and errors across the poorer performing methods might help understand why they are performing badly. Is it that boundary conditions are not being adhered to? Maybe there are regions of high PDE loss in the resulting solution? Perhaps small changes (e.g. weighing boundary conditions effectively) might lead to improved performance.
soundness: 3
strengths: 1. KANs are recent and might have much scope for optimisation. The interpolation scheme in KANs seems to be an important point to consider in practical use, so it is good that the authors draw attention to this.
2. Given the recency of KANs, the study provides more independent comparisons between KANs and MLPs which might be of interest to the wider community.
3. The authors do not straightforwardly substitute Sinc interpolants into the KAN architecture, but properly consider its principled application in terms of optimal step sizes, required exponential decay of the approximated function and how to circumvent this etc.
summary: The work proposes using Sinc interpolations replace the splines used within recently popularised Kolomogorov Arnold Networks (KANs). The work outlines foundational background required of Sinc interpolation and demonstrates some nice properties of Sinc interpolation on some specific benchmarks. It then outlines certain properties that must hold for effective Sinc interpolation and incorporates these within its KAN framework. Benchmarks are performed against against baselines comprising of two different multilayer perceptron architectures, and two other KAN architectures. These experiments encompass a range of function approximation tasks for manufactured functions, and physics-informed tasks.
weaknesses: 1. The manufactured solutions used in the learning for approximation secion (3.1) are arbitrary and seemingly not very relevant to real world applications. They are both manufactured, and all are one dimensional.
2. While the performance of SincKANs is good in physics-informed settings, it feels like some scrutiny of the results is missing. There are many reasons why a physics-informed problem might fail to converge. It might not be that the reason for better performance in a physics-informed problem is a better architecture.

Meta-review: This paper proposes the use of sinc interpolation in Kolmogorov-Arnold networks (KANs) with the goal of achieving better function interpolation and for the solution of PDEs in the context of physics-informed neural networks (PINNs). I thank the authors and the reviewers for their discussions. From these, I believe improving KANs is an interesting research direction, in particular, within the context of solving PDEs in PINNs. 

The major criticism of this paper is in terms of the experimental evaluation, with regards to, e.g., the focus of the paper (should it be on boundary layer problems?), the lack of robustness, more visualizations, better explanations of the KANs underlying mechanism, and the unsuitability of the benchmarks problems used for evaluation. 

I strongly disagree with the belief of some reviewers (Reviewer mr4i, other reviewers at ICLR and in other top ML venues) that an ML paper must be shown to be the best across all datasets and (worse) that those problem where the method does not do well should not be reported. In fact, as a community, we need to commend authors to do otherwise. However, I believe the point on the unsuitability of the benchmarks used is an important one. Although the authors have  added new results, some of the reviewers remained unconvinced. Because of this and the additional issues outlined above, I recommend rejection.

Decision: Reject