On the expressiveness and spectral bias of KANs

## Abstract

Abstract Kolmogorov-Arnold Networks (KAN) [ 46 ] were very recently proposed as a potential alternative to the prevalent architectural backbone of many deep learning models, the multi-layer perceptron (MLP). KANs have seen success in various tasks of AI for science, with their empirical efficiency and accuracy demonstrated in function regression, PDE solving, and many more scientific problems. In this article, we revisit the comparison of KANs and MLPs, with emphasis on a theoretical perspective. On the one hand, we compare the representation and approximation capabilities of KANs and MLPs. We establish that MLPs can be represented using KANs of a comparable size. This shows that the approximation and representation capabilities of KANs are at least as good as MLPs. Conversely, we show that KANs can be represented using MLPs, but that in this representation the number of parameters increases by a factor of the KAN grid size. This suggests that KANs with a large grid size may be more efficient than MLPs at approximating certain functions. On the other hand, from the perspective of learning and optimization, we study the spectral bias of KANs compared with MLPs. We demonstrate that KANs are less biased toward low frequencies than MLPs. We highlight that the multi-level learning feature specific to KANs, i.e. grid extension of splines, improves the learning process for high-frequency components. Detailed comparisons with different choices of depth, width, and grid sizes of KANs are made, shedding some light on how to choose the hyperparameters in practice.

## 1 Introduction

Recently, in [], a novel architecture called Kolmogorov-Arnold Networks (KANs) was proposed as a potentially more accurate and interpretable alternative to standard multi-layer perceptron (MLP) [, ]. The KAN architecture leverages the Kolmogorov-Arnold representation theorem (KART) [, ] to parameterize functions. KANs share the same fully connected structures as MLPs, while putting learnable activation functions on edges as opposed to a fixed activation function on nodes for MLPs. B-splines are used to parameterize the learned nonlinearity and in practice one can go beyond the two-layer construction indicated by KART. KANs can be conceptualized as a hybrid of splines and MLPs, and the combination of compositional nonlinearity with 1D splines contributes to both the accuracy and interpretability of KANs.

In this article, we study the approximation theory and spectral bias of KANs and compare them with MLPs. Specifically, we show that any MLP with the ReLU^k activation function can be reparameterized as a KAN with a comparable number of parameters. This establishes that the representation and approximation power of KANs is at least as great as that of MLPs. On the other hand, we also show that any KAN (without SiLU non-linearity) can be represented using an MLP. However, the number of parameters in the MLP representation is larger by a factor proportional to the grid size of the KAN. While we do not know if our construction is optimal, this suggests that KANs with a large grid size may be more efficient at parameterizing certain classes of functions than MLPs. We combine these results with existing optimal approximation rates for MLPs to obtain approximation rates for KANs on Sobolev spaces.

Next, we study the spectral bias phenomenon in the training of KANs. Standard MLPs with ReLU activations (or even tanh) are known to suffer from the spectral bias [, , ], in the sense that they will fit low-frequency components first. This is in contrast to traditional iterative numerical methods like the Jacobi method which learn high frequencies first []. Although the spectral bias acts as a regularizer which improves performance for machine learning applications [, , , , ], for scientific computing applications it may be necessary to learn high-frequencies as well.
To alleviate the spectral bias, high-frequency information has to be encoded using methods like Fourier feature mapping
[, , , ], or one needs to use nonlinear activation functions more similar to traditional methods; see for example the hat activation function [] which resembles a finite element basis. We theoretically consider a single KAN layer and show that it does not suffer from the spectral bias by analyzing gradient descent for minimizing a least squares objective. Although our analysis is necessarily highly simplified, we argue that it provides some evidence and intuition showing that KANs will have a reduced spectral bias compared with MLPs.

Finally, we study the spectral bias of KANs experimentally on a wide variety of problems, including 1D frequency fitting, fitting samples from a higher-dimensional Gaussian kernel, and solving the Poisson equation with a high-frequency solutions. Based upon the results of our experiments, KANs consistently suffer less from the spectral bias. This also helps explain why KANs are more subject to noise and overfitting, as observed in
[], while grid coarsening (the opposite of grid extension) helps increase the spectral bias and reduces overfitting. Notice that the highest frequency that a single layer of KANs can learn is restricted to the number of grid points we use. The compositional structure associated with the depth also plays a role in learning the different frequencies. We study the effect of depths, widths, and grid sizes of KANs for the spectral bias, and draw conclusions about best practices when choosing KAN hyperparameters.

#### Our contribution.

Our goal in this paper is to theoretically compare the KAN architecture with the commonly used MLP architecture. Our specific contributions are as follows:

- •
We compare the approximation and representation ability of KANs and MLPs. We show that KANs are at least as expressive as MLPs. We use this to obtain approximation rates for KANs on Sobolev spaces.
- •
We theoretically analyze gradient descent applied to optimize the least squares loss using a single layer KAN. Based upon this analysis, we argue that KANs, unlike MLPs, do not suffer much from a spectral bias.
- •
We provide numerical experiments demonstrating that KANs exhibit less of a spectral bias than MLPs on a variety of problems. This validates our theory and also provides a partial explanation for the success of KANs on problems in scientific computing.

## 2 Prior Work

A theory on the approximation ability of KANs in the function class of compositionally smooth functions was proposed as KAT in []. A convergence result independent of the dimension was obtained, leveraging the approximation theory of 1D splines. Compared with the universal approximation theorem of MLPs, KANs take advantage of the intrinsically low-dimensional compositional representation of underlying functions. This result shares an analogy to the rate in generalization error bounds of finite training samples, for a similar space studied for regression problems; see [, ], and also specifically for MLPs with ReLU activations []. On the other hand, for general Sobolev or Besov spaces, sharp approximation rates have been obtained for ReLU-MLPs (and more generally MLPs with most piecewise polynomial activation functions) [, , ]. These rates exhibit the curse of dimensionality, which is unavoidable due to the fact that Sobolev and Besov spaces with fixed smoothness are very large in high dimensions.

There are also subsequent works exploring other parametrizations of activation functions in the KAN formulation, including special polynomials [, , ], rational functions [], radial basis function [, ], Fourier series [], and wavelets [, ]. Active follow-up research focuses on applying KANs to various domains, such as partial differential equations [, , ] and operator learning [, , ], graphs [, , , ], time series [, , , ], computer vision [, , , , , ], and various scientific problems [, , , , , , , , , ]. In the KAN 2.0 paper [], multiplication was introduced as a built-in modularity of KANs and the connection between KANs and scientific problems was further established in terms of identifying important features, modular structures, and symbolic formulas. In this article, we focus on the original B-spline formulation, one of the reasons being its alignment with continual learning and adaptive learning; see also [].

The spectral bias of MLPs has been studied both experimentally and theoretically in a variety of works, see for instance [, , , , , , , , , , ] and the references therein. This phenomenon is proposed as explaining the regularizing effect of stochastic gradient descent [, ]. So far, the spectral bias of KANs has not been studied in detail, and it is this gap that we aim to close in this work.

## 3 Representation and Approximation

In this section, we study how to represent ReLU^k MLPs using KANs with degree $k$ splines and vice-versa. Using this, we establish approximation rates for KANs from the corresponding results for MLPs.

### 3.1 Review of the KAN Architecture

We recall the following definitions of the Kolmogorov-Arnold Network (KAN) architecture introduced in []. We refer to [] for a thorough treatment of all details mentioned here.

#### KAN architecture

The Kolmogorov-Arnold representation theorem (KART) states that any multivariate continuous function on a bounded domain can be represented as a finite composition of univariate continuous functions and addition. Specifically for a continuous $f:[0,1]^{n}\to\mathbb{R}$, there exists continuous 1D functions $\phi_{q,p},\Phi_{q}$ such that (see for instance [])

$$ $f(\mathbf{x})=f(x_{1},\cdots,x_{n})=\sum_{q=1}^{2n+1}\Phi_{q}\left(\sum_{p=1}^ {n}\phi_{q,p}(x_{p})\right).$ (1) $$

Inspired by KART corresponding to a two-layer neural network representation, the authors in [] unified the inner and outer functions via the proposed KAN layers and generalized the idea of composition to arbitrary depths. A KAN can be represented by an integer array $[n_{0},n_{1},\cdots,n_{L}]$. Here $L$ is the number of layers and $n_{0}$ and $n_{L}$ are input and output dimensions respectively. $n_{l}$ denotes the number of neurons in the $l$-th neuron layer. Usually, we choose $n_{1}=\cdots=n_{L-1}=W$ as the width and refer to $L$ as the depth of the KAN. Layer $l$ maps a vector $\mathbf{x}_{l}\in\mathbb{R}^{n_{l}}$ to $\mathbf{x}_{l+1}\in\mathbb{R}^{n_{l+1}}$, via nonlinear activations and additions, in the matrix form as

$$ $\mathbf{x}_{l+1}=\underbrace{\begin{pmatrix}\phi_{l,1,1}(\cdot)&\phi_{l,2,1}( \cdot)&\cdots&\phi_{l,n_{l},1}(\cdot)\\ \phi_{l,1,2}(\cdot)&\phi_{l,2,2}(\cdot)&\cdots&\phi_{l,n_{l},2}(\cdot)\\ \vdots&\vdots&&\vdots\\ \phi_{l,1,n_{l+1}}(\cdot)&\phi_{l,2,n_{l+1}}(\cdot)&\cdots&\phi_{l,n_{l},n_{l+ 1}}(\cdot)\\ \end{pmatrix}}_{\mathbf{\Phi}_{l}}\mathbf{x}_{l}.$ (2) $$

The overall network can be written as a composition of $L$ KAN layers as

$$ ${\rm KAN}(\mathbf{x})=(\mathbf{\Phi}_{L-1}\circ\cdots\circ\mathbf{\Phi}_{1} \circ\mathbf{\Phi}_{0})\mathbf{x}.$ (3) $$

Each activation function $\phi=\phi_{l,i,j}$ in (2) is parametrized by a linear combination of $k$-th order B-splines and a SiLu nonlinearity. Namely

$$ $\phi(x)=w_{b}x/(1+e^{-x})+\sum_{i=0}^{G+k-1}c_{i}B_{i}(x),$ (4) $$

where $G$ is the grid size, $B_{i}$ are spline basis and $c_{i}$, $w_{b}$ are trainable coefficients. The ranges of the grid points of the B-splines are updated on the fly, based on the range of the output of the previous layer. For $\phi$ on a bounded interval $[a,b]$, consider the uniform grid $\{t_{0}=a,t_{1},t_{2},\cdots,t_{G}=b\}$, uniformally extended to $\{t_{-k},\cdots,t_{-1},t_{0},\cdots,t_{G},t_{G+1},\cdots,t_{G+k}\}$. On this grid there are $G+k$ B-spline basis functions, with the $i^{\rm th}$ B-spline $B_{i}(x)$ being non-zero only on $[t_{-k+i},t_{i+1}]$.

#### Grid extension

A very important structure of KANs proposed in [] is the grid extension technique, inheriting the multi-level fine-graining from splines. One can first train KANs with fewer parameters to a desired accuracy, before making the spline grids finer with initialization from the coarser grid and continuing the training procedure. Grid extensions for all splines in a KAN can be performed independently and adaptively. We highlight that this technique is specific to the choice of splines as parameterizations of activation functions, and is particularly useful in the training process; see for example the discussion of spectral bias in the next section.

#### Approximation theory, KAT

We recollect the approximation theory established for compositionally smooth functions in [].

###### Theorem 3.1 .

Let $\mathbf{x}=(x_{1},x_{2},\cdots,x_{n})$.
Suppose that a function $f(\mathbf{x})$ admits a representation

$$ $f=(\mathbf{\Phi}_{L-1}\circ\mathbf{\Phi}_{L-2}\circ\cdots\circ\mathbf{\Phi}_{1 }\circ\mathbf{\Phi}_{0})\mathbf{x}\,,$ (5) $$

as in (3), where each one of the $\Phi_{l,i,j}$ is $(k+1)$-times continuously differentiable. Then there exists a constant $C$ depending on $f$ and its representation, such that we have the following approximation bound in terms of the grid size $G$: there exist $k$-th order B-spline functions $\Phi_{l,i,j}^{G}$ such that for any $0\leq m\leq k$, we have the bound for the $C^{m}$-norm

$$ $\|f-(\mathbf{\Phi}^{G}_{L-1}\circ\mathbf{\Phi}^{G}_{L-2}\circ\cdots\circ \mathbf{\Phi}^{G}_{1}\circ\mathbf{\Phi}^{G}_{0})\mathbf{x}\|_{C^{m}}\leq CG^{- k-1+m}\,.$ (6) $$

###### Theorem 3.1 .

### 3.2 Reparametrization of KANs and MLPs

Next, we compare the approximation capabilities of KANs and MLPs. In particular, we have the following result showing that any ReLU^k MLP can be exactly represented using a KAN with degree $k$ splines which is only slightly larger. We remark that this includes the case where $k=1$, i.e. the popular and standard ReLU activation function.

###### Theorem 3.2 .

Let $\Omega\subset\mathbb{R}^{d}$ be a bounded domain. Suppose that a function $f:\mathbb{R}^{d}\rightarrow\mathbb{R}$ can be represented by an MLP with width $W\geq 1$, depth $L\geq 1$, and activation function $\sigma_{k}=\max(0,x)^{k}$ for $k\geq 1$. Then there exists a KAN $g$ with width $W$, depth at most $2L$, and grid size $G=2$ with $k$-th order B-spline functions such that

$$ $g(x)=f(x)$ (7) $$

for all $x\in\Omega$.

On the other hand, we may ask whether every KAN can also be expressed using an MLP of a comparable size. If the functions $\phi_{l,i,j}$ in the KAN representation have weight $w_{b}\neq 0$ in (3), then it is clear that we can not represent the resulting KAN using an MLP with activation $\sigma_{k}$, since the smooth function $f(x)=x/(1+e^{-x})$ is not a polynomial. Essentially, the SiLU nonlinearity cannot be captured by a ReLU^k MLP. However, if this nonlinearity is not present, then a KAN can be represented using a ReLU^k MLP.

###### Theorem 3.3 .

Suppose that a function $f:[0,1]^{d}\rightarrow\mathbb{R}$ can be represented by a KAN with width $W\geq 1$, depth $L\geq 1$, and grid size $G$ with $k$-th order B-spline functions. Assume also that the weight $w_{b}=0$ in (3) for all of the activations $\phi_{l,i,j}$ appearing in the KAN. Then there exists an MLP with width $(G+2k+1)W$, depth at most $2L$, and activation function $\sigma_{k}=\max(0,x)^{k}$ which represents $f$.

We remark that the width in this result scales like $O(GW)$ so that the number of parameters in the MLP scales like $O(G^{2}W^{2}L)$, while the number of parameters in the KAN scales like $O(GW^{2}L)$. This appears to indicate that KANs with a large number of breakpoints at each node may be more efficient in expressing certain functions than MLPs, although we don’t know whether the preceding theorem is sharp.

Using these results, we can draw conclusions about the approximation capabilities of KANs by leveraging existing results about MLPs (see for instance [, , , , , , , , , , , ]). For example, we have the following result giving optimal approximation rates for very deep KANs on Sobolev spaces (see for instance [] for the background on Sobolev spaces).

###### Corollary 3.4 .

Let $\Omega\subset\mathbb{R}^{d}$ be a bounded domain with smooth boundary, $s>0$ and $1\leq p,q\leq\infty$ be such that $1/q-1/p<s/d$. This guarantees that the compact Sobolev embedding $W^{s}(L_{q}(\Omega))\subset\subset L_{p}(\Omega)$ holds.

Let $W_{0}:=W_{0}(d)$ be a fixed width (depending upon the input dimension $d$). Then for any $f\in W^{s}(L_{q}(\Omega))$ and any $L\geq 1$, there exists a KAN $g$ with width $W_{0}$, depth $L$, and grid size $G=2$ with $k$-th order $B$-spline functions such that

$$ $\|f-g\|_{L_{p}(\Omega)}\leq CL^{-2s/d},$ (8) $$

where $C$ is a constant independent of $L$.

This result, which follows immediately from Theorem 3.2 and the approximation rates for ReLU (and more generally piecewise polynomial) neural networks derived in [], shows that very deep KANs attain an exceptionally good approximation rate on Sobolev spaces. In particular, in terms of the number of parameters $P$ they attain an approximation rate of $O(P^{-2s/d})$, while a classical (even non-linear) method of approximation can only attain a rate of $O(P^{-s/d})$ []. This phenomenon, which is often called superconvergence [], also occurs for very deep ReLU^k networks. However, it comes at the cost of parameters which are not encodable using a fixed number of bits and thus is not practically realizable [, ].

###### Remark 3.5 .

Compared with the approximation theory KAT in Theorem 3.1 stated for compositionally smooth functions where the curse of dimensionality does not appear in the convergence rate, Corollary 3.4 is stated for a larger class of Sobolev functions.

###### Theorem 3.2 .

###### Theorem 3.3 .

###### Corollary 3.4 .

###### Remark 3.5 .

## 4 Spectral Bias

In this section, we study the spectral bias of KANs and compare it to that of MLPs. The spectral bias, or frequency principle, refers to the observation that neural networks trained with gradient descent tend to be biased toward lower frequencies, i.e. that lower frequencies are learned first. This phenomenon has been well-documented and studied for MLPs, see for instance [, , , , , , , , , ], and it is our purpose here to develop an analogous theory for KANs. We remark that although the spectral bias is considered a form of regularization which is desirable for machine learning tasks [, , , , ], for scientific computing applications it is typically important to capture all frequencies and so the spectral bias may negatively affect the performance of neural networks for such applications [, , , ].

### 4.1 Spectral bias theory for shallow KANs

We consider the spectral bias properties of KANs with a single layer. This theory is very similar to the theory developed in [, , ] for the spectral bias of single hidden layer MLPs. The key observation is that a single layer KAN is a linear model. In particular, we see that if $L=1$ then the KAN applied to an input $\mathbf{x}\in\mathbb{R}^{d}$ is (for simplicity, we consider the case of a KAN without the SiLu non-linearity)

$$ ${\rm KAN}(\mathbf{x},\theta)_{i}=\sum_{j=1}^{d}\sum_{l=1}^{G+k-1}c_{ijl}B_{l}( \mathbf{x}_{j}),$ (9) $$

where $\theta=\{c_{ijl}\}$ are the parameters of the KAN. Here $i=1,...,d^{\prime}$ where $d^{\prime}$ is the dimension of the output, $j=1,...,d$, and $l=1,...,G+k-1$. Note that the only parameter here is the grid size $G$, since the width is determined by the input and output dimensions.

Based upon this, we can analyze least squares fitting with shallow KANs. In particular, let $\Omega=[-1,1]^{d}$ be the (symmetric) unit cube in $\mathbb{R}^{d}$, let $f:\Omega\rightarrow\mathbb{R}^{d^{\prime}}$ be a target function we are trying to learn, and consider the (continuous) least squares regression loss

$$ $L(\theta)=\int_{\Omega}\|f(\mathbf{x})-{\rm KAN}(\mathbf{x},\theta)\|^{2}d \mathbf{x}.$ (10) $$

Due to the representation (9), this loss function is quadratic in the parameters $\theta$. Let $M$ denote the corresponding Hessian matrix, i.e. so that

$$ $L(\theta)=(1/2)\theta^{T}M\theta+b^{T}\theta.$ $$

This Hessian matrix (indexed by $i,j,l$) is given by

$$ $M_{(i,j,l),(i^{\prime},j^{\prime},l^{\prime})}=\begin{cases}\int_{\Omega}B_{l} (\mathbf{x}_{j})B_{l^{\prime}}(\mathbf{x}_{j^{\prime}})d\mathbf{x}&i=i^{\prime }\\ 0&i\neq i^{\prime}.\end{cases}$ (11) $$

The convergence of gradient descent on the least squares regression is determined by the eigendecomposition of the Hessian matrix $M$ which is estimated in the following theorem. The theorem is essentially a generalization of the fact that the Gram matrix of the B-spline basis is well conditioned (see for instance [], Theorem 4.2 in Chapter 5).

###### Theorem 4.1 .

Given a single hidden layer KAN with grid size $G$, degree $k$ B-splines, input dimension $d$ and output dimension $d^{\prime}$, let $M$ denote the Hessian matrix defined in (11) corresponding to the least squares fitting problem (10). Then the eigenvalues $0\leq\lambda_{1}(M)\leq\cdots\leq\lambda_{N}(M)$ (here $N=(G+k-1)dd^{\prime}$) satisfy

$$ $\frac{\lambda_{N}(M)}{\lambda_{d^{\prime}(d-1)+1}(M)}\leq Cd$ (12) $$

for a constant $C$ depending only on the spline degree $k$.

Theorem 4.1 shows that away from $d^{\prime}(d-1)$ eigenvectors the matrix $M$ is well conditioned. This means that gradient descent will converge at the same rate in all directions orthogonal to these $d^{\prime}(d-1)$ eigenvectors. Note that since the number of eigenvectors we must remove is independent of the grid size $G$, we expect that when $G$ is relatively large most components of the KAN will converge at roughly the same rate toward the solution. Thus the KAN with a large number of grid points will not exhibit the same spectral bias toward low frequencies seen by MLPs. We remark that in constrast the Hessian associated with a two layer ReLU MLP with width $n$ has a condition number which scales like $n^{4}$ [], which explains the strong spectral bias exhibited by ReLU MLPs.

###### Remark 4.2 .

We note that the $d^{\prime}(d-1)$ eigenvectors which must be excluded is not an artifact of the proof. In fact, this is due to the fact that the KAN parameterization is not unique. Indeed, the constant function $f(x)=1$ can be parameterized in $d$ different ways by using the B-splines in each of the $d$ different directions. This ambiguity gives rise to directions in parameter space where the function parameterized by the KAN doesn’t change and this results in degenerate eigenvectors of the matrix $M$.

The analysis given is necessarily highly simplified and heuristic. In particular, we only analyze a single layer of the KAN network and consider the continuous least squares loss. Nonetheless, we argue that it gives an explanation for why we would expect KANs to have a significantly different spectral bias than MLPs, and in particular why we expect that they learn all frequencies roughly similarly. In the remainder of this section, we experimentally test this hypothesis and compare the spectral bias of KANs with MLPs on a variety of simple problems. We implement these numerical experiments using the pykan package version $0.2.5$.

###### Theorem 4.1 .

###### Remark 4.2 .

### 4.2 1D waves of different frequencies

In the first example, we take the same setting as in [] and study the regression of a linear combination of waves of different frequencies.
Consider the function prescribed as

$$ $f(x)=\sum A_{i}\sin\left(2\pi k_{i}z+\varphi_{i}\right),\quad k=(5,10,\cdots,4 5,50).$ $$

The phases $\varphi_{i}$ are uniformly sampled from $[0,2\pi]$ and we consider two cases of amplitudes: one with equal amplitude $A_{i}=1$ and another with increasing amplitude $A_{i}=0.1i$. We use a neural network, either ReLU MLP or KAN, to regress $f$ sampled at $200$ uniformly spaced points in $[0,1]$, with full batch ADAM iteration as the optimizer with a learning rate of $0.0003$. For MLPs, we train with $80000$ iterations as in []; for KANs, we only train with 8000 iterations. Normalized magnitudes of discrete Fourier
transform at frequencies $k_{i}$ are computed as $|\tilde{f}_{k_{i}}/A_{i}|$ and averaged over 10 runs of different phases.

We plot the evolution of $|\tilde{f}_{k_{i}}/A_{i}|$ during training across all frequencies; see Figures 1, 2 for comparisons of MLPs and KANs with different sizes for equal and increasing amplitudes respectively. KANs suffer significantly less than MLPs from spectral biases. Once the size of KANs, especially the grid size and depth is large enough, KANs almost learn all frequencies at the same time, while even very deep and wide MLPs still have difficulties learning higher frequencies, even with $10$x epochs!

Figure: Figure 1: 1D wave dataset, where the target function has equal amplitudes of different frequency modes. Under various hyperparameters, MLPs manifest strong spectral biases (top), while KANs do not (bottom). Note that the y axis (training steps) of MLP is 10 times that of KAN.
Refer to caption: extracted/5894833/1D_frequency/width256depth10grid5modelMLP.png

Figure: Figure 2: 1D wave dataset, where the target function has increasing amplitudes of different frequency modes. Under various hyperparameters, MLPs manifest severe spectral biases (top), while KANs do not (bottom). Note that the y axis (training steps) of MLP is 10 times that of KAN.
Refer to caption: extracted/5894833/1D_frequency/width256depth10grid5modelMLP_inc.png

### 4.3 Gaussian random field

In this example, we consider fitting functions sampled from a Gaussian random field.
The target function $f$ is sampled from a $d$-dimensional Gaussian random field with mean zero and covariance $\exp(-|x-y|^{2}/(2\sigma^{2}))$. Here small $\sigma$ corresponds to rough functions and large $\sigma$ corresponds to smooth functions.

To approximate the Gaussian random field, we sample $f$ using the KL expansion []. We sample $N=5000$ points uniformly from $[-1,1]^{d}$ and calculate the (empirical) covariance matrix $K$. Then we truncate its first $m<N$ eigenpairs $\lambda_{i},\phi_{i}$, with the cutoff threshold $\lambda_{m+1}<0.1\lambda_{1}\leq\lambda_{m}$ and sample $f$ approximately via

$$ $f=\sum_{i\leq m}\lambda_{i}\xi_{i}\phi_{i},$ $$

where $\xi_{i}$ are i.i.d standard Gaussians $N(0,1)$. For $f$ with different scales $\sigma$ and dimensions $d$, we split the points into $80\%$ training and $20\%$ testing points. We use MLPs and KANs with different sizes to regress on the training set, with the mean squared loss as the loss function. For MLPs, we use $500$ iterations of LBFGS iteration, and for KANs, we use the grid extension technique, with grid sizes $(10,20,30,40,50)$, each trained with $100$ iterations of LBFGS.

We plot the loss curves here and compare the losses of different scales $\sigma$ and dimensions $d$, using an MLP of $6$ layers and $256$ neurons in each hidden layer, and KANs with $10$ neurons in each hidden layer and $2,3,4$ layers; see Figure 3 for the regression loss on the training set with dimensions $2,3,4$ and scales $2^{i}$, $i=0,-1,-2,-3$. We see that for larger scale and smoother functions, MLP performs better, while for smaller scale and rough functions, KANs perform better without suffering much from spectral biases, and grid extension is especially helpful. We remark that one can choose smaller grid sizes of KANs for smoother functions and obtain more accurate regressions.

Precisely since KANs are not susceptible to spectral biases, they are likely to overfit to noises. As a consequence,
we notice that KANs are more subject to overfitting on the training data regression when the task is very complicated; see the second line of Figure 4. On the other hand, we can increase the number of training points to alleviate the overfitting; see the last line of Figure 4 where we increased the number of training and test samples by $10$x. We remark that the current implementation of grid extension is prone to oscillation after refining grids during the undersampled regime, as observed in [], and we will improve it in future works.

Figure: Figure 3: The Gaussian random field dataset. Training losses of MLP and KANs, with different scales and dimensions.
Refer to caption: extracted/5894833/GRF/trainDim2Scale0.125.png

Figure: Figure 4: The Gaussian random field dataset. Test losses of MLP and KANs, with different scales and dimensions. Increasing the number of samples by $10$x helps overfitting.
Refer to caption: extracted/5894833/GRF/testDim2Scale0.125.png

### 4.4 PDE example

In this example, we solve the 1D Poisson equation with a high-frequency solution, similar to []. To be precise, consider the equation with zero Dirichlet boundary condition

$$ $-u_{xx}=f\,\,\,\,\text{in}[-1,1]\,,\quad u(-1)=u(1)=0\,.$ (13) $$

Here for a frequency $k\in\mathbb{N}$, the right-hand-side and the associated true solution are

$$ $f=\pi^{2}\sin(\pi x)+\pi^{2}{k}\sin(k\pi x),\quad u=\sin(\pi x)+\frac{1}{k} \sin(k\pi x).$ $$

The different frequencies are normalized in a way that for $k>1$, the ground truth has the same energy ($H^{1}$) norm. We use the variational form of the elliptic equation and the associated Deep Ritz Method []. Parametrizing $u$ by a neural network, we minimize the loss

$$ $\lambda\int_{-1}^{1}\left(\frac{1}{2}u_{x}^{2}-fu\right)dx+u^{2}(-1)+u^{2}(1).$ $$

For frequencies $k=2,4,8,16,32$, we use $2000$ uniformly spaced sample points and the neural network using an MLP of 6 layers and 256 neurons in each hidden layer and a KAN of 2 layers with 10 neurons in the hidden layer. We choose the hyperparameter $\lambda=0.01$ balancing the energy and boundary loss and perform LBFGS iterations. For MLPs, we use 200 iterations, and for KANs, we use grid sizes $(20,40)$, each trained with 100 iterations. We plot the relative $L^{2}$ and $H^{1}$ losses compared to the ground truth in Figure 5. We can see that KANs perform consistently better, and the residue barely deteriorates when the frequency increases, whereas it becomes extremely hard for MLPs to optimize when $k=16,32.$

Figure: Figure 5: Solving PDEs. $L^{2}$ and $H^{1}$ losses of MLP and KAN with different frequencies of the solution.
Refer to caption: extracted/5894833/PDE1d/scale_2.png

## 5 Concluding Remarks

In this work, we have compared the approximation and representation properties of KANs and MLPs. Based upon our theoretical and empirical analysis, we conclude that while KANs and MLPs are very similar from the perspective of approximation theory, i.e. the both parameterize similar classes of functions, they differ significantly in terms of their training dynamics. Specifically, KANs do not exhibit the same spectral bias toward low frequencies that MLPs do. Future work includes developing theory which can describe the training of deeper KANs, performing a more extensive experimental investigation of the spectral bias of KANs on more complicated problems, and investigating whether the reduced spectral bias of KANs improves their performance on scientific computing applications such as solving PDEs.

## Appendix A Proofs

###### Proof of Theorem 3.2 .

We will show that each layer of an MLP with the activation function $\sigma_{k}$ can be represented by a KAN with two hidden layers, width $W$ and grid size $G=2$ with degree $k$ B-splines. By composing such layers, we obtain the desired result.

For a single layer of MLP, we consider the linear part and the non-linear activation separately. We first observe that on any compact subset of $\mathbb{R}^{W}$ the linear function

$$ $x_{i}=\sum_{j=0}^{W}a_{ij}x^{in}_{j}+b_{i}$ (14) $$

can be represented with a single KAN layer of width $W$ by setting $\phi_{ij}$ to the linear function

$$ $\phi_{i,j}(x)=a_{ij}x+\frac{b_{i}}{n}.$ (15) $$

We claim that this linear function can be exactly represented on any interval $[-R,R]$ in the form (4). To do this, we first set $w_{b}=0$ and choose the grid points for the B-splines to be

$$ $\{-(2k-1)R,-(2k-3)R,...,-R,R,...,(2k-3)R,(2k-1)R\}.$ $$

Note that based upon the KAN architecture, this corresponds to the extension of the uniform grid $t_{0}=-R,t_{1}=R$ which has grid size $G=1$. It is also easy to verify that there are $(k+1)$ B-splines supported on this grid, whose restriction to $[-R,R]$ span the space of polynomials of degree $k$. Thus, in particular, any linear function on $[-R,R]$ can be represented as a linear combination of these B-splines.

Next, we consider the non-linear activation, which is given by the coordinatewise application of $\sigma_{k}$, i.e.

$$ $x_{i}^{out}=\sigma(x_{i}).$ (16) $$

This can be represented by a single hidden layer KAN by setting

$$ $\phi_{i,j}(x)=\begin{cases}\sigma_{k}(x)&i=j\\ 0&i\neq j.\end{cases}$ (17) $$

We claim that the functions $\sigma_{k}$ can be represented in the form (4) on any finite interval $[-R,R]$. To to this, we again set $w_{b}=0$ and choose the grid points for the B-splines to be

$$ $\{-kR,-(k-1)R,...,-R,0,R,...,(k-2)R,(k-1)R\}.$ $$

This grid is the grid extension of the uniform grid $t_{0}=-R,t_{1}=0,t_{2}=R$ which has grid size $G=2$. It is easy also to verify that there are $(k+2)$ B-splines supported on this grid and that any piecewise polynomial on $[-R,R]$ with a single breakpoint at 0 0 which is $C^{k-1}$ is a linear combination of these B-splines. Hence the function $\sigma_{k}$ can be represented on $[-R,R]$ in the form (4) using this grid.

The proof is now completed by composing these layers and choosing $R$ sufficiently large so that for any input $x\in\Omega$ (which is bounded) the inputs and outputs of every neuron in the original MLP lie in the interval $[-R,R]$.

∎

###### Proof of Theorem 3.3 .

The proof follows from the fact that each KAN activation function $\phi_{l,i,j}$ can be represented by a ReLU^k neural network with one hidden layer of width $G+2k+1$. This is due to the fact that each $\phi_{l,i,j}$ is a linear combination of $B$-splines (since we are assuming that $w_{b}=0$ in (4)). This implies that it must be a compactly support piecewise polynomial spline of degree $k$ whose $(k-1)$-st derivative changes only on the extended grid which has $G+2k+1$ points. We can now easily construct a shallow ReLU^k network with one hidden node per grid point which matches the function $\phi_{l,i,j}$.
∎

###### Proof of Theorem 4.1 .

We first observe from (11) that the matrix $M$ is block diagonal with $d^{\prime}$ identical blocks. Denoting these $(G+k-1)d\times(G+k-1)d$ blocks by $B$, it thus suffices to prove that

$$ $\frac{\lambda_{(G+k-1)d}(B)}{\lambda_{d}(B)}\leq C.$ (18) $$

To do this, we analyze the blocks $B$ and note that they take the form

$$ $B=\begin{pmatrix}C&D&\cdots&D\\ D&C&\cdots&D\\ \vdots&\vdots&\ddots&\vdots\\ D&D&\cdots&C\end{pmatrix}.$ (19) $$

Here the diagonal sub-blocks $C\in\mathbb{R}^{(G+k-1)\times(G+k-1)}$ are the Gram matrix of the one-dimensional B-spline basis, i.e.

$$ $C_{ij}=\int_{0}^{1}B_{i}(x)B_{j}(x)dx,$ (20) $$

and the off-diagonal sub-blocks $D\in\mathbb{R}^{(G+k-1)\times(G+k-1)}$ are rank one matrices

$$ $D=vv^{T},$ (21) $$

where the vector $v\in\mathbb{R}^{G+k-1}$ is given by

$$ $v_{i}=\int_{0}^{1}B_{i}(x)dx.$ (22) $$

It is well-known that the Gram matrix $C$ is well-conditioned uniformly in $G$ for a fixed $k$, i.e. $\lambda_{G+k-1}(C)/\lambda_{1}(C)\leq K$ for a fixed constant $K$ depending only upon $k$. See for instance [], Theorem 4.2 in Chapter 5, where it is shown that the $L_{2}$-norm of a spline and the properly scaled $\ell_{2}$-norm of its B-spline coefficients are equivalent up to a constant depending only on $k$. This is equivalent to the well-conditioning of the Gram matrix $C$.

In addition, we can easily verify using Jensen’s inequality (or Cauchy-Schwartz) that $D\preceq C$. Indeed, letting $w\in\mathbb{R}^{G+k-1}$ we see that

$$ $w^{T}Dw=\left(\int_{0}^{1}f(x)dx\right)^{2}\leq\int_{0}^{1}f(x)^{2}dx=w^{T}Cw,$ (23) $$

where the function $f(x)=\sum_{i=1}^{G+k-1}w_{i}B_{i}(x)$.

Let $\mathbb{1}\in\mathbb{R}^{d}$ be the vector of ones and note that

$$ $(v\otimes\mathbb{1})(v\otimes\mathbb{1})^{T}=\begin{pmatrix}D&D&\cdots&D\\ D&D&\cdots&D\\ \vdots&\vdots&\ddots&\vdots\\ D&D&\cdots&D\end{pmatrix}$ (24) $$

so that $B-(v\otimes\mathbb{1})(v\otimes\mathbb{1})^{T}$ is a block diagonal matrix with diagonal blocks $C-D$. We proceed to upper bound the largest eigenvalue of $B$ by

$$ $\begin{split}\lambda_{(G+k-1)d}(B)&=\max_{\|w\|=1}w^{T}Bw\\ &=\max_{\|w\|=1}w^{T}\begin{pmatrix}C-D&0&\cdots&0\\ 0&C-D&\cdots&0\\ \vdots&\vdots&\ddots&\vdots\\ 0&0&\cdots&C-D\end{pmatrix}w+(w^{T}(v\otimes\mathbb{1}))^{2}.\end{split}$ (25) $$

Writing $w=(w_{1},...,w_{d})$ with $w_{i}\in\mathbb{R}^{G+k-1}$ and $\sum_{i=1}^{G+k-1}\|w_{i}\|^{2}=1$ and using that $D=vv^{T}$, we get the bound

$$ $\begin{split}\lambda_{(G+k-1)d}(B)&\leq\max_{\|w_{1}\|^{2}+\cdots+\|w_{d}\|^{2 }=1}\sum_{i=1}^{d}w_{i}^{T}Cw_{i}+\left(\sum_{i=1}^{d}v^{T}w_{i}\right)^{2}- \sum_{i=1}^{d}(v^{T}w_{i})^{2}\\ &\leq\max_{\|w_{1}\|^{2}+\cdots+\|w_{d}\|^{2}=1}\sum_{i=1}^{d}w_{i}^{T}Cw_{i}+ (d-1)\sum_{i=1}^{d}(v^{T}w_{i})^{2}\end{split}$ (26) $$

Since $D\preceq C$ we have $(v^{T}w_{i})^{2}\leq w_{i}^{T}Cw_{i}$ which gives the bound

$$ $\lambda_{(G+k-1)d}(B)\leq d\max_{\|w_{1}\|^{2}+\cdots+\|w_{d}\|^{2}=1}\sum_{i= 1}^{d}w_{i}^{T}Cw_{i}=d\lambda_{G+k-1}(C).$ (27) $$

Next, we lower bound the $d$-th eigenvalue of $B$. For this, we use the Courant-Fisher minimax theorem to see that

$$ $\lambda_{d}(B)=\max_{W_{d}}\min_{w\in W_{d},~{}\|w\|=1}w^{T}Bw,$ (28) $$

where the maximum is taken over all subspaces of $W_{d}$ of codimension $<d$. We consider the specific subspace

$$ $W_{d}=\{(w_{1},...,w_{d});~{}v^{T}w_{i}=0~{}\text{for all $i=1,...,d$}\}\oplus \text{span}(v\otimes\mathbb{1})$ (29) $$

and observe that for any $(w_{1},...,w_{d})$ with $v^{T}w_{i}=0$ we have

$$ $w^{T}Bw=\sum_{i=1}^{d}w_{i}^{T}Cw_{i}\geq\lambda_{1}(C)\sum_{i=1}^{d}\|w_{i}\| ^{2}=\lambda_{1}(C)\|w\|^{2},$ (30) $$

while for the $w=v\otimes\mathbb{1}$ (which is orthogonal) we have

$$ $w^{T}Bw=\sum_{i=1}^{d}v^{T}Cv+(d-1)\sum_{i=1}^{d}\|v\|^{2}\geq\lambda_{1}(C)d \|v\|^{2}=\lambda_{1}(C)\|w\|^{2}.$ (31) $$

Thus, $\lambda_{d}(B)\geq\lambda_{1}(W)$. Combining these bounds and using the well-conditioning of the Gram matrix $C$, we get

$$ $\frac{\lambda_{(G+k-1)d}(B)}{\lambda_{d}(B)}\leq d\frac{\lambda_{G+k-1}(C)}{ \lambda_{1}(C)}\leq K$ (32) $$

for a constant $K$ which only depends upon $k$. This completes the proof.

∎

###### Proof of Theorem 3.2 .

###### Proof of Theorem 3.3 .

###### Proof of Theorem 4.1 .

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: 1.Theoretical Expansion: How does the depth of Kolmogorov-Arnold Networks (KANs) influence their spectral bias and approximation capabilities? Could a deeper theoretical analysis be provided to explore this relationship?

2.Enhanced Experimental Design:Could the authors include more rigorous statistical analysis and comparative experiments with a broader set of baseline models, particularly state-of-the-art neural architectures used in scientific computing?

3.Real-World Applications: How could the practicality of KANs be demonstrated in more diverse and realistic scenarios? Are there plans to showcase examples that highlight their advantages and limitations?

4. Detailed Discussion: Could the authors provide a more detailed discussion on the computational cost, scalability, and limitations of KANs? What are the key factors that may affect their broader applicability?

5. The author should
soundness: 3
strengths: 1.Theoretical Comparison of Expressiveness: The authors establish that KANs are at least as expressive as MLPs. Specifically, they show that any MLP can be reparameterized as a KAN with degree k splines which is only slightly larger. Conversely, they demonstrate that KANs can also be represented by MLPs, but the number of parameters increases significantly with the grid size of the KAN. This implies that KANs with large grid sizes may be more efficient in approximating certain classes of functions.
2. The exploration of KANs’ spectral bias and their comparison with MLPs provides a novel perspective. The use of B-splines and the grid extension method offer unique contributions. However, much of the theoretical groundwork relies heavily on established techniques.
summary: The paper addresses the theoretical and empirical properties of Kolmogorov-Arnold Networks (KANs), particularly focusing on their approximation capabilities and spectral bias. Given the growing interest in alternative neural network architectures for function approximation, the study of KANs is relevant to the broader machine learning and scientific computing community. However, while the comparison with MLPs is insightful, the direct impact and importance of KANs over existing methods are not convincingly demonstrated.
weaknesses: 1. The parameter count in KANs and MLPs are not directly comparable, and the authors have blurred the distinction between these concepts. Additionally, the depth of KANs differs from that of MLPs, and the authors should not directly compare them, as this leads to a conceptual confusion. 
We acknowledge that the conceptual confusion arises from the discussion in Section 3, particularly in Theorem 3.1- 3.4, where we compare the parameter count and depth between KANs and MLPs. This issue is also present in the experimental results in Sec4.1 and 4.2, where parameter comparisons are made across different models. 

2. Theorem 4.1 lacks clarity regarding its implications for KAN's performance on high-frequency data. The authors should explicitly state whether the theorem suggests that KANs outperform other models in handling high-frequency data. Additionally, it would be beneficial to provide a brief discussion or corollary connecting the theorem to spectral bias or high-frequency performance to clarify its practical significance.

3.The theoretical analysis provided in the paper is limited to shallow KANs, which restricts the generalizability of the conclusions regarding spectral bias. To enhance the practical relevance, the authors should discuss the limitations of their current analysis and suggest potential methods for extending the theoretical framework to deeper KANs. This would provide a more comprehensive understanding of the spectral bias in real-world applications.

4.While the experiments cover diverse applications, they lack rigorous statistical validation to substantiate the claimed advantages of KANs. The authors should include specific statistical tests or validation methods to strengthen their results. Furthermore, more detailed comparisons with state-of-the-art baseline methods would help provide a stronger foundation for the claimed benefits of KANs, especially in practical applications like PDE solving and Gaussian random field fitting.

5.Incomplete Problem Contextualization: The paper does not clearly establish the practical significance of overcoming spectral bias in real-world tasks, nor does it compare KANs against state-of-the-art methods beyond MLPs.

6.Limited Discussion on Limitations: The potential downsides of KANs, such as computational cost, overfitting, and practical implementation challenges, are not thoroughly addressed.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: I have following questions:

(1).  All the theorems 3.1, 3.2 and 3.3 are extremely mathematical, and their assumptions are not agreeable with practical cases. How can the authors explain the main differences?

(2). For the examples of PDE, the equation is very simplie, and the functions are of sufficient smoothness. If the functions concerned becomes chirp function and so on, how about the performance?

(3) In Fig. 4, the performance of approximation does not always tend to better. Can you explain the causes?
soundness: 3
strengths: This paper characterizes the representation capacility of approximation of KAN and MLP, with a considerably rigorous mathematical analysis.
summary: This paper focuses on two aspects of KAN and MLP. The first issue is the approximation and representation capacility to functions of both networks, and the second issue is the spectral bias of approximation. Both issues are very important and attractive.
weaknesses: On the expressiveness, it is merely a theoretical interest, and the proof of mathematical analysis is over simplified by assumpting extremely shallow KAN network, which violates the practical deep network situation, and can not be regarded as equalivent to general case or nonlinear effect.  

The experiments are not sufficient, and the paper is mainly of theoretical approach.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: - Is the comparison between KANs with kth order B-splines with MLP modified when considering relu-k instead of simple relu?
- In Figure 3 and 4 the learning of KANs display instabilities in contrary to MLP which are attributed to grid extension during the undersampled regime, I don't really understand what is meant by this, is it that the grid size of KAN is progressively extended during learning? 

Finally as a minor point:
I found that the way the experimental results are presented could be more concise and more informative. Figures 1 and 2 are almost identical, while all the plots given in figure 3 and 4 could be possibly synthetized to give the general behaviour w.r.t. to k, grid size, dimension and scale, and the legend should be magnified, I can't read anything on a printed version.
soundness: 3
strengths: the questions addressed in the paper are interesting and some of the statements are quite strong, in particular th. 4.1 on the characterization of the spectral bias. Overall, my non-specialist viewpoint  on this paper is that it looks like a rather clear and solid paper, with interesting statements.
summary: This paper is both a theoretical and experimental study of KANs, a recently introduced architecture with parametrizable activation functions,
aiming at characterizing their expressivity and spectral biases in comparison to MLPs. Since KANs offer a lot of new potentialities, it is
a legitimate and relevant research subject to try to characterize these  in  relation to properties of the data, therefore expressivity and spectral biases constitute obviously a good angle of analysis. 

So the first theoretical result concerns expressivness: it tells that an arbitrary MLP (with activation function in C^k typically) can be exactly mapped onto a KAN (with B-splines of order k) with a depth multiplied by 2 and requiering a grid size of 2, so overall the number of parameters is multiplied by 4 (if I interpret well the result, this could be stated more explicitly actually) in this mapping. In reverse they find a mapping of a KAN to an MLP with an increase of parameters of G, the grid size of the original KAN. These constructions lead the authors to conclude qualitatively that KAN are more expressive than MLP. As acknowledged by the authors, this is not a sharp result as it is not excluded that a better construction of the mappings could be found leading to different conclusions, but these constructions are interesting on their own. A more concrete statement on expressivity is given by corollary 3.4, giving the speed of approximating of any function on a Sobolev space w.r.t the number of parameters of a KAN. Here, as a non specialist of this kind of result I become slightly confused. The statement  is seemingly based on generic results on MLP, and seems not to be specific to KAN but more to multi-layer models, as it exploit the  aforementioned mapping of the KAN to MLP, so that I wonder why MLP with relu-k would not lead to the same statement?
There is sentence discussing this at the end of the paragraph which is cryptic to me as it refers to some apparently non-standard MLP with relu-k.  As a result the overall take-home message of this part is unclear to me. Is there any advantage to be attributed to KAN wrt to comparable MLP under this measure of expressiveness? Maybe the statement 3.4 is a profound and central statement of the paper, but the discussion should make it clear, by first giving how standard MLP with relu-k compare and then also maybe what are the practical consequence of such statement if any?

The second theoretical point concerns spectral bias, and theorem 4.1 to me is a nice and non-trivial statement showing that a single layer of B-splines is well conditioned, especially when the grid size becomes large, by looking at the spectrum of the Hessian of the loss. This fully characterizes the conditioning of the problem with explicit dependency w.r.t. embedding dimension and grid size. The comparison is done with MLP with Relu using a result given in Hong et al. (2022) obtained with similar techniques (Courant Fisher theorem) showing  a much larger spectral bias for MLP. I wonder however if a  comparison with the spectral bias of MLP with relu-k (unfortunately not to given in Hong et al, but I presume that similar techniques would lead to the result) would not be more relevant as they are in the same class of regularity as KAN with B-splines of order k?

The experimental part focuses on the spectral bias question on diverse examples. A comparison is shown between KAN with kth order B-splines and MLP with relu, showing a clear advantage to KAN regarding spectral bias, but again I wonder why the comparison is not done with the spectral bias of MLP with relu-k, if this  would not be be more meaningful?
weaknesses: I found however a potential problem  of coherence concerning the comparison of KANs with MLPs: with relu-k at the beginning which is indeed legitimate,  and then basic relu starting from corollary 3.4.

---

confidence: 2
contribution: 2
initial_rating: 6
presentation: 2
questions: 1. could you expand on the consequences of Corollary 3.4
2. would be interesting to understand the consequence of thr.3.2 and th.3.3 for more "mainstream" architecture. For example, relu^k k=1 is probably the most common, while when gradients are needed SiLU SiLU-based activation functions are required, which is typical for scientific applications
3. would be nice to expand the Hessian analysis and connect with the NTK one
4. would be possible to perform the spectral analysis for section 4.4, instead of the error trend, if not, what is the test error?
soundness: 3
strengths: As described above:

1. find the connection between two specific classes of MLP and KAN. 
2. interpretation of the spectral bias using the Hessian matrix
3. some experiments to visualize the spectral bias
summary: The paper addresses the expressivity Kolmogorov-Arnold Networks (KAN) and the spectral bias.

A rigorous analysis of the representation power of the KAN vs MLP is welcome. 

I think the most important contribution of the paper is to show the relationship between MLP with ReLU^k activation function and KAN without bias. Th.3.2 and Th.3.3 bounds are not tight. A conclusion of the authors is that MLP scales with G^2 while KAN with G, the grid size. The additional contribution Cor3.4 is not well explained in relationship to MLP, but the authors highlight the potential advantage of KANs. 
 
The Analysis of the spectral bias seems a repetition of the analysis provided in [1], even if the authors analyze the Hessian matrix instead of the NTK matrix and show the elements of the Hessian. The authors are able to justify the reason why low frequency are not prioritized as in MLP, even if the connection with Th.4.1 is not clear.  

The analysis of PDE, seems to be the same are reported in [1], showing better performance of KAN-based models in the training loss. Not sure why the test loss is not reported. 


[1] 1. Wang, Y. et al. Kolmogorov Arnold Informed neural network: A physics-informed deep learning framework for solving forward and inverse problems based on Kolmogorov Arnold Networks. Preprint at https://doi.org/10.48550/arXiv.2406.11045 (2024).
weaknesses: As reported in the summary:
1. the theoretical analysis is limited to specific class of MLP and KAN
2. experiments seem very similar to [1], so it is not clear, apart from Fig.1 and Fig.2, why they are included.

Meta-review: This paper studies KANs and presents a theoretical analysis of their approximation capabilities and spectral bias. The results suggest that, compared to MLPs, KANs provide a more compact representation when learning certain classes of functions. The analysis also shows that KANs are more sensitive to high-frequency components of the data, potentially making them more prone to overfitting. Considering the growing interest in exploring alternatives to conventional neural architectures, this paper provides a timely study and valuable insights. Reviewers appreciate the paper's contributions, highlighting the rigorous theoretical analysis and the interesting nature of the findings. They also value the visualizations and experiments that further support the theoretical claims.

However, reviewers also raised multiple concerns. Reviewer iJru identified several issues, including: an unfair comparison between KANs and MLPs in terms of parameter count and depth; unclear implications of the theory for high-frequency data; the limited scope of the theoretical analysis (also mentioned by T7tm), which focuses only on shallow KANs; experiments lacking comprehensive comparisons with state-of-the-art methods; and inadequate discussion of the limitations of KANs, such as computational cost and potential overfitting. The authors responded to these concerns, largely addressing the issues raised by the reviewer. The reviewer subsequently increased their initial score.

Reviewer EXrE expressed concerns that the "shallow model assumption" used to develop the mathematical analysis of KANs do not reflect practical deep network situations and cannot be considered equivalent to the general case or nonlinear effects. While acknowledging these limitations, the authors provided further context, which the reviewer found satisfactory. Reviewer Wf2h pointed out an issue with the comparison of KANs with MLPs, for which the authors provided an adequate explanation. Reviewer T7tm expressed concerns about the limited nature of the theoretical analysis (limited to shallow networks) and the similarity of the experiments to those in [1]. The authors acknowledged the similarity, admitting it was an oversight, but explained that [1] uses examples for function fitting, not PDE solving.

In summary, the majority of reviewers rated the paper in the "accept" zone (with one recommending a clear accept), and only one reviewer rated it as slightly below the acceptance threshold. Overall, the paper is slightly above the borderline, and given its strengths, as pointed out by the reviewers as well, and the importance of the problem studied, I believe it makes interesting and valuable contributions, and I recommend accept.

Decision: Accept (Poster)