On the expressiveness and spectral bias of KANs
Authors: Yixuan Wang
Jonathan W. Siegel
Ziming Liu
Thomas Y. Hou
Venue: ICLR 2025 Poster (2025)
Keywords: Kolmogorov-Arnold Network
Spectral Bias
Approximation Theory
Primary area: learning theory

Abstract: Kolmogorov-Arnold Networks (KAN) \cite{liu2024kan} were very recently proposed as a potential alternative to the prevalent architectural backbone of many deep learning models, the multi-layer perceptron (MLP). KANs have seen success in various tasks of AI for science, with their empirical efficiency and accuracy demonstrated in function regression, PDE solving, and many more scientific problems.
 
In this article, we revisit the comparison of KANs and MLPs, with emphasis on a theoretical perspective. On the one hand, we compare the representation and approximation capabilities of KANs and MLPs. We establish that MLPs can be represented using KANs of a comparable size. This shows that the approximation and representation capabilities of KANs are at least as good as MLPs. Conversely, we show that KANs can be represented using MLPs, but that in this representation the number of parameters increases by a factor of the KAN grid size. This suggests that KANs with a large grid size may be more efficient than MLPs at approximating certain functions. On the other hand, from the perspective of learning and optimization, we study the spectral bias of KANs compared with MLPs. We demonstrate that KANs are less biased toward low frequencies than MLPs. We highlight that the multi-level learning feature specific to KANs, i.e. grid extension of splines, improves the learning process for high-frequency components.  Detailed comparisons with different choices of depth, width, and grid sizes of KANs are made, shedding some light on how to choose the hyperparameters in practice.

Published as a conference paper at ICLR 2025
ON THE EXPRESSIVENESS AND SPECTRAL BIAS OF KANS
Yixuan Wang Jonathan W. Siegel
Ziming Li Thomas Y. Hou'
1 California Institute of Technology
2 Texas A&M University
3 Massachusetts Institute of Technology
4 The NSF Institute for Artificial Intelligence and Fundamental Interactions
ABSTRACT
Kolmogorov-Aroold Networks (KAN) Liu et al. (2024e) were very recently pro- posed as a potential alternative to the prevalent architectural backbone of many deep learning models, the multi-layer perceptron (MLP). KANs have seen success in various tasks of AI for science, with their empirical efficiency and accuracy demonstrated in function regression, PDE solving, and many more scientific prob- lems.
In this article, we revisit the comparison of KANs and MLPs, with emphasis on theoretical perspective. On the one hand, compare the representation and approximation capabilities of KANs and MLPs. We establish that MLPs can be represented using KANs of comparable size. This shows that the approximation and representation capabilities of KANs are at least as good as MLPs. Conversely, we show that KANs be represented using MLPs, but that in this representation the number of parameters increases by a factor of the KAN grid size. This suggests that KANs with a large grid size may be more efficient than MLPs at approx- imating certain functions. On the other hand, from the perspective of learning and optimization, we study the spectral bias of KANs compared with MLPs. We demonstrate that KANs are less biased toward low frequencies than MLPs. We highlight that the multi-level learning feature specific to KANs, i.e. grid extension of splines, improves the learning process for high-frequency components. Detailed comparisons with different choices depth, width, and grid sizes of KANs are made, shedding some light on how to choose the hyperparameters in practice.
INTRODUCTION
Recently, in Liu al. (2024e). a novel architecture called Kolmogorov- Networks (KANs) was proposed as potentially more accurate and interpretable alternative to standard multi-layer perceptron (MIP) Cybenko (1989); Hornik er al. (1989). The KAN architecture leverages the Kolmogorov- Arnold representation theorem (KART) Kolmogorov (956 1957) to parameterize functions. KANs share the same fully connected structures as MLPs, while putting learnable activation functions on edges opposed to fixed activation function nodes for MLPs. B-splines used to parameterize the learned nonlinearity and in practice one can go beyond the two-layer construction indicated by KART. KANs can be conceptualized as hybrid of splines and MLPs, and the combination of compositional nonlinearity with ID splines contributes to both the accuracy and interpretability of KANs.
In this article, we study the approximation theory and spectral bias of KANs and compare them with MLPs. Specifically, we show that any MLP with the ReLU* activation function can be reparameter- ized as KAN with a comparable number of parameters. This establishes that the representation and approximation power of KANs is at least as great as that of MLPs. On the other hand, we also show that any KAN (without SiLU non- -on-liiearity) can be represented using an MLP. However, the number of parameters the MLP representation larger by a factor proportional to the grid size of the KAN.
roywang @ caltech.edu "These authors contributed equally this work.

---

Published as a conference paper at ICLR 2025
While we do not know if our construction is optimal in the deep case, this suggests that KANs with large grid size may be more efficient at parameterizing certain classes of functions than MLPs. We combine these results with existing optimal approximation rates for MLPs obtain approximation rates for KANs on Sobolev spaces.
Next, we study the spectral bias phenomenon in the training of KANs. Standard MLPs with ReLU activations (or even tanh) are known to suffer from the spectral bias Rahaman et al. (219)) 9: Xu
al. (2019a b). in the sense that they will fit low-frequency components first. This in contrast to traditional iterative numerical methods like the Jacobi method which learn high frequencies first
Xu et al. (2019a). Although the spectral bias acts as regularizer which improves performance for machine learning upplications Rahaman et al.] / et al. (202T); Poggio et al. (2018); Zhang ct al. 2020); Fridovich-Kaa Keil et al. (2022), for scientific computing applications it may be necessary to learn high-frequencies as well. To alleviate the spectral bias, high-frequency information has to be encoded using methods like Fourier feature mapping Siizmannn et al. (0220;; Tancik et al.) (2020); Benbarka al. (2022); Novello al. (2024), or one needs to use nonlinear activation functions more similar to traditional methods; see for example the hat activation function Hong et al. (2022) which resembles finite element basis. We theoretically consider single KAN layer and show that it does not suffer from the spectral bias by analyzing gradient descent for minimizing a least squares objective. Although our analysis is necessarily highly simplified, we argue that provides some evidence and intuition showing that KANs will have a reduced spectral bias compared with MIPs.
Finally, we study the spectral bias of KANs experimentally on wide variety of problems, including ID frequency fitting, fitting samples from a higher-dimennsinal Gaussian kernel, and solving the Poisson equation with a high-frequency solutions. Based upon the results of our experiments, KANs consistently suffer less from the spectral bias. This also helps explain why KANs are more subject to noise and overfitting, as observed in Shen et al., (2024), while grid coarsening (the opposite of grid extension) helps increase the spectral bias and reduces overfitting. Notice that the highest frequency that a single layer of KANs can learn is restricted to the number of grid points we use. The compositional structure associated with the depth also plays role in learning the different frequencies. We study the effect of depths, widths, and grid sizes of KANs for the spectral bias, and draw conclusions about best practices when choosing KAN hyperparameters.
Our contribution. Our goal in this paper is to theoretically compare the KAN architecture with the commonly used MLP architecture. Our specific contributions are as follows:
We compare the approximation and representation ability of KANs and MLPs. We show that KANs are at least as expressive as MLPs. We use this to obtain approximation rates for KANs on Sobolev spaces.
We theoretically analyze gradient descent applied to optimize the least squares loss using a single layer KAN. Based upon this analysis, we argue that KANs, unlike MLPs, do not suffer much from a spectral bias.
We provide numerical experiments demonstrating that KANs exhibit less of a spectral bias than MLPs on variety of problems. This validates our theory and also provides partial explanation for the success of KANs on problems in scientific computing.
PRIOR WORK
theory on the approximation ability of KANs in the function class of compositionally smooth functions was proposed as KAT in Liu et al. (024c). A convergence result independent of the dimension was obtained, leveraging the approximation theory of ID splines. Compared with the universal approximation theorem of MLPs, KANs take advantage of the intrinsically low-dimensional compositional representation of underlying functions. This result shares an analogy to the rate in generalization error bounds of finite training samples, for a similar space studied for regression problems; see Horowitz & Mammen (2007); ;; Kohler & Langer (202I). and also specifically for MLPs with ReLU activations Schmmdtt Hiiber (2020). On the other hand, for general Sobolev or Besov spaces, sharp approximation rates have been obtained for ReLU-MLPs (and more generally MLPs with most piecewise polynomial activation functions) Yarotsky (2017); [Brrlett et al. 2 (ieeell (2023). These rates exhibit the curse of dimensionality, which is unavoidable due to the fact that Sobolev and Besov spaces with fixed smoothness are very large in high dimensions.
2

---

Published as a conference paper at ICLR 2025
There are also subsequent works exploring other parametrizations of activation functions in the
KAN formulation, including special polynomials Aghaei (2024a); [Seydi] (2024a); |SS] (2024). rational
functions |Ahhaii| (2024b), radial basis function Li Ta 2024). Fourier series Xu et al. (2024a)
Fourier
and wavelets Booorgasl & Chen 2024); [Sydd] 2024b). Active follow-up research focuses on
applying KANs various domains, such as partial differential equations Wang al. (2024); Shukla
iet al. (2024); Rigas et al. (2024) and operator learning Abueidda et al. (2024): [Shukla et al.
(2024);
Nehma & Tiwari (222). graphs Bresson et al. (2024); De Carlo al. (2024); Kiamari et (2024)
Zhang & Zhang (2024) time series Vaca- et al. (2024); Genet & (2024); Xu et al.
(2024b); Genet& & Inzirillo (2024a), computer vision Cheon (0024)) Azam & Akhtar (2024); Li
et al. [2024a); Cheon (2024a); [Syydi (2024b); [Bodner et al. (2024), and various scientific problems
Liu et al. (2024b c): Yang et al. (2024); Herbozo Contreras et al. (2024); Kundu et al. (2024); Li
[ et al. (2024b); Ahmed & Sifat (2024) : Liu et al. (2024a); Peng et al. (2024); Pratyush et al. (222))
In the KAN 2.0 paper Liu et al. (2024d), multiplication was introduced as built-in modularity of KANs and the connection between KANs and scientific problems was further established in terms of identifying important features, modular structures, and symbolic formulas. In this article, we focus on the original B B-spline formulation, one of the reasons being its alignment with continual learning and adaptive learning; see also Rigas et al. (2024).
The spectral bias of MLPs has been studied both experimentally and theoretically in a variety of works, see for instance Rahaman et al. (2019); Zhang et al. (2021 12 Xu et al. (2019a al. ( Hong et al.
(2022) Poggio al. (2018): Cai & Xu (201); Basri et al. (2020); Fridovich-
al. (2022):
Zhang et al. ): Ronen et al. (2119) and the references therein. This phenomenon is proposed as explaining the regularizing effect of stochastic gradient descent Rahaman et al. (201)); Fridovich- Keil
et al. (2022). 2. In Wang et al. (2024) an empirical analysis of the eigenvalues of the Neural Tangent KernelJacot et al.j (2018) matrix is performed, which hints the better performance of KANs for fitting high frequencies. However, the spectral bias of finite width KANs has not yet been considered, and it this gap that we aim to close in this work.
3 REPRESENTATION AND APPROXIMATION
In this section, we study how to represent ReLU* MLPs using KANs with degree k splines and vice-versa. Using this, we establish approximation rates for KANs from the corresponding results for MLPs. The measure of complexity that we consider in this section is the number of parameters of the KAN or MLP model. We remark that the number of parameters may not always be the most useful measure of complexity, especially considering that we are comparing different architectures, but is generally indicative of the efficiency of the model. We leave the problem of comparing KANs and ML.s using different measures of complexity to future work.
3.1
REVIEW OF THE KAN ARCHITECTURE
We recall the following definitions of the Kolmogorov-Amold Network (KAN) architecture introduced in Liu et (2024e). We refer to Liu al. (2024e) for thorough treatment of all details mentioned here.
KAN architecture tivariate continuous function on a bounded domain can be represented as a finite composition of univariate continuous functions and addition. Specifically for continuous f 00. 1)" - R, there exists continuous 1D functions &g.p. & such that (see for instance Braun & Griebel (2009)
The Kolmogorov-Aroold representation theorem (KART) states that any mul-
2n+1
f(x) = f(a ... Ind
Σ p=1
(1)
q
Inspired by KART corresponding to a two-layer neural network representation. the authors in Liu et al.
(2024c) unified the inner and outer functions via the proposed KANN layers and generalized the idea of composition to arbitrary depths. A KAN can be represented by an integer array [No. ... .z). Here L is the number of layers and no and n,, are input and output dimensions respectively. 71 denotes the number of neurons in the 1-th neuron layer. Usually, choose n1 n W
3

---

Published as a conference paper at ICLR 2025
as the width and refer L as the depth of the KAN. Layer I maps vector X E R to X1+1 R
via nonlinear activations and additions, in the matrix form as
(1,1(()
(12,1(-)
.1))(())
212,(())
X/+1
X1.
11.1 (+)
012,m41 (+)
The overall network can be written composition of L KAN layers as
KAN(x) (L1-1 1 Pp)x.
(3)
( is parametrized by linear combination of k-th order B-splines and SiLu nonlinearity. Namely
Each activation function ø 01,,, in
<(x) 1/r/(1+ Σ cB,(s),
where G the grid size, B, are spline basis and C₁ Wh are trainable coefficients. The ranges of the grid points of the B-splines are updated on the fly, based on the range of the output of the previous layer. For & on bounded interval (a, b), consider the uniform grid {4o = a, (1, 1.. ,Ia b}. uniformally extended to {t_x. ... f-1, fo. ... LG. fG+1,'.. (G++). )- On this grid there are G + k B-spline basis functions, with the jth B-spline (x) being non-zero only on [t_ 1241).
Grid extension A very important structure of KANs proposed in Liu al. (2024c) is the grid extension technique, inheriting the multi-level fine-graining from splines. One can first train KANs with fewer parameters a desired accuracy. before making the spline grids finer with initialization from the coarser grid and continuing the training procedure. Grid extensions for all splines in a KAN can be performed independently and adaptively. We highlight that this technique is specific to the choice of splines parameterizations of activation functions, and is particularly useful in the training process; see for example the discussion of spectral bias in the next section.
We recollect the approximation theory established for composition- ally smooth functions in Liu et al. (2024e).
Approximation theory, KAT
Theorem 3.1. Let x (I1, F2,
.n). Suppose that a function fxx) admits representation
= (4 L- ................................ 0 Do)x,
(5)
as in 3). where each one of the Pi,is is (k + 1) time continuously differentiable. Then there exists a constant C depending on f and its representation, such that we have the following approximation bound in terms of the grid size G: there exist k-th order B B-spline functions &6 such that for any
0 < m S. k, we have the bound for the CTm Cmm-norm
Ilf - ( 1-11 & 22-20.... & x6)xlcc S CG-K--4m.
(6)
3.2 REPARAMETTRIZATION OF KANS AND MIPS
Next, compare the approximation capabilities of KANs and MLPs. In particular, have the following result showing that any R MLP can be exactly represented using a KAN with degree k splines which is only slightly larger. We remark that this includes the where k = 1, i.e. the popular and standard ReLU activation function.
Theorem 3.2. Let Ω C Rd be a bounded domain. Suppose that a function f : Rdd + TR can be represented by MLP with width W 2 1, depth L > 1, and activation function dx max(00 x))a for k 2 1. Then there exists a KAN with width W, depth at most 2L, and grid size G = 2 with k-th order B-spline functions such that
g(z) f(x)
for all x. E Ω.
4

---

Published as a conference paper at ICLR 2025
On the other hand, we may ask whether every KAN can also be expressed using MLP of comparable size. If the functions Ō1,6, in the KAN representation have weight wa * in (3), then it is clear that we can not represent the resulting KAN using MLP with activation oe since the smooth function f(x = /1 (1 + e e-*) is not polynomial. Essentially, the SiLU nonlinearity cannot be captured by a ReLU* MLP. However, if this nonlinearity is not present, then KAN can be represented using ReLU* MLP.
Theorem 3.3. Suppose that a function f [0. + can be represented by a KAN with width W 2 I, depth L 2 I. and grid size G with k-thh order B-spline functions. Assume also that the weight wa in ) for all of the activations $1.5 appearing in the KAN. Then there exists an MLP with width (G + 2k + 1)W2, depth at most 2L, and activation function ok = max(0, x/* which represents f.
We remark that the width in this result scales like O(GW?) so that the number of parameters in the MLP scales like O(G*W*L) while the number of parameters in the KAN scales like 2 This indicates that KANs with a large number of breakpoints at each node may be more efficient in expressing certain functions than MLPs. When the grids of the spline basis are different across each neuron pairs, which is natural after grid update in training, then there are (G + 2k + 1)W different break points of the splines for each KAN layer. we restrict ourselves to using only 1 hidden layer of ReLU* for each KAN layer, as in the construction of the proof in appendix C then by He et al. (2018) follows that the width of ReLU* R sharp. However, we don't t know whether the preceding theorem is sharp in the deep case.
MLP has to be at least (G + 2k + I)W² and the theorem
Using these results, we can draw conclusions about the approximation capabilities of KANs by leveraging existing results about MLPs (sce for instance Barron (1933); Leshno et al. (993);
Klusowski & Barron (2018); Siegel & Xu 2020 2022|; Hon & Yang (2022) y Lu et al. (2021) Shen
al.] [2022); ; Yarotsky (2018); [Siggel (2023); (2023); Yang al. (2023); Yang & Lu (2023). For example,
we have the following result giving optimal approximation rates for very deep KANs on Soboley
spaces (see for instance Adams & Fournier (2003) for the background on Sobolev spaces).
Corollary 3.4. Let Ω C 12d be bounded domain with smooth boundary, s 0 and < p,q S oc be such that 1/q- - 1/p < s/d. This guarantees that the compact Sobolev embedding (*,L((())) CC
L holds.
Let Wo == (old) be a fixed width (depending upon the imput dimension d). Then for any f E (L, and any L 1, there exists KAN g with width Wo, depth L, and grid size G 2 with k-th order B B-spline functions such that
Ilf- alle,199) S CL-22/14
(8)
where C' is a constant independent of L.
This result, which follows immediately from Theorem 3.2 and the approximation rates for ReLU (and more generally piecewise polynomial) neural networks derived in Siegel (2023), shows that very deep KANs attain an exceptionally good approximation rate on Sobolev spaces. In particular, in terms of the number of parameters P they attain an approximation rate of (PP-a/dd) while a classical (even non-linear) method of approximation can only attain a rate of (PP-//d) Deve (1988). This phenomenon, which is often called superconvergence De Vore al. (202I), also occurs for very deep ReLU* networks. However, it comes at the cost of parameters which are not encodable using fixed number of bits and thus is not practically realizable Yarotsky & Zhevnerchuk (2020) Siegel (2023).
Remark 3.5. Compared with the approximation theory KAT in Theorem 3.1 stated for composition- ally smooth functions where the curse of dimensionality does not appear in the convergence rate, Corollary 3.4 is stated for a larger class of Sobolev functions.
4 SPECTRAL BIAS
In this section, we study the spectral bias of KANs and compare it to that of MLPs. The spectral bias, or frequency principle, refers to the observation that neural networks trained with gradient descent tend to be biased toward lower frequencies, i.c. that lower frequencies are learned first. This phenomenon has been well-documented and studied for MLPs, see for instance Rahaman et al. Rattance/Raaamann
(2019): Zhang et al. (2021) Xu et al. (201a b); Hong al. (2022) PPoggio et al. (2018): Cai & Xu
5

---

Published as a conference paper at ICLR 2025
(2022) Zhang et al. (2023), and it is purpose here to develop an analogous theory for KANs. We remark that although the spectral bias is considered a form of regularization which is desirable for machine learning tasks Rahaman et al. (2019); Zhang et al. (2021); ) Poggio et al. 2018); Zhang et al. (2020); FFidovichh Keil et al. (2022). p for scientific computing applications it is typically important to capture all frequencies and so the spectral bias
(2019) ; Basni et al. (2020); Fridooichh- Keil et al.
may negatively affect the performance of neural networks for such applications Wang et al. (2022);
Rathore al. (2024) Cai & Xu (2019) Hong al. (222)).
4.1 SPECTRAL BIAS THEORY FOR SHALLOW KANS
We consider the spectral bias properties of KANs with single layer. This theory is very similar to the theory developed in Hong et al. (2022); Zhang et al. (2023); Ronen et al. (2019) for the spectral bias of single hidden layer MLPs. The key observation is that single layer KAN is linear model. In particular, we see that if L = 1 then the KAN applied to an input x E I is (for simplicity, we consider the case of KAN without the SiLu non-linearity)
G+R-1
KAN(X, 0),
Σ
Σ
Ciji Bxy)),
(9)
where 0 {cigt} are the parameters of the KAN. Here i 1, I,... d' where d' is the dimension of the output, j = 1.., d, and / 1... G k - 1. Note that the only parameter here is the grid size G, since the width is determined by the input and output dimensions.
Based upon this, we can analyze least squares fitting with shallow KANs. In particular, let Ω [-1, 1] be the (symmetric) unit cube in Rd, let f Ω + IRd" be target function we are trying to learn, and consider the (continuous) least squares regression loss
(( = / llf(x)) - KAN(X, ()]
(10)
Due to the representation 9.. this loss function is quadratic in the parameters 0. Let M denote the corresponding Hessian matrix, i.e. so that
[(0) = (1/2)0" MO + D"O.
This Hessian matrix (indexed by i,j, 1) is given by
Ja Bx 0
i= i 1'
Mas.
(11)
The convergence of gradient descent the least squares regression is determined by the eigen decomposition of the Hessian matrix M which estimated in the following theorem. The theorem is generalization of the fact that the Gram matrix of the B-spline basis is well conditioned (see for instance De Vore & Lorentz (1993), Theorem 4.2 in Chapter 5).
Theorem 4.1. Given single hidden layer KAN with grid size G, degree B-splines, input dimension d and output dimension d', let M denote the Hessian matrix defined in (11) corresponding to the
least squares fitting problem (10. Then the eigenvalues 0 S (1M S ... S AA (M) (here
N (G k 1akk-))ee 1)ddd) satisfy
AN(M) & Cd
(12)
(rdd1))1(M)
for a constant C depending only on the spline degree k.
Spectral bias refers to the fact that lower frequencies converge much faster in training, which is related to the ill-condition property of the Hessian matrix in shallow models. Theorem shows that away from d' 1) eigenvectors the matrix .M is well conditioned. This means that gradient descent will converge at the same rate in all directions orthogonal to these d' d'(d - 1) eigenvectors. Note that since the number of eigenvectors we must remove is independent of the grid size G. we expect that when G is relatively large most components of the KAN will converge at roughly the same rate toward the solution. Thus the KAN with a large number grid points will not exhibit the same spectral bias toward low frequencies seen by MI.Ps.

---

Published as a conference paper at ICLR 2025
Remark 4.2. We remark that in constrast the Hessian associated with two layer ReLU MLP with width n has a condition number which scales like n Hong et al. (2022). ). which explains the strong spectral bias exhibited by ReLU MLPs. On the other hand, established in Remark 14 in Zhang et al. 2023), ReLU MLP with width n has a condition number at the order of n 2+22 One can also observe in Sec 4.3 that they perform even worse for high-frequency tasks.
Remark 4.3. We remark that in Wang et al. (2024) a spectral bias analysis for infinite width KANs is performed empirically using an analysis of the associated neural tangent kernel for specific tasks.
Theorem 4.1 complements this work by giving an analysis in the finite width regime as well.
Remark 4.4. We note that the - eigenvectors which must be excluded is not an artifact of the proof. In fact, this is due to the fact that the KAN parametrization is not unique. Indeed, the constant function f(x)
can be parameterized d different ways by using the B-splines each of the different directions. This ambiguity gives rise to directions in parameter space where the function parameterized by the KAN doesn't 't change and this results in degenerate eigenvectors of the matrix M.
The analysis given is necessarily highly simplified and heuristic. In particular, only analyze single layer of the KAN network and consider the continuous least squares loss. Nonetheless, we argue that it gives an explanation for why would expect KANs to have significantly different spectral bias than MLPs, and in particular why we expect that they learn all frequencies roughly similarly. In the remainder of this section, we experimentally test this hypothesis and compare the spectral bias of KANs with MLPs on a variety of simple problems. We implement these numerical experiments using the pykan package version 0.2.5.
4.2 1D WAVES OF DIFFERENT FREQUENCIES
In the first example, we take the same setting as in Rahaman et al. (2019) and study the regression of a linear combination of waves of different frequencies. Consider the function prescribed as
f(x) Σ A, sin (2Th,z + p(), kc (5, 10. ... .4,500.
The phases 9 are uniformly sampled from [0, 2 and we consider two cases of amplitudes: one with equal amplitude A; = 1 and another with increasing amplitude A, 0. We use neural network, either ReLU MIPP or KAN, to regress I' sampled at 200 uniformly spaced points in [0, 1]. with full batch ADAM iteration as the optimizer with learning rate of 0.0003. For MLPs, we train with 80000 iterations as in Rahaman et al. (2019); for KANs, we only train with 8000 iterations. Normalized magnitudes of discrete Fourier transform at frequencies k; are computed as / and averaged over 10 runs of different phases.
We plot the evolution of /A, during training across all frequencies; see Figure and Figure in the appendix B for comparisons of MLPs and KANs with different sizes for equal and increasing amplitudes respectively. KANs suffer significantly less than MLPs from spectral biases. Once the size of KANs, especially the grid size and depth is large enough. KANs almost learn all frequencies at the same time, while even very deep and wide MLPs still have difficulties learning higher frequencies, even with 10x epochs!
4.3 GAUSSIAN RANDOM FIELD
In this example, we consider fitting functions sampled from a Gaussian random field. The target function f is sampled from a d-dimensional Gaussian random field with mean zero and covariance exp(-lx - y/² ((2=)). Here small a corresponds to rough functions and large o corresponds to smooth functions.
To approximate the Gaussian random field, we sample f using the KL expanssin Karhunen (947)). We Karhuiinn
sample N 5000 points uniformly from [-1, 1]d and calculate the (empirical) covariance matrix K. Then we truncate its first m < N cigenpairs Xes di. with the cutoff threshold Am+1 < 0.1A, S Am and sample f approximately via
1= I= XiEroi
7

---

Published as a conference paper at ICLR 2025
03
02
43
Mode 4/0013 Duephe Gribel
iroeaton Wertod Diand CHiOO
irossaion Wordd Oecnel CRCO
-04
Figure ID wave dataset, where the target function has equal amplitudes of different frequency modes, Under various hyperparameters, MLPs manifest strong spectral biases (top), while KANs do (bottom). Note that the y axis (training steps) of MLP 10 times that of KAN.
where Ça are i.i.d standard Gaussians ((0, 1). For í with different scales and dimensions d, we split the points into 80% training and 20% testing points. We use MLPs and KANs with different sizes to regress the training set, with the mean squared loss as the loss function. For MLPs, use 500 iterations of LBFGS iteration, and for KANs, we use the grid extension technique, with grid sizes (10, 20, 30, 40, 50), each trained with 100 iterations of LBFGS. We can see that ReLU MLPs performs even worse than a standard MLP.
We plot the loss curves here and compare the losses of different scales and dimensions d, using an MLP of 256 neurons in each hidden layer and optimized over its depth, and KANs with 10 neurons in each hidden layer and 2, layers;
Figure 2 for the regression loss the training set with dimensions 2, 3, 4 and scales 2', = 0, 1, 2, 3. We see that for larger scale and smoother functions, MLP performs better, while for smaller scale and rough functions, KANs perform better without suffering much from spectral biases, and grid extension is especially helpful. We remark that one can choose bigger grid sizes of KANs for smoother functions and obtain more accurate regressions.
Precisely since KANs are not susceptible spectral biases, they are likely to overfit noises. As consequence, we notice that KANs are more subject overfitting on the training data regression when the task is very complicated; see the second line of Figure 3 On the other hand, we can increase the number of training points to alleviate the overfitting; see the last line of Figure 3 where increased the number of training and test samples by 10x. We remark that the current implementation of grid extension is prone to oscillation after refining grids during the undersampled regime, observed in Rigas et al. 2024). and we will improve it in future works.
4.4 PDE EXAMPLE
In this example, solve the ID Poisson equation with a high-frequenee solution, similar to Xu et al. (0019)). To be precise, consider the equation with zero Dirichlet boundary condition
-2z = f in[-1,1], u(-1) = u(1) 0.
(13)
Here for frequency l N, the hight-hand-sice and the associated true solution are
f = 2 sin(xx)) + π sinT)z)) u == sin + k
sin(kar).
8

---

Published as a conference paper at ICLR 2025
19:00
Dext S1
Drd Sciadh
Figure 2: The Gaussian random field dataset. Training losses of MLP and KANs, with different scales and dimensions.
Figure 3: The Gaussian random field dataset. Test losses of MLP and KANs, with different scales and dimensions. Increasing the number of samples by 10x helps overfitting.

---

Published as a conference paper at ICLR 2025
The different frequencies normalized in way that for A the ground truth has the same energy (H norm. We use the variational form of the elliptic equation and the associated Deep Ritz Method Yu et al. 2018). Parametrizing by neural network, minimize the loss
2 /f"
1 fu dr + (-(-1) + (1). 2
For frequencies k 2, 2, 8, 16, 32, we use 2000 uniformly spaced sample points and the neural network using an MLP of layers and 256 neurons in each hidden layer and KAN of 2 layers with 10 neurons in the hidden layer. We choose the hyperparameter À = 0.01 balancing the energy 200 iterations, and for KANs,
and boundary loss and perform LBFGS iterations. For MLPs, we use grid sizes (20, 40), each trained with 100 iterations. We plot the relative 1,2 and H losses that KANs perform consistently better, and
compared the ground truth in Figure 4 We the residue barely deteriorates when the frequency increases, whereas it becomes extremely hard for MLPs optimize when k 16, 32. We refer 2D case the appendix
wAk
071
00.1
wmmmmm
10"4
10-1
10
3
phan Www mww Mwl
KARIC
KOSEZ
M
10-4
Barabon
Figure 4: Solving PDEs. L and H losses of MLP and KAN with different frequencies of the solution.
CONCLUDING REMARKS
In this work, we have compared the approximation and representation properties of KANs and MLPs. Based upon our theoretical and empirical analysis, we conclude that while KANs and MLPs are very similar from the perspective of approximation theory, i.e. the both parameterize similar classes of functions, they differ significantly in terms of their training dynamics. Specifically, KANs do not exhibit the same spectral bias toward low frequencies that MLPs do. Future work includes developing theory which can describe the training of deeper KANs, performing a more extensive experimental investigation of the spectral bias of KANs more complicated problems, and investigating whether the reduced spectral bias of KANs improves their performance on scientific computing applications such solving PDEs.
10

---

Published as a conference paper at ICLR 2025
6 ACKNOWLEDGEMENTS
The research of YW and TYH was in part supported by NSF Grant DMS- 2205590 and the Choi Family Gift Fund. JWS supported by the National Science Foundation (DMSS-4223055) as well as the Office of Naval Research (MURI ONR grant N00014-20-1-22 10-7777 We acknowledge Dr. Juncai He for identifying a minor mistake in the proof of Thm 3.3 and the referees' comments for improving the paper.
REFERENCES
Diab W Abueidda, Panos Pantidis, and Mostafa Mobasher. Deepokan: Deep operator network based on kolmogorov arnold networks for mechanics problems. arXiv preprint arXiv:2405. 19143, 2024.
Robert A Adams and John JF Fournier. Sobolev spaces. Elsevier, 2003.
Alireza Afzal Aghaei. fkan: Fractional kolmogorov- networks with trainable jacobi basis functions. arXiv preprint arXiv:2406.07455. 2024a.
Alireza Afzal Aghaci. rkan: Rational kolmogorov-amoll networks. arXiv preprint arXiv:2406. 144459440..14455 2024b.
Tashin Ahmed and Md Habibur Rahman Sifat. Graphkan: Graph kolmogorov arnold network for small molecule- protein interaction predictions. In JCML'24 Workshop fL Life and Material Science: From Theory to Industry Applications, 2024.
Basim Azam and Naveed Akhtar. Suitability of kans for computer vision: A preliminary investigation. arXiv preprint arXiv:2406.00000 2024.
Andrew R Barron. Universal approximation bounds for superpositions of sigmoidal function. IEEE Transactions on Information theory, 39(3):930-94) 1993.
Peter L Bartlett, Nick Harvey, Christopher Liaw, and Abbas Mehrabian. Nearly-tight vc-dimension and pseudodimension bounds for piecewise linear neural networks. Journal of Machine Learning Research, 20(63):1-17, 2019.
Ronen Basri, Meirav Galun, Amnon Geifman, David Jacobs, Yoni Kasten, and Shira Kritchman. Frequency bias neural networks for input of non-uniform density. In International conference machine learning, pp. 685-694. PMLR, 2020.
Nuri Benbarka, Timon Höfer, Andreas Zell, al. Seeing implicit neural representations fourier series. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2041-2050, 2022.
Alexander Dylan Bodner, Antonio Santiago Tepsich, Jack Natan Spolski, and Santiago Pourteau. Convolutional kolmogorov-arnold networks. arXiv preprint arXiv:2406.13//5. 2024.
Zavareh Bozorgasl and Hao Chen. Wav-kan: Wavelet kolmogonov-aroold networks. arXiv preprint arXiv:2405. 12833, 2024.
Jürgen Braun and Michael Griebel. On a constructive proof of kolmogorov's 's superposition theorem. Constructive approximation, 30:653-677 2009.
Roman Bresson, Giannis Nikolentzos, George Panagopoulos, Michail Chatzianastasis, Jun Pang, and Michalis Vazirgiannis. Kagnns: Kolmogorov-arood networks meet graph learning. arXiv preprint arXiv:2406. 18380, 2024.
Wei Cai and Zhi Qin John Xu. Multi-scale deep neural networks for solving high dimensional pdes. 2019.
arXiv preprint
Minjong Cheon. Demonstrating the efficacy of kolmogorov- arool networks in vision tasks. arXiv preprint arXiv:2406.149 149/6, 2024a.
11

---

Published as a conference paper at ICLR 2025
Minjong Cheon. Kolmogorov arnold network for satellite image classification in remote sensing. arXiv preprint arXiv:2406.00600 2024b.
George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems, 2(4):303-314, 1989.
Gianluca De Carlo, Andrea Mastropietro, and Aris Anagnostopoulos. Kolmogorov-armold graph neural networks. arXiv preprint arXiv:2406. 18354, 2024.
Ronald De Vore, Boris Hanin, and Guergana Petrova. Neural network approximation. Acta Numerica, 30:327-44 44227-44,,22 2021.
Ronald A Vore. Nonlinear approximation. Acta numerica, 7:51-150, 1998.
Ronald DeVore and George Lorentz. Constructive approximation. Grundlehren der mathematis-
chen Wissenschaflen, 1993.
Sara Fridovich-Kan Keil, Raphael Gontijo Lopes, and Rebecca Roelofs. Spectral bias in practice: The role of function frequency in generalization. Advances in Neural Information Processing Systems, 35:7368-7333 2022.
Remi Genet and Hugo Inzirillo. A temporal kolmogorov-armold transformer for time series forecasting. arXiv preprint 24Xi::2406002866 2024a.
Remi Genet and Hugo Inzirill.. Tkan: Temporal kolmogorov-arnold networks. arXiv preprint arXiv:2405.07344, 2024b.
Juncai He, Lin Li, Jinchao Xu, and Chunyue Zheng. Relu deep neural networks and linear finite elements. arXiv preprint vrXiv:1807.0333 2018.
Luis Fernando Herbozo Contreras, Jiashuo Cui. Leping Yu, Zhaojing Huang, Armin Nikpour, and Omid Kavehei. Kan-eeg: Towards replacing backbone-mlp for an effective seizure detection system. medRxiv, pp. 2024-06, 2024.
Sean Hon and Haizhao Yang. Simultaneous neural network approximation for smooth functions. Neural Networks, 154:152-164, 2022.
Qingguo Hong, Jonathan W Siegel, Qinyang Tan, and Jinchao Xu. On the activation function dependence of the spectral bias of neural networks. arXiv preprint (Bt::2208.049224 2022.
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators, Neural networks, 2(5):359-366, 1989.
Joel Horowitz and Enno Mammen. Rate-optimal estimation for a general class nonparametric regression models with unknown link functions. Annals of Statistics, 3566):259--2619, 2007.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Kari Karhunen. Under lineare methoden in der wahr scheinlichkeitsrechnnng Annales Academiae Scientiarun Fennicae Series Al: Mathematia Physica, 47, 1947.
Mehrdad Kiamari, Mohammad Kiamari, and Bhaskar Krishnamachari. Gkan: Graph kolmogorov- arnold networks. arXiv preprint arXiv:2406. 06470, 2024.
Jason M Klusowski and Andrew R Barron. Approximation by combinations of ReLU and squared ReLU ridge functions with Ll and l controls. IEEE Transactions Information Theory, 64(12) (4(1)): 7649-7656, 2018.
Michael Kohler and Sophie Langer. On the rate of convergence of fully connected deep neural network regression estimates. The Annals of Statistics, 49(4):2211-224) 2021.
A.N. Kolmogorov. On the representation continuous functions of several variables as superpositions of continuous functions of a smaller number of variables. Dokl. Akad. Nauk, 108(2), 1956.
12

---

Published as a conference paper at ICLR 2025
Andrei Nikolaevich Kolmogorov. On the representation of continuous functions of many variables by superposition of continuous functions of one variable and addition. In Doklady Akademii Nauk, volume 114, pp. 953-956. Russian Academy of Sciences, 1957.
Akash Kundu, Aritra Sarkar, and Abhishek Sadhu. Kanqas: Kolmogorov arnold network for quantum architecture search. arXiv preprini arXiv:2406. 17630, 2024.
Moshe Leshno, Vladimir Ya Lin, Allan Pinkus, and Shimon Schocken. Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural networks, 6(6):861 1-66 1993.
Chenxin Li, Xinyu Liu, Wuyang Li. Cheng Wang, Hengyu Liu, and Yixuan Yuan. U-kan makes strong backbone for medical image segmentation and generation. arXiv preprint arXiv:2406.02918, 2024a.
Xinhe Li, Zhuoying Feng, Yezeng Chen, Weichen Dai, Zixu He, Yi Zhou, and Shuhong Jiao. Coeff- kans: A paradigm to address the electrolyte field with kans. arXiv preprint arXiv:2407.20265 2024b.
Ziyao Li. Kolmogorov-armold networks are radial basis function networks. arXiv:2405.0672), 2024.
arXiv preprint
Hao Liu, Jin Lei, and Zhongzhou Ren. From complexity to clarity: Kolmogorov-aroold networss in nuclear binding energy prediction, 2024a. /tt/://tpp///rrrrr..orr/////////
Mengxi Liu, Sizhen Bian, Bo Zhou, and Paul Lukowicz. ikan: Global incremental learning with kan for human activity recognition across heterogeneous datasets. arXiv preprint wrXiv:2406.01646, 2024b.
Mengxi Liu, Daniel Geißler, Dominique Nshimyimana, Sizhen Bian, Bo Zhou, and Paul Lukowiez. Initial investigation of kolmogorov-armold networks (kans) as feature extractors for imu based human activity recognition. arXiv preprint arXiv:2406. 19006.//9994 /914, 2024c.
Ziming Liu, Pingchuan Ma, Yixuan Wang, Wojciech Matusik, and Max Tegmark. Kan 2.0:
Kolmogorov-arnold networks meet science. arXiv preprint arXiv:2448.102005 2024d.
Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruchle, James Halverson, Marin Soljačić, Thomas Y Hou, and Max Tegmark. Kan: Kolmogorov-arnold networks. arXiv preprint arXiv:2404. 19756, 2024e.
Jianfeng Lu, Zuowei Shen, Haizhao Yang, and Shijun Zhang. Deep network approximation for smooth functions. SIAM Journal on Mathematical Analysis, 53(5):5465-5506, 2021.
George Nehma and Madhur Tiwari. Leveraging kans for enhanced deep koopman operator discovery. arXiv preprint arXiv:2406.02875, 2024.
Tiago Novello, Diana Aldana, and Luiz Velho. Taming the frequency factory of sinusoidal networks. arXiv preprint arXiv:2407.21122 2024.
Yanhong Peng, Miao He, Fangchao Hu, Zebing Mao, Xia Huang, and Jun Ding. Predictive modeling of flexible ehd pumps using kolmogorov-amold networks. arXiv preprint arXiv:2405.07488, 2024.
Tomaso Poggio, Kenji Kawaguchi, Qianli Liao, Brando Miranda, Lorenzo Rosasco, Xavier Boix, Jack Hidary, and Hrushikesh Mhaskar. Theory of deep learning iii: the non-overfitting puzzle. CBMM Memo, 73(1-88):2 2018.
Pawel Pratyush, Callen Carrier, Suresh Pokharel, Hamid Ismail, Meenal Chaudhari, and Dukka B KCC Calmphoskan: Prediction of general phosphorylatio sites in proteins via fusion of codon aware embeddings with amino acid aware embeddings and wavelet-based kolmogorov arnold network. bioRxiv, pp. 2024-07, 2024.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International conference machine learning, pp. 5301-5310. PML.R, 2019.
13

---

Published as a conference paper at ICLR 2025
Pratik Rathore, Weimu Lei, Zachary Frangella, Lu Lu, and Madeleine Udell. Challenges in training pinns: loss landscape perspective. In Forty-first International Conference on Machine Learning. 2024.
Spyros Rigas. Michalis Papachristou, Theofilos Papadopoulos, Fotios Anagnostopoulos, and Georgios Alexandridis. Adaptive training of grid-dependent physics-informed kolmogorov-arnole networks. arXiv preprint arXiv:2407. 17 2024.
Basri Ronen, David Jacobs, Yoni Kasten, and Shira Kritchman. The convergence rate of neural networks for learned functions of different frequencies. Advances Neural Information Processing Systems, 32, 2019.
Johannes Schmidt- Hieber. Nonparametric regression using deep neural networks with relu activation function. The Annals of Slatistics, 48(4); 1875, 2020.
Seyd Teymoor Seydi. Exploring the potential of polynomial basis functions in kolmogorov-aroold net- works: A comparative study of different groups of polynomials. arXiv preprini arXiv:2406600258,, 2024a.
Seyd Teymoor Seydi. Unveiling the power of wavelets: wavelet-based kolmogorov-arnold network for hyperspectral image classification. arXiv preprint arXiv:2406.0786) 2024b.
Haoran Shen, Chen Zeng, Jiahui Wang, and Qiao Wang. Reduced effectiveness of kolmogonov-arnold networks on functions with noise. arXiv preprint arXiv:2407.14882, 2024.
Zuowei Shen, Haizhao Yang, and Shijun Zhang. Optimal approximation rate of ReLU networks in terms of width and depth. Journal de Mathématiques Pures et Appliquées, 157:101-135, 2022.
Khemraj Shukla, Juan Diego Toscano, Zhicheng Wang, Zongren Zou, and George Em Karniadakis. A comprehensive and fair comparison between mlp and kan representations for differential equations and operator networks. arXiv preprint urXiv:2406.029)) 02917, 2024.
Jonathan W Siegel. Optimal approximation rates for deep relu neural networks on Sobolev and Besov spaces. Journal of Machine Learning Research, 24(357):1-52, 2023.
Jonathan W Siegel and Jinchao Xu. Approximation rates for neural networks with general activation functions. Neural Networks, 128:313-322 2020.
Jonathan Siegel and Jinchao Xu. Sharp bounds on the approximation rates, metric entropy, and n-widths of shallow neural networks. Foundations Computational Mathematics, pp. 1-57, 2022.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances neural information processing systems, 33:7462-7473, 2020.
Sidharth SS. Chebyshev polynomial- based kolmogorov-anne arroold networks: An efficient architecture for nonlinear function approximation. arXiv preprint arXiv:2405.07200, 2024.
Hloang-Thang Ta. Bsrbf-kan: A combination of b-splines and radial basic functions in kolmogorov- arnold networks. arXiv preprint arXiv:2406. 2024.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara -ridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in neural information processing systems, 33:7537-7547, 2020.
Cristian J (kans) for time series analysis. arXiv preprint arXiv:2405.08799 2024.
Luis Blanco, Roberto Pereira, and Màrius Caus. Kolmogorov networks
Sifan Wang, Xinling Yu, and Paris Perdikaris. When and why pinns fail to train: A neural tangent kernel perspective. Journal of Computational Physics, 449: 10768, 2022.
14

---

Published as a conference paper at ICLR 2025
Yizheng Wang, Jia Sun, Jinshuai Bai, Cosmin Anitescu, Mohammad Sadegh Eshaghi, Xiaoying Zhuang. Timon Rabezuk, and Yinghua Liu. Kolmogorov aroll informed neural network: A physics-informed deep learning framework for solving pdes based kolmogorov arnold networks. arXiv preprint arXiv:2406. 1045, 2024.
Jinchao Xu. Finite neuron method and convergence analysis. Communications in Computational Physics, 28(5): 177-117777 2020.
Jinfeng Xu, Zheyu Chen, Jinze Li, Shuo Yang, Wei Wang, Xiping Hu, and Edith NgH Fourierkan- gef: Fourier kolmogorov-arnole network-an effective and efficient feature transformation for graph collaborative filtering. arXiv preprint arXiv:2406.01034, 2024a.
Kumpeng Xu, Lifei Chen, and Shengrui Wang. Kolmogorov-armold networks for time series: Bridging predictive power and interpretability. arXiv preprint arXiv:2406.02496, 2024b.
Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo, Yanyang Xiao, and Zheng Ma. Frequency principle: Fourier analysis sheds light deep neural networks. arXiv preprint arXiv: 190.06633,, 2019a.
Zhi-Qin John Xu, Yaoyu Zhang, and Yanyang Xiao. Training behavior of deep neural network in frequency domain. In Neural Information Processing: 26th International Conference, ICONIP 20/9, Sydney, NSW, Australia, December 12-15, 2019, Proceedings, Part 26, pp. 264-274. Springer, 2019b.
Shangshang Yang, Linrui Qin, and Xiaoshan Yu. Endowing interpretability for neural cognitive diagnosis by efficient kolmogorov-amoll networks. arXiv preprint S4Ki::2405.13999 2024.
Yahong Yang and Yulong Lu. Optimal deep neural network approximation for korobov functions with respect to sobolev norms. arXiv preprint arXiv:2311.04779, 2023.
Yahong Yang, Yue Wu, Haizhao Yang, and Yang Xiang. Nearly optimal approximation rates for deep super relu networks sobolev spaces. arXiv preprint 10X 10766, 2023.
Dmitry Yarotsky. Error bounds for approximations with deep relu networks. Neural Networks, 94: 103-114, 2017.
Dmitry Yarotsky. Optimal approximation of continuous functions by very deep ReLU networks. In Conference Learning Theory, pp. 639 649. PMLR, 2018.
Dmitry Yarotsky and Anton Zhevnerchuk. The phase diagram of approximation rates for deep neural networks. Advances in neural information processing systems, 33:1
2020.
Bing Yu al. The deep ritz method: a deep learning- based numerical algorithm for solving variational problems. Communications in Mathematics and Statistics, 6(1):1-12, 2018.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications the ACM, 64(3):107-115, 2021.
Fan Zhang and Xin Zhang. Graphkan: Enhancing feature extraction with graph kolmogorov arnold networks. arXiv preprint arXiv:2406. 13597, 2024.
Shijun Zhang, Hongkai Zhao, Yimin Zhong, and Haomin Zhou. Why shallow networks struggle with approximating and learning high frequency: A numerical study. arXiv preprint arXiv:2306.174: 2023.
Xiao Zhang, Haoyi Xiong, and Dongrui Wu. Rethink the connections among generalization, memo- rization and the spectral bias of dnns. arXiv preprint arXiv:2004.1395.4 2020.
15

---

Published as a conference paper at ICLR 2025
A 2D POISSON EQUATION WITH HIGH FREQUENCY COMPONENTS
In this experiment, we solve a 2D Poisson equation with a figh-reeueecy solution using the Deep Ritz method. The equation with zero Dirichlet boundary condition has the form
Myy f in(o, 1J²,, u(x, 0) u(z, 1) = u(0, y) w(1, - 0.
(14)
with right-hand side and corresponding exact solution depending on the frequency k
2nx²² sin(
sim(k sin (ATry),
u = sin(Tx)) sin(Ty))+ + T
sin(krx) sin (kTy).
For frequencies A 2, 4, 16, we use a uniform grid with 101 sample points per direction with MLP of 6 layers and 256 neurons in each hidden layer and a KAN of 2 layers with 10 neurons in the hidden layer. We use LBFGS for 50 iterations and plot the loss compared the ground truth in Figure 5 We observe similarly as in the ID case that KANs outform MLPs when solving equation with multiple scales.
More importantly, MIPs cannot learn the correct shape, even with k = 4 after 100 iterations, whereas KANs can learn solution quite close to the ground truth with just 5 iterations for and 10 iterations for k = 16; see Figure 6
14
301
184
84-
15
8
801
184
Figure 5: 2D Poisson. Losses of MLP and KAN with different frequencies of the solution.
B ID WAVES OF INCREASING AMPLITUDES
C PROOFS
Proof of Theorem 3.2 We will show that each layer of an MLP with the activation function d can be represented by KAN with two hidden layers, width W and grid size G with degree B-splines. By composing such layers, we obtain the desired result.
For single layer of MLP, we consider the linear part and the non-linear activation separately. We first observe that on any compact subset of R the linear function
w
in + by asjay
(15)
j=0
can be represented with single KAN layer of width W by setting di to the linear function
(x, = aiya
(16)
We claim that this linear function can be exactly represented on any interval [-R, R] in the form To do this, we first set Wi 0 and choose the grid points for the B-splines to be
(- 1)R, - (2k 3) R) .... - Rt R. .... (2k - R, (2k: 1)R).
Note that based upon the KAN architecture, this corresponds to the extension of the uniform grid to = - R,t, = R which has grid size G = 1. It is also easy to verify that there are (A: + 1) B-splines supported on this grid, whose restriction to R, R] span the space of polynomials of degree k. Thus,
16

---

Published as a conference paper at ICLR 2025
JAN_scalo__
1.0
0.8
0.6
0.6
0.6
0.4
0.0
0.0
0.0
0.0
0.2
0.4
0.6
c.0
0.4
n2
0.4
0.5
C.0
0.2
0
0.4
0.5
0.8
10
0.8
1.
0.0
0.8
1.0
KAN_ KAN_scale__ 16
truth_sccet scale 16
1.0
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
1.0
0.8
0.8
0.6
0.6
0.0
0.2
0.4
0.2
0.4
0.4
0.6
0.2
0.4
0.6
0.2
0.8
0.0
0.8
1.0
Figure 6: 2D Poisson. True solution and neural net solutions.
MalddeL Pirs Copi
44
Wethdd
1500 1113
110000000 181
Figure 7: ID wave dataset, where the target function has increasing amplitudes of different frequency modes. Under various hyperparameters, MLPs manifest severe spectral biases (top), while KANs do not (bottom). Note that the axis (training steps) of MLP is 10 times that of KAN.
17

---

Published as a conference paper at ICLR 2025
in particular, any linear function on [-R, R] can be represented linear combination of these B-splines.
Next, we consider the non-linear activation, which is given by the coordinatewise application of o i.c.
s.put (xx)).
(17)
This can be represented by single hidden layer KAN by setting
{ 0
(x)
i= j
(1)(2))
(18)
i + j.
We claim that the functions σg can be represented in the form (4) on any finite interval [-R, R). To to this, we again set 0 and choose the grid points for the B-splines to be
{- (k: I)R, .... R. ......
R) (h: R)
This grid is the grid extension of the uniform grid to - R2,4 0,12 R which has grid size G= easy also to verify that there are (A: + 2) B-splines supported on this grid and that any piecewise polynomial on [- R. R] with a single breakpoint at O which is C*-1 is linear combination of these B-splines. Hence the function σg can be represented on R, R in the form using this grid.
The proof is now completed by composing these layers and choosing R sufficiently large so that for any input x Zt (which is bounded) the inputs and outputs of every neuron in the original MLP lie in the interval [RR, It].
Proof of Theorem 3.3 The proof follows from the fact that each KAN activation function Φ1,i,j can be represented by ReLU* neural network with one hidden layer of width G + 2k + 1. This is due the fact that each di. a linear combination of B-splines (since we are assuming that w, (4). This implies that it must be a compactly support piecewise polynomial spline of degree & whose (T 1)-st derivative changes only on the extended grid which has 2k + points. Based upon this, can construct shallow ReLU network with one hidden node per grid point which matches each function Øt.i,j (see for instance Xu (2020)). Since there are a total of W such functions in each layer, this means that we can implement a single KAN layer using a shallow ReLU* network of obtain the desired MLP representing the
in
width (G + 2k + L)W? By composing these networks, KAN. We remark that after grid extension, the grid points for each for each of the spline functions $1.i, may become different. For this reason, it is not clear whether the width in this construction can be reduced.
Proof of Theorem 4.7
We first observe from 11) that the matrix M is block diagonal with d' identical blocks. Denoting these (G + h 1)d x (G + k 1)d blocks by B, it thus suffices to prove that
M(t+--na(B) SC BB
(19)
To do this, we analyze the blocks and note that they take the form
C D
D
C
D
B
(20)
:
D
D
C
Here the diagonal sub-blocks C' IR R(C+A-IIX(G+RRII) are the Gram matrix of the one-dimensional B-spline basis, i.e.
Cjj Cg= / (x)r)
(21)
and the off-diagonal sub-blocks D
are rank one matrices
D == vu
(22)
18

---

Published as a conference paper at ICLR 2025
where the vector RG+k-1 is given by
= (x(x)dx.
(23)
It is well-known that the Gram matrix C' is well-conditioned uniformly in G for fixed k. i.e. AG+R-1 (C)) S K for fixed constant K depending only upon k. See for instance Vore & Lorentz (1993). Theorem 4.2 in Chapter 5, where it is shown that the Lo-norm of spline and the properly scaled norm of its B-spline coefficients are equivalent up to a constant depending only on k. This is equivalent to the well-conditioning of the Gram matrix C.
In addition, we can easily verify using Jensen's 's inequality (or Cauchy- Schwartz) that N C. Indeed, that
letting E IR GG
w Dw = So f(w)dl)) $ $f f(x) = w Cw.,
(24)
where the function f(x) = Eim G+ w,),(r).
Let E Rd the vector of ones and note that
D D
D
D
D
(v (v01)(081)+
(25)
so that B - (u (v & 1)Th is block diagonal matrix with diagonal blocks C - D. We proceed to upper bound the largest eigenvalue B by
())) 1l
w Bw
D
0
C
(26)
max T
w + (w ( (
wl|=
C - D
Writing w (Wi,.., ..., wa) with w, RG++-1 and G+k-1 get the bound
Di=1
|| 2 and using that D vo". we
Maa)--na(B))) <
max
wi T Cw; +
Σ TTwe We
(0 w;)
||w + +1+++++laaa 1|= 1m1
(27)
Σ
wif Cw, + (d (d-1)) Σ (oFwwy wx
max
||,,12 124 + 11
Since D C. we have (v wz)2 < wi Ow: which gives the bound
\(G+R-1)a(B)) (3) S
max
Σ w Tw; dDo+--1C))))
(28)
[| [o/]|2 1 ||==1
i=1
Next, we lower bound the d-th eigenvalue of B. For this, we use the Courant-Fistie minimax theorem to see that
AB max min wT Bw, Wa weWa, ||]]|=
(29)
where the maximum is taken all subspaces of Wa of codimension < d. We consider the specific subspace
Wa = ((wi,... wa); v² w; = for all = 1,..,, d} @ span(v @ 1)
(30)
and observe that for any (Wi, wa) with oT u: 0 we have
d
" Bw = Σ w Cw. 2 Ar Σ
1 (C)
(31)
19

---

Published as a conference paper at ICLR 2025
while for the u @ 1 (which is orthogonal) we have
w² Bw Σ Co + (d 1) Σ ||al 2 ([00))/["=>((()lp
(32)
Thus, Ad(B) 2 \ Combining these bounds and using the well-conditionin of the Gram matrix C, we get
M(a+ Aa(B)
S d AG+R-I(0) < K
(33)
(I
for a constant K which only depends upon This completes the proof.
20


Reviews:
[{"review_id": "ESBrk7jbm5", "reviewer": "Reviewer_iJru", "summary": "The paper addresses the theoretical and empirical properties of Kolmogorov-Arnold Networks (KANs), particularly focusing on their approximation capabilities and spectral bias. Given the growing interest in alternative neural network architectures for function approximation, the study of KANs is relevant to the broader machine learning and scientific computing community. However, while the comparison with MLPs is insightful, the direct impact and importance of KANs over existing methods are not convincingly demonstrated.", "questions": "1.Theoretical Expansion: How does the depth of Kolmogorov-Arnold Networks (KANs) influence their spectral bias and approximation capabilities? Could a deeper theoretical analysis be provided to explore this relationship?\n\n2.Enhanced Experimental Design:Could the authors include more rigorous statistical analysis and comparative experiments with a broader set of baseline models, particularly state-of-the-art neural architectures used in scientific computing?\n\n3.Real-World Applications: How could the practicality of KANs be demonstrated in more diverse and realistic scenarios? Are there plans to showcase examples that highlight their advantages and limitations?\n\n4. Detailed Discussion: Could the authors provide a more detailed discussion on the computational cost, scalability, and limitations of KANs? What are the key factors that may affect their broader applicability?\n\n5. The author should", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.Theoretical Comparison of Expressiveness: The authors establish that KANs are at least as expressive as MLPs. Specifically, they show that any MLP can be reparameterized as a KAN with degree k splines which is only slightly larger. Conversely, they demonstrate that KANs can also be represented by MLPs, but the number of parameters increases significantly with the grid size of the KAN. This implies that KANs with large grid sizes may be more efficient in approximating certain classes of functions.\n2. The exploration of KANs’ spectral bias and their comparison with MLPs provides a novel perspective. The use of B-splines and the grid extension method offer unique contributions. However, much of the theoretical groundwork relies heavily on established techniques.", "weaknesses": "1. The parameter count in KANs and MLPs are not directly comparable, and the authors have blurred the distinction between these concepts. Additionally, the depth of KANs differs from that of MLPs, and the authors should not directly compare them, as this leads to a conceptual confusion. \nWe acknowledge that the conceptual confusion arises from the discussion in Section 3, particularly in Theorem 3.1- 3.4, where we compare the parameter count and depth between KANs and MLPs. This issue is also present in the experimental results in Sec4.1 and 4.2, where parameter comparisons are made across different models. \n\n2. Theorem 4.1 lacks clarity regarding its implications for KAN's performance on high-frequency data. The authors should explicitly state whether the theorem suggests that KANs outperform other models in handling high-frequency data. Additionally, it would be beneficial to provide a brief discussion or corollary connecting the theorem to spectral bias or high-frequency performance to clarify its practical significance.\n\n3.The theoretical analysis provided in the paper is limited to shallow KANs, which restricts the generalizability of the conclusions regarding spectral bias. To enhance the practical relevance, the authors should discuss the limitations of their current analysis and suggest potential methods for extending the theoretical framework to deeper KANs. This would provide a more comprehensive understanding of the spectral bias in real-world applications.\n\n4.While the experiments cover diverse applications, they lack rigorous statistical validation to substantiate the claimed advantages of KANs. The authors should include specific statistical tests or validation methods to strengthen their results. Furthermore, more detailed comparisons with state-of-the-art baseline methods would help provide a stronger foundation for the claimed benefits of KANs, especially in practical applications like PDE solving and Gaussian random field fitting.\n\n5.Incomplete Problem Contextualization: The paper does not clearly establish the practical significance of overcoming spectral bias in real-world tasks, nor does it compare KANs against state-of-the-art methods beyond MLPs.\n\n6.Limited Discussion on Limitations: The potential downsides of KANs, such as computational cost, overfitting, and practical implementation challenges, are not thoroughly addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3IemAFyEEK", "reviewer": "Reviewer_EXrE", "summary": "This paper focuses on two aspects of KAN and MLP. The first issue is the approximation and representation capacility to functions of both networks, and the second issue is the spectral bias of approximation. Both issues are very important and attractive.", "questions": "I have following questions:\n\n(1).  All the theorems 3.1, 3.2 and 3.3 are extremely mathematical, and their assumptions are not agreeable with practical cases. How can the authors explain the main differences?\n\n(2). For the examples of PDE, the equation is very simplie, and the functions are of sufficient smoothness. If the functions concerned becomes chirp function and so on, how about the performance?\n\n(3) In Fig. 4, the performance of approximation does not always tend to better. Can you explain the causes?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper characterizes the representation capacility of approximation of KAN and MLP, with a considerably rigorous mathematical analysis.", "weaknesses": "On the expressiveness, it is merely a theoretical interest, and the proof of mathematical analysis is over simplified by assumpting extremely shallow KAN network, which violates the practical deep network situation, and can not be regarded as equalivent to general case or nonlinear effect.  \n\nThe experiments are not sufficient, and the paper is mainly of theoretical approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K7swx27he3", "reviewer": "Reviewer_Wf2h", "summary": "This paper is both a theoretical and experimental study of KANs, a recently introduced architecture with parametrizable activation functions,\naiming at characterizing their expressivity and spectral biases in comparison to MLPs. Since KANs offer a lot of new potentialities, it is\na legitimate and relevant research subject to try to characterize these  in  relation to properties of the data, therefore expressivity and spectral biases constitute obviously a good angle of analysis. \n\nSo the first theoretical result concerns expressivness: it tells that an arbitrary MLP (with activation function in C^k typically) can be exactly mapped onto a KAN (with B-splines of order k) with a depth multiplied by 2 and requiering a grid size of 2, so overall the number of parameters is multiplied by 4 (if I interpret well the result, this could be stated more explicitly actually) in this mapping. In reverse they find a mapping of a KAN to an MLP with an increase of parameters of G, the grid size of the original KAN. These constructions lead the authors to conclude qualitatively that KAN are more expressive than MLP. As acknowledged by the authors, this is not a sharp result as it is not excluded that a better construction of the mappings could be found leading to different conclusions, but these constructions are interesting on their own. A more concrete statement on expressivity is given by corollary 3.4, giving the speed of approximating of any function on a Sobolev space w.r.t the number of parameters of a KAN. Here, as a non specialist of this kind of result I become slightly confused. The statement  is seemingly based on generic results on MLP, and seems not to be specific to KAN but more to multi-layer models, as it exploit the  aforementioned mapping of the KAN to MLP, so that I wonder why MLP with relu-k would not lead to the same statement?\nThere is sentence discussing this at the end of the paragraph which is cryptic to me as it refers to some apparently non-standard MLP with relu-k.  As a result the overall take-home message of this part is unclear to me. Is there any advantage to be attributed to KAN wrt to comparable MLP under this measure of expressiveness? Maybe the statement 3.4 is a profound and central statement of the paper, but the discussion should make it clear, by first giving how standard MLP with relu-k compare and then also maybe what are the practical consequence of such statement if any?\n\nThe second theoretical point concerns spectral bias, and theorem 4.1 to me is a nice and non-trivial statement showing that a single layer of B-splines is well conditioned, especially when the grid size becomes large, by looking at the spectrum of the Hessian of the loss. This fully characterizes the conditioning of the problem with explicit dependency w.r.t. embedding dimension and grid size. The comparison is done with MLP with Relu using a result given in Hong et al. (2022) obtained with similar techniques (Courant Fisher theorem) showing  a much larger spectral bias for MLP. I wonder however if a  comparison with the spectral bias of MLP with relu-k (unfortunately not to given in Hong et al, but I presume that similar techniques would lead to the result) would not be more relevant as they are in the same class of regularity as KAN with B-splines of order k?\n\nThe experimental part focuses on the spectral bias question on diverse examples. A comparison is shown between KAN with kth order B-splines and MLP with relu, showing a clear advantage to KAN regarding spectral bias, but again I wonder why the comparison is not done with the spectral bias of MLP with relu-k, if this  would not be be more meaningful?", "questions": "- Is the comparison between KANs with kth order B-splines with MLP modified when considering relu-k instead of simple relu?\n- In Figure 3 and 4 the learning of KANs display instabilities in contrary to MLP which are attributed to grid extension during the undersampled regime, I don't really understand what is meant by this, is it that the grid size of KAN is progressively extended during learning? \n\nFinally as a minor point:\nI found that the way the experimental results are presented could be more concise and more informative. Figures 1 and 2 are almost identical, while all the plots given in figure 3 and 4 could be possibly synthetized to give the general behaviour w.r.t. to k, grid size, dimension and scale, and the legend should be magnified, I can't read anything on a printed version.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "the questions addressed in the paper are interesting and some of the statements are quite strong, in particular th. 4.1 on the characterization of the spectral bias. Overall, my non-specialist viewpoint  on this paper is that it looks like a rather clear and solid paper, with interesting statements.", "weaknesses": "I found however a potential problem  of coherence concerning the comparison of KANs with MLPs: with relu-k at the beginning which is indeed legitimate,  and then basic relu starting from corollary 3.4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RfJD6kt3mR", "reviewer": "Reviewer_T7tm", "summary": "The paper addresses the expressivity Kolmogorov-Arnold Networks (KAN) and the spectral bias.\n\nA rigorous analysis of the representation power of the KAN vs MLP is welcome. \n\nI think the most important contribution of the paper is to show the relationship between MLP with ReLU^k activation function and KAN without bias. Th.3.2 and Th.3.3 bounds are not tight. A conclusion of the authors is that MLP scales with G^2 while KAN with G, the grid size. The additional contribution Cor3.4 is not well explained in relationship to MLP, but the authors highlight the potential advantage of KANs. \n \nThe Analysis of the spectral bias seems a repetition of the analysis provided in [1], even if the authors analyze the Hessian matrix instead of the NTK matrix and show the elements of the Hessian. The authors are able to justify the reason why low frequency are not prioritized as in MLP, even if the connection with Th.4.1 is not clear.  \n\nThe analysis of PDE, seems to be the same are reported in [1], showing better performance of KAN-based models in the training loss. Not sure why the test loss is not reported. \n\n\n[1] 1. Wang, Y. et al. Kolmogorov Arnold Informed neural network: A physics-informed deep learning framework for solving forward and inverse problems based on Kolmogorov Arnold Networks. Preprint at https://doi.org/10.48550/arXiv.2406.11045 (2024).", "questions": "1. could you expand on the consequences of Corollary 3.4\n2. would be interesting to understand the consequence of thr.3.2 and th.3.3 for more \"mainstream\" architecture. For example, relu^k k=1 is probably the most common, while when gradients are needed SiLU SiLU-based activation functions are required, which is typical for scientific applications\n3. would be nice to expand the Hessian analysis and connect with the NTK one\n4. would be possible to perform the spectral analysis for section 4.4, instead of the error trend, if not, what is the test error?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "As described above:\n\n1. find the connection between two specific classes of MLP and KAN. \n2. interpretation of the spectral bias using the Hessian matrix\n3. some experiments to visualize the spectral bias", "weaknesses": "As reported in the summary:\n1. the theoretical analysis is limited to specific class of MLP and KAN\n2. experiments seem very similar to [1], so it is not clear, apart from Fig.1 and Fig.2, why they are included.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)