Generalization Bounds and Model Complexity for Kolmogorov–Arnold Networks
Authors: Xianyang Zhang
Huijuan Zhou
Venue: ICLR 2025 Poster (2025)
Keywords: Deep neural networks
Empirical process
Generalization bounds
Kolmogorov-Arnold representation theorem
Model complexity
Primary area: learning theory
TL;DR: Generalization Bounds and Model Complexity for Kolmogorov–Arnold Networks

Abstract: Kolmogorov–Arnold Network (KAN) is a network structure recently proposed in Liu et al. (2024) that offers improved interpretability and a more parsimonious design in many science-oriented tasks compared to multi-layer perceptrons. This work provides a rigorous theoretical analysis of KAN by establishing generalization bounds for KAN equipped with activation functions that are either represented by linear combinations of basis functions or lying in a low-rank Reproducing Kernel Hilbert Space (RKHS). In the first case, the generalization bound accommodates various choices of basis functions in forming the activation functions in each layer of KAN and is adapted to different operator norms at each layer. For a particular choice of operator norms, the bound scales with the $l_1$ norm of the coefficient matrices and the Lipschitz constants for the activation functions, and it has no dependence on combinatorial parameters (e.g., number of nodes) outside of logarithmic factors. Moreover, our result does not require the boundedness assumption on the loss function and, hence, is applicable to a general class of regression-type loss functions. In the low-rank case, the generalization bound scales polynomially with the underlying ranks as well as the Lipschitz constants of the activation functions in each layer. These bounds are empirically investigated for KANs trained with stochastic gradient descent on simulated and real data sets. The numerical results demonstrate the practical relevance of these bounds.

Published as a conference paper at ICLR 2025
GENERALIZATION BOUNDS AND MODEL COMPLEX- ITY FOR KOLMOGOROY-ARNOLD NETWORKS
Xianyang Zhang
Huijuan Zhou
Department of Statistics
School of Statistics and Data Science Shanghai University of Finance and Economics Shanghai, China zh thouhuijuannmar shufe. edu. cn
Texas A&M University
College Station, TX 77843, USA
zhangxiany@stat. tamu edu
ABSTRACT
Kolmogorov-Amold Network (KAN) is network structure recently proposed Liu et al. (2024c) that offers improved interpretability and a more parsimonious design many science-oriented tasks compared to multi-layer perceptrons. This work provides a rigorous theoretical analysis of KAN by establishing generaliza- tion bounds for KAN equipped with activation functions that are either represented by linear combinations of basis functions lying in a low-rank Reproducing Ker- nel Hilbert Space (RKHS). the first case, the generalization bound accommo- dates various choices of basis functions in forming the activation functions in each layer KAN and adapted to different operator norms at each layer. For a partic- ular choice of operator norms, the bound scales with the norm of the coefficient matrices and the Lipschitz constants for the activation functions, and it has no dependence combinatorial parameters (e.g., number of nodes) outside of loga- rithmic factors, Moreover, our result does not require the boundedness assumption on the loss function and, hence, applicable to general class of regression-type loss functions. In the low-rank case, the generalization bound scales polynomially with the underlying ranks as well tions in each layer. These bounds are empirically investigated for KANs trained with stochastic gradient descent on simulated and real data sets. The numerical results demonstrate the practical relevance of these bounds.
Lipschitz constants of the activation func-
INTRODUCTION
The Kolmogorov-Amme Amnol representation theorem (KART) states that if f is a multivariate continuous function defined on [0, 1j then can be written finite composition of continuous functions of a single variable, and the binary operation of addition. More specifically, f can be represented by two-layer network in the form of
2d
f(z1.... Fzd
Σ the
Σ Wholaz)
1=0
where u/. : [0, 1] + RR and v/ : R + R. Motivated by KART, Liu et al. 2024c) introduced the Kol- mogorov-Aroodd Networks (KANs), expanding on the original two-layer network accommodate arbitrary depths. Mathematically, the KANs can be concisely described through the compositions of L. multivariate vector-valued functions:
KAN(x) L L-100... (x)),, x R
where L is matrix of univariate functions:
11,1,1
1/,,1
212,11
21,2,2
V

---

Published as a conference paper at ICLR 2025
with Wa R R. Here, we have defined
di-t
vx(x)= (e11(1))) (x))....)
Σ (e
for x = (dt,,..,,dd,... ) E Rdi-1 and Si S L. The node values for the ith layer are given by
x; = ( -i,d,) 4, V 1- o..... .... 1x()) where Ti Edi Waya (x9-1,6) is defined recursively.
34
Figure 1: Illustration of Kolmogorov-Arrol Networks, where the edges are associated with one- dimensional trainable functions while the nodes perform summation.
In contrast to multi-layer perceptions (MLPs), which use fixed activation functions on nodes, KANs utilize learnable activation functions along the edges. This architectural shift eliminates the need for conventional linear weight matrices. Instead, KANs replace each weight parameter with trainable one-dimensional function. In KANs, nodes serve aggregation points, summing incoming sig- nals without applying non-linear transformations; see Figure for an illustration. Liu et al. (2024c) demonstrated the potential of KANs for science-oriented tasks due to accuracy and interpretabil- ity. Subsequent research has ex explored the application of KANs in various domains such graphs (Bresson et al. [ 2024 De Carlo et al.) 20244) Kiamari et al. (022)), partial diffcrential equations (Wang cet al. 2024] Shukla al. (024), operator learning (Abueidda et al. Neh4aa & Tiwari 2024),
tabular data (Poeta et al. 2024). time series (Genet & Inziriloo] 2024, Vaca-Rubio et al. 2024) Xu
al. 2024b), human activity recognition (Liu al. 2024b) neuroscience (Herbozo Contreras al.,
2024 Yang et al. 2024), quantum science (Kundu et al. 2024 Li et al.) 2024b), computer vision
Azam & Akhtar 2024 Bodner al. 2024 Cheon 2024 Li al. 2024at [Sydi 2024b), kernel
learning (Zinage et al. 2024), nuclear physics (Liu et al. 2024a). electrical engineering (Peng et al.,
2024]. and biology (Pratyush et al. 2024). Unlike MLPs, KANs allow flexible way specify and
estimate the activation functions from training samples. Various choices of activation functions have Liu been recently investigated, including B-splines Liu et al. (2024c), wavelets (Seydi) 2024b Bozorgasl
& Chen 2024). radial basis functions (Li) 2024 Ta 2024), Fourier series (Xu et al.) (222aa) finite basis (Howard et al.) 2024), Jacobi basis functions (Aghaci) (024a)), polynomial basis functions (Syddi 2024a), and rational functions (Aghaci, 2024b). Additional techniques for KANs have also been proposed, such as regularization (Altarabichi 2024). Kansformer (combining transformer with KANs) (Chen et al. 2024), adaptive grid update (Rigas et al.) 2024), federated learning (Zeydan ct al. 2024), and Convolutional KANs (Bodner et al. 2024). A survey on the recent development of KANs can be found in Hou & Zhang 2024).
2

---

Published as a conference paper at ICLR 2025
1.1 CONTRIBUTIONS
Despite the increasing interest in applying KANs to various scientific problems and exploring the best ways to construct the network structure and training process, there is currently no recent re- search focusing on quantifying the complexity of KANs the best of our knowledge. Additionally, there is a need to understand how the performance of KANS is impacted by network structure and its complexity. The contribution of this work can be summarized as follows.
Theorem below will give the rigorous statement the generalization bound for KANs, which (i) accommodates different choices of basis functions in forming the activation func- tions each layer; (ii) adapted to various operator norms each layer; (iii) scales with the l norm of the coefficient matrices and the Lipschitz constants for the activation func- tions for a particular choice of operator norms; (iv) has no dependence on combinatorial parameters (e.g.. number of nodes) outside of logarithmic factors; (v) does not require the boundedness assumption on the loss function.
Theorem presents the generalization bound when the activation functions for each layer belong to Reproducing Kernel Hilbert Space (RKHS) with a low-rank structure. The bound scales polynomially with the underlying ranks as well as the Lipschitz constants of the activation functions in each layer.
We empirically investigate these bounds when the KANs are trained with stochastic gra- dient descent (SGD) simulated and real data sets. The results shed new light on how the performance of KANs affected by the underlying network structure and the network complexity. In particular, we observe that the complexity measure derived in Section 2 is tightly correlated with the excess loss, demonstrating the complexity measure's 's practical relevance. This theoretical-empirrcal connection suggests potential strategy for regu- larization, where the complexity measure could be used as a regularizer in designing the network structure and during the training process, either explicitly or implicitly.
1.2 RELATED WORKS
Generalization bounds for MLPs have long been studied in the literature on learning theory; see, e.g., Bartlet (1996); Anthony et al. (1999). More recently, Bartlet et al. (2017) provided a margin bound and corresponding Rademacher analysis that can be adapted to various operator norms at each layer. The proof technique therein was inspired by the early result for two-layer networks in Bartlet (196). Farrell al. (202T) established non-asymptotic bounds for MLPs for general class of nonparametric regression-tyee loss functions. [Schmidt- sparsely connected structure achieve the minimax rates of convergence (up logarithmic factor) under general composition assumption on the regression function. The aforementioned work has demonstrated that combining empirical process tools with considerations specific to deep neural networks (DNNs) can yield meaningful results. However, the practical relevance of these results for guiding DNN construction and training requires further investigation. Additionally, alternative theoretical tools have been developed to analyze DNNs, including the neural tangent kernel Jacot et al. 2018) and random matrix theory Mei & Montanari (2022).
(2020) showed that MLPs with
The contribution of the present work is provide a generalization bound for the novel KAN network structure, which is fundamentally different from the traditional MLPs as demonstrated in Liu et al. 2024cc The current analysis uses covering numbers and is closely related earlier covering number bounds in Anthony et al. (199) and the recent results in [artlett et al. (2017) for MLPs. The current proof also utilizes Maurey's 's sparsification lemma with suitable adaption the current setting. However, it is important to highlight some key differences with existing results for MLPs. Firstly, our results can be seen as a non-trivial extension of MLP results more flexible network structure specified through a more general choice of activation functions. Secondly, BBrrlett et al. (017)) focuses on the margin-based multiclass generalization bound, which has a bounded loss function (more precisely, the ramp loss). In contrast, our results cover wider range of loss functions (including the ramp loss for classification and general class of regression-type loss functions), which not required to be bounded. Thirdly, the results for KANs with low-rank structure in Section 2.3 appear to be new, and we are not aware of comparable results for MLPs in the recent literature.
3

---

Published as a conference paper at ICLR 2025
The rest of the article is organized as follows: We provide the generalization bounds for KANs equipped with activation functions that are either represented by a set of basis functions or lying in low-rank space in Sections 2.2 and 2.3] respectively. Section Doctionn3preee presents numerical results to support the theoretical findings. Section 4 concludes and discusses a few open problems for future research. The appendix contains all the proofs, additional discussions, and numerical results.
STATISTICAL ANALYSIS
We focus the supervised learning task, which aims to predict response using set of covariates.
Suppose we have n independent observations (x,,3k E X' x Y with S i S generated from some distribution P. where C Rd and y C Rdd, Let ((: be loss function defined on We consider the KAN-induced neural network class
M (f(-) V L- ..... 1.()) L E Fi, 1,2, L}.
for some pre-specified function classes Fú, where & E Fi is a map from R4 to Rd, Given any M, we define the generalization eπror (or risk) R(f) [f)( y)] for (x. y) P. To find the optimal predictor within M, we minimize the empirical risk function
min JEM T2
Σ [(f(x)).
Our goal here is to establish a bound on R(f) for every E M. In particular, let f be the predictor resulting from an optimization scheme (e.g.. SGDD, Adam, or L--FGGS that solves (2). Our result implies bound on the generalization error R(f).
2.1 NOTATION AND DEFINITIONS
We introduce the notion of proper covering numbers. For set U4 equipped with some norm || ||.
let N (l,,, e, II) be the smallest number of a subset V of U such that for any u E U, there exists
a v E so that ||0 u|| S e. Let X = [x1,..., , x,] E Ru for x; E Rd and F a class of
multivariate vector-valued functions mapping from IR to R For any V E F, write Lr (X)
(xI),. (x..)) E Raxm For matrix A (aij), we denote by ||A]|2 V Eis dz its Frobenius norm.
2.2 RISK ANALYSIS: ACTIVATION FUNCTIONS REPRESENTED BY BASIS FUNCTIONS
Let (X) = ( o 1 o (X) V, E Fi, 1,2, ..........) i). where ( X is the identity map. Given e,, P: > 0 for 1 1 i < L, we define 8 = e and $5 = Σ [= )3 e for for > The subscript in the norm
... L, where we set I[ indicates an index and does not refer to any 4 norm.
2.....
below merely
Proposition 1. Suppose Li Fi is map from Rdi-1 to R4, which satisfies [X) ((X)), s X - - for P: > 0 and X, X' e Buxd, Then we have
(L),
L-1
(3)
II
N( L 10... ( & :.....
sup
0=
where when i = 0, the first term in the product is given by N ((V(()) v WeF.),,
Proposition establishes an upper bound on (X),.X 81,, 1x.)) in terms of the covering numbers
N( L 0..0..... (X) T F.+1), (41. k+1))
It is built upon iterative argument that generalizes those for MLPs (Anthony et al. 1999 Bartlett et al. 2017) and is adapted to various operator norms (i.c., ll at each layer. To better understand these quantities, will start by presenting result when the activation functions can be written
4

---

Published as a conference paper at ICLR 2025
as linear combinations of of basis functions. This is common practice, and the literature has investigated several choices of basis functions, such as B-splines. wavelets, radial basis, Fourier
series, finite basis, etc.; see, e.g., Liu et al. (2024c); |Sydi (2024b); Li (022); Xu et al. (2024a);
Howard et al. (2024). The proof of the following result relies on Maurey's sparsification lemma stated in the appendix; also Lemma of (2002).
Proposition 2. Suppose (x) (x)(x), (m)))) with = Ei= Bigit (x) for
m. Here {gij i = ..2...., m:j = 1.2..... .P) is a set of basis functions defined on Rd. Let B = (Bty) E TRa and G(X) = (xx)) E IRmxxxn Define |
(2 Σ Big )
and
8/0 1/s
|G
Σ M (xx))l
for 0 < 20S2 and r, 8,83 > with 1/r+1/8=1 We assume that [B[|, S bm.p.n and [[G(() (XX)))l,, S Cm.p.ns where bmp.p. Cm..pn are both allowed to grow with m. p and n. Then, we have
62
pm
2n Smpp log(2mp))
log (X) T : : F
S
62
With the above results, following assumptions.
provide bound the covering number of Hy (X). We make the
Assumption 1. Assume that |X||2 S D for some constant D > O.
Assumption 2. Suppose v belongs to the following function space
Fi V -V1, (VI,...,
(-) (x) (xx)ll2 S pollx [al|x-300 - -x'122
di
(O)) & C₁ [Bll = Σ
B
for some constants pt, Bi, Oi 0. Further, assume that
[g (x) 9 (x')
di xx x112.
(4)
We make of remarks regarding the above assumptions.
Remark 1. Note that the bound the le norm of X is allowed to grow in theorem. Moreover, Assumption [ can be relaxed by requiring the bound to hold with high probability, which can be tails.
justified when the distribution of X has sufficiently light tail, e.g., sub-Guassian or sub-exponential
Remark 2. The form of V in Assumption 2 indeed more general than the additive structure in (1)
11. as 9ig is multivariate function. Consider T (V1,..., Va.) where
hr
(x)
Σ (xg))
(in Σ Begnggh(ag))
(5)
and Uj.k is set of univariate basis functions. This can be viewed special form of the activation function specified in Assumption where the set of basis functions g is given by (0j. 1 j < 1 < k < b.).
Remark 3. When (1) b is set of univariate B-spline basis functions with degree and the knots
(g(0) we have
1j < 2p/ A ! where A (1) = mmax (4!.)) <(9
Remark 4. Let B (( 0 < < d, j < pr3l be the number of non-zero
coefficients in B, Suppose maxij
|/011 < Bmax. Then have |B:]]] Bmax Bıllo-
5

---

Published as a conference paper at ICLR 2025
Remark 5. The tightest Lipschitz constant for L is given by
L
(x)
1 (x')l2
O ssu xxx x
x' 2|2
When (x) Ax+ b is linear, p is simply the spectral norm of the matrix A (denoted by ||A||
write (x) Ag(x), where A E R4 (d.-16) with the (i. (i-1)by+ element of A being B..jkk. and g(x) E R24116 with the ((j 1)6 +h element of
When & takes the form in (
g(xx being (1,{( We can deduce that p I | E of if [1), 9j,k (x')) I as |r |r/- (1) Moreover, under Condition (4) p S |l Dū, where we define ei = maxi,j Gij
Let C max; Ct. Further, define à Σ
05 with
2/3
2/3
L
0: = B,/ 2/3 G
I -=1+1
Pj
c
Pk +
Pke
j=0 k=i-j+1
The value of à indicates how complex the network is. It influenced by the I norm of the coefficient matrices (i.e., B2) and is also affected by the product of the Lipsehitz constants (pt) for the activation
functions at each layer. Note that for MLPs, the corresponding Lipschitz constants are equal to the spectral norms of the weight matrices. When C' (i.e., C₁ for all S S L), a, (Bre, 1 and
2/3
I. D
t
ā
==
II j=1
Pi
Σ i=1
(Bic,)2/3.
Theorem 1. Under Assumptions
12
ās
log (2dp)
log lHg 6 (12)
2
where d max; di and max, P..
is worth noting that the above bound has no dependence on combinatorial parameters such as the number of nodes and the number of basis functions for each activation function outside of the logarithmic factor. Built upon the above results, we now derive a bound on the generalization error R(f). We impose the following assumption the loss function.
Assumption 3. Given v E y, L(: v) is Lipschitz in the sense that [L(u, e) L(wl v) S B(v)llu- u' for any u, u' E y and B(+) y R2 Further suppose C(f(-),) E [0, M) any f E M. Theorem 2. Under Assumptions 7 we have with probability greater than 1
144/[(log(nM/ v1
B(J) <
L( ((((x.), v.)
7
4M2 log(2/e)
32M log(2/e)
n
3n
for any f E M, where Ç= = a log (2dp) max, (T,)
In Section A.I we compare this bound with a corresponding bound for MLPs in Bartlett et al. (017))
Assumption 3 requires the loss to be bounded, which is satisfied by the ramp loss in multiclass classification. However, this excludes many other unbounded loss functions. By using truncation argument, we can relax the boundedness condition on in Assumption 3 This relaxation would allow the result below to cover a general class of nonparametric regression-type loss functions, including squared loss, pinball loss, and Huber loss.
y, L(., v) is Lipschitz in the sense that [L(u, v) L('', v) < B(v) |u u' 2'112 for any u. u' E y and B(.) y - Ryo Further suppose supjeM ((f(),,) S G( and E[G (x. v) < C' < 00 for (x, y) and some 1, and E[ (y:)) C" <00 s' >
Assumption 4. Given

---

Published as a conference paper at ICLR 2025
Theorem 3. Under Assumptions 72 and 4
have with probability greater than 7 7
1
(n (3v V1)
R(f) (fx)), v1) +
n
1=1
2 12)(
32 log (2/c)
2C"
n1 1)/(2e}
3n 12/
T/ Tm (8-1)/(22)
for 6, T,, () > and any f EM where Co = ãs log(2d)) (nC" (nC" /T)2//
As an implication of the above theorem, consider the empirical risk minimization problem (2). where M (f(-)) L 1, L-
( M E Fi, 1,2,.... ...., L} .} with Fi specified in Assumption 2 Due to the uniformity over the class M, the risk bound in Theorem 3 holdd for any
solution to Problem (2) resulting from optimization scheme (assuming that the resulting solution
still belongs to MV. In Section A.2 we derive an improved high probability bound when G(x. y)
and (y) both have sub-exponential tails.
Let f = Jrg (x), y) for y) ~ P and R((") be the corresponding generalization error. As byproduct of Theorem
obtain a bound on the excess risk.
Corollary 1. Let f E .M satisfy that Σi= (x;) yi
I Ei= (x,) y:). Suppose As-
sumptions
72 and 4 hold, where 2 in Assumption 4
We have with probability greater than
1 21/
R() - R(S*) 144/Goflogge (2+11//(2))
(ontt///e (33 (3vCo) v13
2(C")a/> log(2/c)
(1 + 112
1/2
n
n
(n 1))((2s) + 20"
32 30g(2//E)
1)/2-1)/(22))
for T, O, where So ã looo
2T)2/s'.
The condition Σ ((f(x)), vs) S Ei= [( (x,), y:) only requires f to have an empirical loss smaller than that induced by f" and does necessitate that f be the global minimizer of Problem 2
2.3
RISK ANALYSIS: ACTIVATION FUNCTIONS IN LOW-RANK SOBOLEV SPACE
This subsection studies the case where the activation function 1i belongs to an RKHS with a low- rank structure. Note that when V (x) Ax is an affine map, the low-rank assumption translates into a low-rank condition on the matrix A.
To proceed, we let (22) be an RKHS on Ω S Rd associated with reproducing kernel K. Denote by Illl, the corresponding RKHS norm. We focus the scenario where K. is the isotropic Matérn kernel function, i.e.,
K(X) F(p - d/2)2 d/2-1
||||
|x|2 1 10 4/2 K
after a proper reparametrization, where Kp-d/2 is the modified Bessel function of the second kind, and || 1l22 denotes the Euclidean norm. The parameter > d/2 controls the smoothness of functions (2004), (12). Corollary 10.13 Wendland (2004), and the extension theorem in (DeVore & Sharpley 1993) imply that the corresponding RKHS coincides with Sobolev space with smoothness To model vector-valued functions, we consider the Cartesian product of (Nx, denoted by A (9). given by
N ( (S2) 1 = (Vi...., vm) th E NNc( < < m)
Let (pan(e).... w'm) be the linear space spanned by (m and dim dim (spann( (V1...., vm)) be its dimension. For (Vi,..., 4m) E ENem write | vom = ΣM" 11 11 Define
A, { (iiy...,,m)) E A m dim (s (.,)).,,,)) 5 sr}

---

Published as a conference paper at ICLR 2025
and A,( V Di 2ي (x²).
= A,
we I R). Recall that (X) (x1),, L (x,))) and
We state the following bound on the metric entropy of (X) Ruxm & A,(R) whose proof relies on the equivalence between the RKHS and the Sobolev space.
Proposition 3 (Proposition A.3 of Wang & Zhou (020 Under the above setups, we have
log.N (X)
E Ruxm
V E (,)) e, (..1-112))
4/v
ÜRVTn
ČRVrn
Smr log
1
+r
where is a positive constant.
Assumption 5. Suppose v belongs to the following function space
Fi = v = (vi,..., vd)) E Ar, (Rt) (x) (x))ll S pilx - x'1123. 11lx-xx'122
where Pt, r2 and Rs are some positive constants.
Proposition 4. Set b, CRiyran and b = El=ibe by. Under Assumption
S we have
[bI1,=0+1 Pj
(di-1/w\vi
log. NggN(HL((X)), 112) Edsri
drr
1=1
Theorem 4. Define
(d--1/u)V1
L ders max B(ya)b 1=1
{ = Σ
II Pj j=++1
where b= Et by with by = /ri. Suppose = max, de > u. Under Assumptions and have with probability greater than 6
6C'(E)
R()) ((((x))V))
n/ 1/ 1dof
1=1
4M2 log(2/c)
32M (2/(2/))
3n
for any .M and some constant C" 0.
Theorem generalizes the results for low-rank kernel ridge regression to multi-layer network structure induced by KANs with activation functions belonging to an RKHS. The generalization bound here scales polynomially with the underlying ranks as well as the Lipschitz constants of activation functions in each layer. Also, it has no explicit dependence on combinatorial parameters.
Remark 6. Given set of (fixed) activation functions d; function class
K. for 1 S S L, we define the
1
L V E A,, (Ri), (x) (x))) S pilIx x' x'112
Then, the conclusion in Theorem remains true when the activation function in layer belongs to Fr. Suppose 4 is an activation obtained from pre-training, and we perform a fine-tuning to find the activation functions for some new downstream tasks. Then, we require the update after the fine-tuning process to lie on low-rank space Ar, (Rt). This kind of strategy has been shown to be effective for fine-tuning large language models; see, e.g., (Hu et al. 202T).
To conclude this section, we present the following results that are parallel to Theorem and Corol- lary [
Let
(d;
1/2)V1
L. Eo
nC
2/5
1.
Σ dar
b
Pj
(6)
i=1
j=++1
where b= DL by with b, ČR,
8

---

Published as a conference paper at ICLR 2025
Theorem 5. Under Assumptions and]
we have with probability greater than 1 -
(Eo)
R(f) -
(x((x))
+
mlv//+1//2(d/v/
1)1//d
i=1
2 V log(2// (2/e)
l2
2C"
n
1)/)2>)
3m 1)/(2s)
mpnl-11//6
for any E M and E, ,,
Corollary 2. Let f M satisfy that Ex.1 ((;)) y1 S Ei (x;), 9:). Suppose As-
sumptions 72 and 4 hold, where 2 2 in Assumption
4
We have with probability greater than
1-6-r 21,
6C" (E)
2(Cr)2// logg///oo[(2 log(2/e) + +1++
R(A R(F*) 5 ml//d+1)/2 (d/2 1)
n
32 (2/c)
20'
+
3ns 1)/(22) pm(0-1)/(22)
for 6, T, >> O, where Eo is defined in 16)
Remark 7. Combining the arguments from Theorems and | we can derive a generalization bound for KANs in the more general case, This applies when the activation functions of certain layers
[nd
belong the class in Assumption [ while the activation functions of other layers have low-rank
structure, e.g., as specified in Remark 6
3 NUMERICAL STUDIES
We used both the simulated and real datasets demonstrate the relationship between the excess loss (defined as the difference between the test loss and training loss) and the complexity of KANs. We constructed the KANs such that the output of each layer at 0 O, i.c., v.(0) O. Therefore, we used (II/=1 P.) 2/)3 E'' (B,c,)2/3, which was proportional to ā, as the measure of complexity, where the Lipschitz constants Pys were estimated by their upper bounds provided in Remark
For the simulation, we consider two examples in Liu et al. (202cc). Specifically, we generated x; from Unif(-1, independently. and let
(21, .4) exp
(in((((a + x3)) sin(T(23 + x311))
100
1
f2(
exp
sin?
100
for the low-dimensional and high-dimensional cases, respectively. We considered the following four setups:
(i) y = /1 (x T2, x4) x exp(c), 6~ N(- log 1.44/2 log((.04))
(ii) y f2(x1........ ..., 2100) exp(c), N log log(1044)
(iii) Pyy 1) 1 fi (1 F2, 23, x4)
P(y 0) 1 - P(y = 1);
(iv) Ply= 1+ f2(11...... 2100)
P(y = 0) ply 1),
where we have chosen the distribution of e such that the mean and standard deviation of exp(e) are equal to and 0.2, respectively. We set the sample sizes of both the training set and test set to be 10,000 for all four datasets. The shape of KAN used for (i) and (iii) was [4, 50, 100, 50, 1], and was [100, 50, 100, 50, 1] for (ii) and (iv). We refer the readers to Section 2.2 of Li et al. (2024c) for the definition of the shape of a KAN that is represented by an integer array.
We also investigated the MNIST and CIFAR-10 10 datasets. We used the features extracted from pre-trained AlexNet model as input for the KANs. The extracted features were 1000-dimensional for both datasets. We employed the KAN with the shape [1000, 50, 100, 50, 10).

---

Published as a conference paper at ICLR 2025
We run 1000 epochs for each dataset. The results shown in Figure where normalize the values of the complexity measures so that the maximum value of the complexity measure is equal to the last value of the excess loss (see Section A.4 for more details). The plots in Figure SigureZillustrate that our proposed measure of complexity for the KAN networks, o, tightly correlates with the excess loss in all the cases. It is surprising observe that the complexity measure curve closely follows the shape of the excess loss. These results highlight the practical relevance of the generalization bounds derived in Section 2 We refer the readers the appendix for additional numerical results and discussions.
(ii)
Text Less Train Loss
0.30
KAN Complexity
0.25
0.20
0.15
0.10
1.0
0.10
0.05
0.5
0.00
0.00
0.0
400
(iv)
mnist
cifar10
1.0
2.5
2.0
0.8
1.0
1.0
0.2
0.5
200
400
600
800
1000
200
400
600
800
1000
200
400
600
800
1000
Epoch
Figure 2: The excess loss and (normalized) complexity of KANs trained with SGD on the four simulated datasets (i-iv) and the MNIST and CIFARIO datasets. The loss for (iii), (iv), MNIST, and CIFAR1O is the cross-entropy loss, and that for and (ii) is the mean squared error.
4 DISCUSSIONS
We derive generalization bounds for KANs when the activation functions are either represented by linear combinations of basis functions or lying in a low-rank RKHS. These bounds are empirically investigated for the KAN networks trained with SGD simulated and real data sets. The numerical results indicate a strong correlation between the excess loss and the complexity measure, demon- strating the practical relevance of the generalization bounds.
To conclude, we mention few future research directions. (i) Additional empirical studies and the- oretical investigation are needed to understand the superiority of KANs over MLPs; (ii) would be interesting to derive a lower bound and see how well it matches the upper bound to understand the tightness of the derived bounds. Also, additional work is needed to determine which norms are well-adapted to KANS used in practice; (iii) A more careful analysis is needed to show that SGD applied to KAN results in a well-behaved predictor and leads to a refined generalization bound: (iv) Theorem may be improved with relaxed restriction the Lipschitz constants of the basis func-
tions {9, ): (v) was observed that explicit regularization contributes little to the generalization performance of neural networks (Zhang et al. 2021). It is of interest to see if similar conclu- sion holds for KANs and whether other types of regularization help to improve the generalization performance. Our theoretical and empirical results suggest close connection between the derived complexity measure and the generalization error. It is of interest to develop (implicit) regularization techniques control this complexity measure to achieve improved generalization performance.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Diab W Abueidda, Panos Pantidis, and Mostafa E Mobasher. Deepokan: Deep operator network based on kolmogorov arnold networks for mechanics problems. arXiv preprint arXiv:2405. 19143, 2024.
Alireza Afzal Aghaei. fkan: Fractional kolmogonov-aroold networks with trainable jacobi basis functions. arXiv preprint arXiv:2406.07456, 2024a.
Alireza Afzal Aghaei.
rkan:
Rational kolmogorov-armold networks
arXiv preprint
2XXY::2406.14495. 2024b.
Mohammed Ghaith Altarabichi. Dropkan: Regularizing kans by masking post-activations. arXiv preprint 1.XXi:2447.130444 2024.
Martin Anthony, Peter L Bartlett, Peter L Bartlett, et al. Neural network learning: Theoretical foundations, volume 9. cambridge university press Cambridge, 1999.
Basim Azam and Naveed Akhtar. Suitability of kans for computer vision: A preliminary investiga- tion. arXiv preprint 07X9v22406.09087, 2024.
Peter Bartlett. For valid generalization the size of the weights is more important than the size of the network. Advances in neural information processing systems, 9, 1996.
Peter L Bartlett, Olivier Bousquet, and Shahar Mendelson. Local rademacher complexities. 2005.
Peter I. Bartlett, Dylan J Foster, and Matus Telgarsky. Spectrally-normalizeed margin bounds for neural networks. Advances in neural information processing systems, 30, 2017.
Alexander Dylan Bodner, Antonio Santiago Tepsich, Jack Natan Spolski, and Santiago Pourteau. Convolutional kolmogorov-armold networks. arXiv preprint arX1v:2406.13/55, 2024.
Zavareh Bozorgasl and Hao Chen. Wav-kan: Wavelet kolmogorov-amold networks. arXiv preprint arXiv:2405. /283,, 2024.
Roman Bresson, Giannis Nikolentzos, George Panagopoulos, Michail Chatzianastasis, Jun Pang, and Michalis Vazirgiannis. Kagnns: Kolmogorov-arnold networks meet graph learning. arXiv preprint ar 2rXiv:2406.1833 2024.
Yifei Chen, Zhu Zhu, Shenghao Zhu, Linwei Qiu, Binfeng Zou, Fan Jia, Yunpeng Zhu, Chenyan Zhang, Zhaojie Fang, Feiwei Qin, al. Sckansformer: Fine-grained classification of bone marrow cells via kansformer backbone and hierarchical attention mechanisms. arXiv preprint arXiv:2406.0990 09931,, 2024.
Minjong Cheon. Kolmogorov-aroold network for satellite image classification in remote sensing. arXiv preprint arXi::2406.00600 2024.
Gianluca De Carlo, Andrea Mastropictro, and Aris Anagnostopoulos. Kolmogorov-amold graph neural networks. arXiv preprint arXiv:2406.18.0 18354, 2024.
Ronald A DeVore and Robert C Sharpley. Besov spaces on domains in Rd. Transactions of the American Mathematical Society, 335(2))843-864, 1993.
Max H Farrell, Tengyuan Liang and Sanjog Misra. Deep neural networks for estimation and infer- ence. Econometrica, 89(1):81-2113 2021.
Remi Genet and Hugo Inzirillo. A temporal kolmogorov-arnold transformer for time series fore- casting. arXiv preprint arXiv:2406. 02486. 2024.
Luis Fernando Herbozo Contreras, Jiashuo Cui, Leping Yu, Zhaojing Huang, Armin Nikpour, and Omid Kavehei. Kan-eeg: Towards replacing backbone-mlp for effective seizure detection system. medRxiv, pp. 2024-06, 2024.
Yuntian Hou and Di Zhang. A comprehensive survey on kolmogorov arnold networks (kan). arXiv preprint arXiv:2407.1 /075, 2024.
11

---

Published as a conference paper at ICLR 2025
Amanda A Howard, Bruno Jacob, Sarah H Murphy, Alexander Heinlein, and Panos Stinis. Finite basis kolmogorov-amoll networks: domain decomposition for data-driven and mmysics-informer problems. arXiv preprint arXiv:2406. 19662, 2024.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2 2021.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kemnl:: Convergence and gen- eralization in neural networks. Advances in neural information processing systems, 31, 2018.
Mehrdad Kiamari, Mohammad Kiamari, and Bhaskar Krishnamachari. Gkan: Graph kolmogorov- arnold networks. arXiv preprint 0rXiv:2406.06470, 2024.
Akash Kundu, Aritra Sarkar, and Abhishek Sadhu. Kanqas: Kolmogorov arnold network for quan- tum architecture search, arXiv preprint arXiv:2406. 17630, 2024.
Chenxin Li, Xinyu Liu, Wuyang Li, Cheng Wang, Hengyu Liu, and Yixuan Yuan. U-kan makes strong backbone for medical image segmentation and generation. arXiv preprint arXiv:2406.02916 2024a.
Xinhe Li, Zhuoying Feng, Yezeng Chen, Weichen Dai, Zixu He, Yi Zhou, and Shuhong Jiao. Coeff- kans: A paradigm address the electrolyte field with kans. arXiv preprint arXiv:2407.2026,, 2024b.
Ziyao Li. Kolmogorov-arrold networks are radial basis function networks. arXiv:2005.06721, 2024.
arXiv preprint
Hao Liu, Jin Lei, and Zhongzhou Ren. From complexity to clarity: Kolmogorov-aroold networks in nuclear binding energy prediction. arXiv preprint arXiv:2407.2073:: 2024a.
Mengxi Liu, Sizhen Bian, Bo Zhou, and Paul Lukowicz. ikan: Global incremental learning with kan for human activity recognition across heterogeneous datasets. arXiv preprint urXiv:2406.01644 2024b.
Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruehle, James Halverson, Marin Soljačić, Thomas Y Hou, and Max Tegmark. Kan: Kolmogorov-arnold networks. arXiv preprint arXiv:2404. 19756, 2024c.
Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics, 75(4)x667-766, 2022.
George Nehma and Madhur Tiwari. Leveraging kans for enhanced deep koopman operator discov- ery. arXiv preprint arXiv:2406.02877 02875, 2024.
Yanhong Peng, Miao He, Fangchao Hu, Zebing Mao, Xia Huang, and Jun Ding. Predictive modeling of flexible chd pumps using kolmogorov-arnold networks. arXiv preprint arXiv:2405.07488, 2024.
Eleonora Poeta, Flavio Giobergia, Eliana Pastor, Tania Cerquitelli, and Elena Baralis. A bench- marking study of aolmogorov-amoodd networks on tabular data. arXiv preprint arXiv:2406.14555 2024.
Pawel Pratyush, Callen Carrier, Suresh Pokharel, Hamid D Ismail, Meenal Chaudhari, and Dukka B KC. Calmphoskan: Prediction of general phosphorylation sites in proteins via fusion of codon aware embeddings with amino acid aware embeddings and wavelet-based kolmogorov arnold network. bioRxiv, pp. 2024-07, 2024.
Spyros Rigas, Michalis Papachristou, Theofilos Papadopoulos, Fotios Anagnostopoulos, and Geor- gios Alexandridis. Adaptive training of crid-dependent physics-infonne kolmogorov-armol net- works. arXiv preprint arXiv:2407. /7611 2024.
12

---

Published as a conference paper at ICLR 2025
Johannes Schmidt-Hieber. Nonparametric regression using deep neural networks with relu activation function. Annals of Statistics, pp. 1875-1897, 2020.
Seyd Teymoor Seydi. Exploring the potential of polynomial basis functions in kolmogorov-
arnold networks: A comparative study of different groups of polynomials. arXiv preprint arXiv:2406.02588. 2024a.
Seyd Teymoor Seydi. Unveiling the power of wavelets: A wavelet-based kolmogorov-arnok net- work for hyperspectral image classification. arXiv preprint arXiv:2406. 07866, 2024b.
Khemraj Shukla, Juan Diego Toscano, Zhicheng Wang, Zongren Zou, and George Em Karniadakis. A comprehensive and fair comparison between mlp and kan representations for differential equa- tions and operator networks. arXiv preprint orXiv:2406.02917 2024.
Hoang- Thang Ta. Bsrbf-kan: A combination of b-splines and radial basic functions in kolmogorov- arnold networks. arXiv preprint arXiv:2406. 11173, 2024.
Cristian Vaca-Rubio, Luis Blanco, Roberto Pereira, and Marius Caus. Kolmogorov-arnolo net- works (kans) for time series analysis. arXiv preprint 2xxx:2445500790 2024.
Wenjia Wang and Yi-Hui Zhou. Reduced rank multivariate kernel ridge regression. arXiv preprint arXiv:2005.01555 2020.
Yizheng Wang, Jia Sun, Jinshuai Bai, Cosmin Anitescu, Mohammad Sadegh Eshaghi, Xiaoying Zhuang, Timon Rabezuk, and Yinghua Liu. Kolmogorov arnold informed neural network: A physics-informed deep learning framework for solving pdes based on kolmogorov arnold net- works. arXiv preprint arXiv:2406. 11045, 2024.
Holger Wendland. Scattered data approximation, volume 17. Cambridge university press, 2004.
Jinfeng Xu, Zheyu Chen, Jinze Li, Shuo Yang, Wei Wang, Xiping Hu, and Edith C-H Ngai. Fourierkan-gef. Fourier kolmogorov-arnold network-an effective and efficient feature transfor- mation for graph collaborative filtering. arXiv preprint 2xxi:244660000:3 2024a.
Kunpeng Xu, Lifei Chen, and Shengrui Wang. Kolmogorov-ammn armold networks for time series: Bridg- ing predictive power and interpretability, arXiv preprint Stti::2400.02-9999 2024b.
Shangshang Yang. Linrui Qin, and Xiaoshan Yu. Endowing interpretability for neural cognitive diagnosis by efficient kolmogorov-amold networks. arXiv preprini 24t.::9905.44999 2024.
Engin Zeydan, Cristian J Vaca-Rubio, Luis Blanco, Roberto Pereira, Marius Caus, and Abdullah Aydeger. F-kans: Federated kolmogorov- arnold networks. arXiv preprint arXiv:2407.20100, 2024.
Chiyuan Zhang, Samy Bengio. Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107- 115. 2021.
Tong Zhang. Covering number bounds of certain regularized linear function classes. Journal of Machine Learning Research, 2(Mar):527-550, 2002.
Shrenik Zinage, Sudeepta Mondal, and Soumalya Sarkar. Dkl-kan: Scalable deep kernel learning using kolmogorov-arnot networks. arXiv preprint 17.,, 2024.
13

---

Published as a conference paper at ICLR 2025
A
APPENDIX
A.1 COMPARISON WITH MLPs
We compare the derived bounds with those for MIPs,, specifically focusing on the bounds discussed in Bartlett et al. (2017). which address the classification problem with ramp loss. The scenario
described in that work closely aligns with the setting of Theorem 2 particularly with C 0 and
max; (U)) being finite. Ignoring smaller order terms, the bound for R(f) 12 Ei=1 ((f(x)),
in Theorem 2 can be expressed as follows
X |RKN log(n) log(2d)) + log(1/c)
n
where
3/2
I. II Ps j=1
1 Σ
RKAN =
(B,c;
measures the KAN complexity. Bartlett et al. (2017). is given by
The corresponding bound for MLPs, derived in Theorem 1.1 of
(
| | log(n) log(W) log(1 (/)) nnn
72
where RA measures model complexity (similar RKAN as defined above), y represents the margin in the ramp loss, and W denotes the network width (analogous d in our case).
When R and Ra are of the same order (denoted by R = RAA). both bounds are also of the same order, up some logarithmic factors. This is expected since R RA indicates that both classes exhibit similar levels of complexity. Conversely, if R = of RAR and RA/ BA/Vn + +00 then the bound for KANs of smaller order than that for MI.Ps. This scenario arises when the true underlying regression function can be more accurately approximated using KANs. For instance, when the true regression function can be exactly represented by KAN network, the complexity required for approximating such a function using MLPs is expected to be higher. As discussed in Liu ctt al. (2024c), KANs may require fewer parameters than MLPs to approximate the same underl ying function, as KANs utilize the inherently www-dimessional representation of the true function, unlike MLPs. We refer the readers to Liu et al. (2024c) for examples where KANs can outperform MLPs with the same model complexity, e.g., Figures 3.1-3.3 therein.
A.2 IMPROVED BOUNDS UNDER SUB-EXPONENTIAL TAILS
We derive an improved high probability bound for the generalization errors under the condition that G(x, y) and (yy) both have sub-exponential tails. We impose the following assumption.
Assumption 6. Given v E y. L(., v) is Lipschitz in the sense that [C(u, v) -L(,, (i/.v)) .v) < B(v)
112 for any u, u' y and B(-) : Rzo- Further suppose supjem [C(((),) G( Both G(x, y) and Bly) have sub-exponential tails, i.e., Te[exp (xGep)))Gxx,)))) u)) S ( and Elexx(Ax (z)) & exp(>2o0z) 2///)/) for Xo, AB in neighborhood of 0.
Theorem 6. Under Assumptions 72 and o we have with probability greater than c T 7
R(F) C(f(x:). v.) + 144//5(log(('nn longl /
log(()) (2/c(2/c
C' log(n) log(2/e) + C'V log(1/1)
+
Vn
3n
for for > 0 and any f E M. where Co = à log(2dū)) loggg"""" (T))² and O' a large enough constant.
Note that the dependency 1/1 in the original bound has been improved to Vlog(1/1) by using Bernstein's inequality. Additionally, the term n (1=s)/(2s) has been refined to n -1/2 due to the exponential tails. In similar fashion, we have the following result, which strengthens Corollary [
14

---

Published as a conference paper at ICLR 2025
Corollary 3. Let f M satisfy that Σi= (x)()))) 2 E (x,*(x,), yı). Suppose Assump- tions] 7 and hoold. We have with probability greater than - 211.
R(f) R()) 141/5 /50 v1
C' Co (n)
(n) /log(12////
n
C" (o)g(()) log(2/e)
+ CVV log(1/n)
3n
for c, T, O, where So = â log log(2dT) (CllolnC"" /T))² and C' is a large enough constant.
We remark that the bounds in Theorem 5 and Corollary 2 can also be enhanced under the condition of sub-exponential tails for G(x, y) and Byy).
A.3 A LOWER BOUND ON THE EMPIRICAL RADEMACHER COMPLEXITY
We derive a lower bound on the empirical Rademacher complexity defined in for the net- work class specified in Assumption 2 From the second layer, suppose the basis functions in- clude the projection onto the first coordinate, i.., there exists a linear combination of the basis
such that 2; { 9ig (x) = Tj for x ..,
(1)
For the first layer, we set uy(2) (x) =
E 3 9ij (x) and 4 (x) = 0 for all 2 2. Let vx((x) (x) = 21 and Let w0 (x) = for 12 2 and S 1 - 1. The activation function for the output layer a real-valued function that maps the input to its first coordinate. Note that the condition | (x) -of" I p||x x' ||2 implied by Σド= | S p\/ maxx <i as 191y (x) (x')
(3)
(x)) S d) ||x - x' 2|2- Therefore,
L o (x) Σ 3 91z (x) and the Rademacher complexity of the network class is lower bounded by that the class
1/2
P1
P1
8
(x(x)
1 9is
(x):
(x):
Σ
s B
5=1
for B1 min( p/ max, C }/ /pi. Then we have
(1)
(X)) = E
1
Σ DE 1=1
sup
exb((x;)
x
E
sup
B302 (x:
(1)
X
B:1|,8|| S
1=1
B E
C'
( (x:))
c
(x:)
X
for some constant C' 0, where we have used the Khintchine inequality get the last inequality.
A.4 ADDITIONAL NUMERICAL RESULTS
In our experiments, the activation functions in the implemented KAN are linear combinations of SiLU function and basis splines, as proposed by Liu et al. (2024c). Specifically, the (I,) for i l,..., L, I...., d L,.... d:1 in
D IS
Vin (z) w k b(x)
(x) b(x) + w spline(x),
where
b(x:) = silu(x)) = 2 + + spline(x) = Σ C.B((z))
and (x) spline basis. See equations (2.10)-(2.12) of Liu et al. (202cc).
15

---

Published as a conference paper at ICLR 2025
display the curves of the KAN complexity as measured by T Pg) 22/3 (B,e,) and the excess loss the same plot. Since the values of these two terms complexity measure is equal to the last value of the excess loss. Specifically, let u (wi....,, uN) be the values of the differences between the test losses and training losses, where N is the number of training epochs, and = (Ui...., UN) ) be the model complexity corresponding to different training epoch. Let Umax max(U1. vI,..., UN and Umin min (i.............................. ........ UN The normalized complexity v' (u,,..., U'x) with
In Figure 2
different scales, we normalize the complexity that the maximum value of the
(e
Umin
JUN
0/
Umax
Umin
We present the curves of the KAN complexity, test loss, and training loss in Figure
The aim of
Loss Train Loss
0.30
0.20
0.25
KAN Comptexity
1.5
0.15
1.0
0.10
0.05
0.05
0.5
0.00
0.00
0.0
200
400
600
800
1000
(iv)
mnist
ci
3.0
1.0-
2.5
0.6
1.0
0.4
1.0
0.5
0.2
0.5
200
4000
600
BD0
1000
0
260
460
860
800
1000
200
400
600
800
1060
Epoch
Figure 3: The training loss, test loss and complexity (normalized with respect to test loss) of KANs trained with SGD on the four simulated datasets (i-iv) and the MNIST and CIFARIO datasets. The loss for (iii), (iv), MNIST, and CIFAR10 is the cross-entropy loss, and that for (i) and (ii) is the mean squared error.
our simulation study is to demonstrate the practical relevance of the generalization bound derived hope to understand the relationship between model complexity,
in Section 2 Specifically, cess risk, and test loss. Our empirical results indicate a strong correlation between excess loss/test loss and model complexity. This finding motivates the need develop regularization techniques, whether explicit or implicit (such as early stopping or penalization), to control this complexity mea- sure, which can potentially enhance generalization performance.
We use the Setups (i) and (iii) for a simple demonstration, where the underlying regression and classification models are both based the synthetic function Si defined in Section 3 We apply the dropout technique with a rate of 0. to the activation functions in KAN networks (referred to as regularized KAN) with the same shapes as we have used in Section 3 (i.c., [4, 50, 100. 50, for both setups). The results are shown in the top two panels of Figure where we also plot the results for non-regularized KANs, i.e., those results in Figure 2 for comparison. The values of excess loss are depicted in their original scales, while the values of the KAN complexity are normalized with respect to their excess losses.
observe that with the dropout technique, the excess losses decrease significantly from 0.33 to 0.03 for Setup (i) and from 2.41 to 0.06 for Setup (iii). For both the regularized KAN and non regularied KAN, we observe that the model complexity tightly correlates with the excess loss. As the two complexity curves plotted with different normalization scales, they cannot be compared directly. Therefore, we plot the ratios of the complexities of the regularized KAN the non-regularized KAN in the bottom panels of Figure 4 We can that
16

---

Published as a conference paper at ICLR 2025
the regularized KAN, which induces lower excess loss than the non-regularized KAN, also has lower complexity.
(iii)
2.5
Train Loss
Test Loss
0.30
KAN Complexity
Train Loss Loss (Drepout)
Complexity (Dropout)
0.20
1.0
0.10
0.5
0.00
0.0
200
400
600
800
1000
200
400
600
800
1000
(i)
(Iil)
Complexity
0.9
0.8
0.8
0.7
0.6
0.6
0.4
0.5
0.4
200
400
600
800
1000
200
400
600
800
1000
Epoch
Figure 4: (Top) The excess loss and (normalized) complexity of KANs trained with SGD the simulated datasets (i) and (iii). The loss for (i) the mean squared error, and that for (iii) is the cross-entropy loss. The green and red curves in the top-right panel overlap. (Bottom) The ratio of
the complexity of the regularized KAN to the non non-regularized KAN.
A.5 AUXILIARY LEMMAS
The following lemma is useful in the proof of Proposition 2 see Lemma 1 of Zhang (2002) and Lemma of Bartlett et al. (2117).
Lemma (Maurey's sparsification lemma). For Hilbert space H with the norm || let U H be given with the representation U E a,V, where Vi H. and a O. Then, for any positive integer k there exists choice of nonnegative integers "ar....,,k) such that Eim K, = and
|a/|
|a/l
||a||p max |[V1
U
V
b
b
where a = (a1,-..,au) aN) and ||l|]]
E O1-
The proof of Theorem makes use of the following two lemmas. The first lemma shows that R(f) can be related to the empirical Rademacher complexity of the neural network function class. The second lemma connects the empirical Rademacher complexity with the corresponding covering number through the integral entropy.
Lemma 2 (Theorem 2.1 in Bartlett et al. (2005)). Let be class of functions that maps X' into
M,, M for some M > 0. Assume that there is some > 0 such that for every f E F, (x)) s
r. Then for every > O, with probability at least c over the data X [xi...., x., we have
1 Σ ((x))
2r log(2/c)
32M log(2/)
sup JEF
E [[f(x)]
(R(F(X)) +
+
3n
i=1
17

---

Published as a conference paper at ICLR 2025
where F(X) {(f(x1), f(xx)) F} and R(F(X)) is the empirical Rademacher com- plexity i.e.,
1 E n
R(F(X))
sup ed (x:)
with 6, being a set of independent Rademacher random variables, i.e., P(e; =1) = 1/2. Lemma (Lemma A.5 in Bartlett et al. (2017). Let be real-valued function class taking values in [0, M] and assume that 0 E F. Then we have
4a
Myn
R(F(X)) S inf a>0
Vlog,N(F(X),,. e, -
A.6 PROOFS OF THE MAIN RESULTS
Proof of Proposition We divide the proof into three steps.
Step 1: Choose an cover N₁ of (X) V E Fij. Then we have
[1 S ((
E e.)) N1.
Step 2: For every & E N, we construct G++1 cover G (b) of (v((): T E Fi11). Since the covers are proper, we have $ = Vi o - o...o (o(X) for some x Fi and Gi+1 (+) Hi+1 (X). Thus, we must have
E F; Fi-1
1 S sup
W( D 1...... (X( & D VEF+))) 1441) = N/+1-
1........ M
Finally, we form the cover
N1+1 = UpEN; LieN
which satisfies that
H+1
Wi+1 WWG S
II N₂.
j=1
Step 3: By the definition of 8; we have StP(+1 + 6s+1 81+1. induction. By construction, we can find f1 E Wi such that
We shall prove equation 3 using
(X) 1 L (x(x <1 81.
Now suppose we can find fo E M2 such that
(xx(X) v oX S si.
By the construction of N1 given f. E N; and Vi+1 E Fit1. we can find fitz E N++1 such that
( v +1 f(())4+1 S 61-1.
Then, by the triangle inequality, we have
(xx) L 41 o...... L X i+1
515+1(X) Mi+i f((X) 1+1 1it o f((X) 14+1 o o.... Po(X)]i41
26+1 + Pit1 ( L o .... L (X)4441
20+1 + P 8;+1-
The conclusion thus follows.
Proof Proposition
2 Let (X)
Ruxm
where its ith column equals
(9is (x1)/bs.. Ity /bbg) with bis are equal to zero. Further, define the set
loyxx and all the other elements
18

---

Published as a conference paper at ICLR 2025
where N 2mp. We view Vi's elements in the Hilbert space "= equipped with the inner
product (A, A) trace(A A A) and norm |All2 for A, A E Roxm We have
N
V (X) =
Teyb (X)
(X)
a.V.
for og 2 0 and Tij = (.g)(
Note that
k= Mii (xx) 12)1/22
max IS:SN
Vill2
max Isism,1S/S
km (x4s (x+) 11)1// S
and
N Σ 1=1
|aa|]
Jai
biy (|(()) S Dm.p.reC.p...
where 1/8+1/r- and 8, O. Therefore, (X) E (lm.ppnCC.p.. Σ ar at 2 0, a }} (this is because the convex hull of (Vi,..., VN) always contains the origin). Now we define
N
N
Cm.p.
C2
K& k; 2
0,
k;
K
k
k
Dmm..nCm.p..
Vi, (f1,.... .g) [N]
By construction, |C&| S & Using Lemma ΣN K = such that
there exists nonnegative integers (ki,..., kN) with
N (X) Dmm.pn
12
62 m.p.n m.p.nnm.p.n m.pm
K.V.
m.p.n m.p.p.
max I5
[
k
By setting
e²
b
2 pn p.p k. Cj forms an cover of (X( E Rexm L E F} and the result
follows.
Proof of Theorem 7 Given 1 1...., V1-1) F. we define (1- 1,.... T 1-1)
[x(1-1) (1-1)
where x (1-1) 1 1 ....... ................................ vo(x for s S n. Then, we have
1/2
| ( L - G (o) = max
Σ
Dey (x} (1-1)) (0)
i,j
for G (X) (0.0] (xx)) (xx) E ddxpxx We note that
|G
1--1) - G (0)
<q <q 1/
(X)))
Sq |
())
(0) (0))))
<a
(pil)
1-1
(X)ll + (0)|l))
Sa
(0)l2
II
Pi X l I P
1-0
C 3
<a
PE D
Pi
19

---

Published as a conference paper at ICLR 2025
By Proposition
we have
log (12)
sup V
log [ I v. v. vo(x): : v F4,1,e Il2)
sup
log o o oolx)) v E Fit1h, 112)
,,..,
Bell (EE -41 Ps + DIT 2
log (2d.p.)
where (.) (-) (0). For any > 0, set
E a,e
T1=++10 Pj
Then, have 81, Ei=r (1)= c and hence
(H))) l12) 5 8 log (2dp)
Proof of Theorem
Define the class Mc(X) (I) (([(x)))))) E M).
By Assumption
3
we have
â log (gg() max, B²
log (l (12)
?
Lemma implies that
R(Mc(X)) < inf aso
4a
12 vvM
V 2
de
VTi
4a
12V5
inf
+
log( M VTn/a)
a>0
Vn.
12v7
12 (2V(lognMM//3/
n
24 (3VC(log(mM//333VVV
"
Using Lemma 2 with 2M2, have with probability greater than 1-c,
R(f)= -
C(f(x.). +
1
4M2 log(2/e)
32M log(2/e)
3n
for any f M.
Proof of Theorem Define the class Mc. (X) = (()( vi) A. C(f(xn), vn) AM):
E M). Following the arguments in the proof of Theorem 2 Mc()) replaced by Mc.M(X). show that with probability greater than 1
14AC(1og(uMM/33C)) v1)
E[C(((x) A M (((())) A
2 l2g(2/c)
32Mlog(2/6
3n
20

---

Published as a conference paper at ICLR 2025
We first note that
P(max (y:+ > E) S P(B()) > v s nO"
12
Setting E = (nCC (nCC" /T)2//, we have Ç $ Co with probability greater than 1 - T. Next, under Assump-
tion 4 we have
[R(f) - E(C(f(x), ^ MI <E((C(((x), (M)1(()))))) MH
C"
1BGG(x,))1GG/S MH
M
Moreover,
where Z4 [((())) vi) - - AM
1
<
Zi
i=1
1=1
M).As
1
S C'
Zz
=M-=11
n1
the conclusion follows by setting C² C²/(n and M = n11(22))
Remark 8. When 2 2, we note that var(C(f(x),)) S E|G (x, v)l < (B[G"xx x))) S 2//lg(2/e) (C")2/s Thus by Lemma 2wit r (C")a/s, the term 17/21 in Theorem can be replaced by 2(00)2 2(Crg//1og((//)) leading to the risk-bound
R() s- n T
((((x:) + 14 (3/Co)) V1)
n
2(C')2/ log(2/e)
32 log(2/c)
20'
+ (m((-11)((2))
(2 1)/(28)
n
Proof of Corollary
7 By Remark
8
and the definition of J. we have with probability greater than
1 -- 77
1 n R() S n
(3vCo) V13
Σ (x.(()))
n
2(C')2/0 log(2/e)
32 log(e/e)
20
+
3n (2=-11))
(m(l-1)/((2))
n
Note that
>
var(L(ff(x))
EIG (x. 8)]
(C')2/a
>
<
ne2
ne²
ne²
The conclusion follows by setting 8 ((C')//e(mm))
Proof of Proposition
The results follow from Propositions T and
3
In particular, by Proposition
we have
log ((H)(())... SL. 112)
sup v
log. N V V (x( v E Futhe44ill 11))
21

---

Published as a conference paper at ICLR 2025
where Fi+1 is defined in Assumption 5 By Proposition 3
log v 1-1 1-10..
(
v A (Ret))), G41, l12)
d//o
Sd4+11+41 log 1 CRI+IVTI+1n
CRit1yTi4z Ti41
+ +1i1
c++1
Gi-1
uniformly over all L 1...... v For any 0, set
bic
|= Pj
Then, we have 8T. = Ei=l (II = and for small enough
b 0
611
d--//
drr; log
1+
+
(di-1/ojv1
dr
where we have used the fact that log(1 + S x for x 2 O.
Proof of Theorem Recall Mc(X) {([(f(x)): 91).......................). ((f(()), f E M}. By As-
sumption 3 we have
(di-1/o)VI
max; Bly, 11 Pj
S E
log ((M (12) _
dor:
drr;
A/vv
1=1
Lemma implies that
4a
12
VTM
E de
R(Mc(X)) S inf as0
Vn
n
cd/v
12g 4a
a dddiv (VnM))
inf a>0
VT.
n
d/v-1
Č'(g)và
(ddvvv-1)1
for some constant C' 0. Using Lemma
with = 2M2, we have with probability greater than
(C)(E)
R(f) <
((f(xx))
1/ 1)
1m
4M2 (2/c)
32M log(2/e)
+
3n
for any M.
Proof of Theorem 5 Define the class Mc. (X) = (x)( v A. c(f(x,), vn) AMM E M). Following the arguments in the proof of Theorem 4 Mc(X) replaced by (X) we have with probability greater than 1 --6,
() A 1 ((f(x)) 1.) AM
6θ'(E)
n n(v/d+1)/2(d/pp 1 1
4M log(2/e)
32M log(2/e)
3n
The rest of the proof is similar those in the proof of Theorem and omit the details.
22

---

Published as a conference paper at ICLR 2025
Proof of Corollary
The proof is similar to the one for Corollary We omit the details.
Proof of Theorem 6 We only highlight the steps where the arguments differ from those for Theorem
3 We let C" be generic constant that can differ from place place. Note that for some small enough À > 0,
P(max (y:) > 8) S
Σ P(B(U)) > V8) S nC' exp(-Ave).
Setting {log(nC') /T)/ have < - Co = à log (log((C) /r) with probability greater than т. Next, under Assumption 6 we have for some small enough > and large enough M,
IR() - B[((f(x) v) MII <E((((((),,) - ()) MH
1) M)]
<C" exp--AM).
Moreover,
1 Σ Z, ((f(x)) 1 Σ [(f(x)). A M <
AM
i=1
i=1
where Z, = G(x,, Mt (G(x,, v.) > MJ. Lez pzz E[Zz]. By Bernstein's inequality, we have for
E - pz > 0 sufficiently small,
P
Σ 71
=P
Σ (Zg --zz) >E E
1z
n
{ f1o-112 (e - -1z)2)
(e-1z)²2)
}):
S cxp
-onmin
C
C
where c is a positive constant. Set M = llg((n) for some large enough k. We have μzz = 12/2 + with, we obtain
O(n-AK - o(n Letting e = C²
2 > 5n
n2 t=1
Thus, the result follows.
Proof of Corollary 3 1-E-7-M 7
By Theorem 6 and the definition of we have with probability greater than
R(F S ((f"(x:), +
C'log(n) (log(//e)
C" (og())) log(2/e) + C²
log(1/1)
+
Vn
3n
By Bernstein's inequality, we have
n Σ
(() R(f*) >5)
(-onmin
E²
s exp Sexp
C²2
n
The conclusion follows by setting C' C'flog(1)) c'(log(1nn/((nn))
23


Reviews:
[{"review_id": "ZfIYNC9WEs", "reviewer": "Reviewer_vaj9", "summary": "This work addresses a gap in the current research on Kolmogorov–Arnold Networks (KANs) by (i) quantifying the complexity of KANs, and (ii) establishing high probability upper bounds of excess risk in regression task. Specifically, this work studies two special classes of KAN: (i) KAN with activation functions represented by basis functions, and (ii) KAN with activation functions on the RKHS induced by Matern kernel.", "questions": "See Weaknesses", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- I found the paper well-written, has a good flow, and I believe it presents very solid theoretical results.\n- The theoretical results are novel as they present the first generalization error bound and complexity estimate of KAN.\n- Frankly, KAN is an \"interesting\" yet \"contentious\" topic in the machine learning community ---   \"interesting\" in the sense that it has caught a lot of attentions and discussions in the community; \"contentious\" in the sense that it hasn’t achieved broad acceptance or visibility, especially since all the cited KAN-related papers (including the very first KAN paper (Liu et al. 2024)) have not been published in major machine learning venues. I believe this paper contribute to the theoretical understanding of KAN by estimating the excess risk in regression setting of two special classes of KAN. Although the current results are insufficient to conclude whether KAN achieves better generalization performance than traditional MLPs, they provide valuable insights that can encourage the community to further investigate the potential advantages and impact of KAN.", "weaknesses": "-  I find the purpose of the numerical experiments in Section 3 unclear. Are these experiments used to corroborate the complexity result in Theorem 1? If so,  are the activation functions in the implemented KAN chosen to be linear combination of basis functions?\n- Can the authors comment further on the convergence rate of excess risk established in Corollary 1 and Corollary 2? What does the parameter $s$ represent? Also, how do these convergence rates compare to that of traditional MLP (say, ReLU feed-forward neural network)? Do we observe any advantage in the generalization of regression by using KAN? \n- It appears that Section 2.3 only considers RKHS induced by Matern kernel, instead of a general low-rank RKHS. Because of the equivalence between this specific RKHS and Sobolev space, would it be a better idea to change the title of Section 2.3 to ``Activation functions on Sobolev space\"?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wiEFHkVHiE", "reviewer": "Reviewer_mrKs", "summary": "This paper explores the generalization capability of Kolmogorov-Arnold Networks (KANs). Specifically, the authors estimated the covering number under certain regularity assumptions and derived the generalization gaps (Theorems 2 and 3) and the excess risk bound (Corollary 1). Additionally, they examined the case where the activation function belongs to a Reproducing Kernel Hilbert Space (RKHS) with a low-rank structure, proving the generalization gaps (Theorems 4 and 5) and the excess risk bound (Corollary 2) that leverage this structure. Experiments validate the relevance of the theoretical bounds to the observed excess risk.", "questions": "- Based on the obtained theory, is it possible to identify a problem setting where KANs outperform MLPs?\n\n- How tight are the obtained results? Is it feasible to derive lower bounds on generalization gaps and excess risk bounds in a simple setting?\n\n- Can you discuss the approximation (representation) capabilities of KANs for certain function classes? This type of approximation analysis is often necessary for fully understanding the statistical performance of the model.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "KANs, inspired by the Kolmogorov-Arnold representation theorem, have gained attention for potentially efficient parameterization compared to MLPs. However, their statistical performance remains unclear, despite numerous attempts to develop more sophisticated structures and applications. To the best of my knowledge, this is the first work to derive generalization bounds for KANs.", "weaknesses": "- Assumption 1 presumes the uniform boundedness of the dataset. Typically, this bound scales with $\\sqrt{n}$, which could degrade the derived bounds.\n\n- Although KANs are expected to demonstrate superior approximation capability due to their inherent nature, there is no comparison with other models such as MLPs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N6S4TyPnfm", "reviewer": "Reviewer_u9e2", "summary": "This paper investigates the generalization ability of Kolmogorov-Arnold Networks (KAN). Multiple generalization bounds are provided for both bounded and unbounded loss functions, which may help understanding how the network complexity impacts the generalization behavior of KANs. The results are further extended to the cases where activation functions belong to RKHS, and empirical results are given to demonstrate the relationship between performance and network complexity.", "questions": "* Could the authors provide some empirical results on the comparison between the bounds and the actual generalization error? This would help convince me that these bounds are indeed non-vacuous in practice.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper provides multiple generalization bounds for both bounded and unbounded loss functions, which may help in understanding how the network complexity impacts the generalization behavior of KANs.", "weaknesses": "* It is consensus that traditional complexity measures (e.g. Rademacher complexity or covering numbers) are not sufficient to explain generalization for deep networks. As the analysis of this paper heavily relies on these metrics, the derived bounds are likely vacuous in practice.\n* The theoretical analysis relies on tons of strong assumptions (e.g. Lipschitzness, boundedness) for almost every single part of the network (input, function space, loss function). To what extent these assumptions can be met in practice needs more clarification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ovHs72jlmA", "reviewer": "Reviewer_xDbU", "summary": "The paper studies generalization bounds for Kolmogorov–Arnold Network (KAN) using covering numbers (through Rademacher complexity). The authors further present a quantity $\\alpha$ which they dub the network complexity and show empirically this measure qualitatively follows the excess risk.", "questions": "- For the CIFAR-10 and MNIST experiments, why did the authors choose to run the experiments on top of the features of AlexNet instead of the “raw” input? How would that change the results presented in the paper?\n- Figure 2 presents the excess loss and the “normalized” complexity measure as a function of the training time, however, it is never stated explicitly how the complexity measure depends on the training time. Am I correct that this dependence comes from the network parameters $B_i$? What are the values of $B_i$ used for the plot? Is it the $\\ell_1$ norms of $\\mathbf{B}_i$?\n- Is it true that the theoretical results do not hold for cross-entropy loss? Can the authors comment on that in light of the use of cross-entropy loss in the experiments? \n- Can the authors provide an intuitive explanation for the role played by the value of $\\phi$ evaluated at $0$ (and the constant $C$)?", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "**Originality**. The contribution seems to be novel. To the best of my knowledge, this is the first work to present generalization bounds for KANs.\n\n**Quality**. The theoretical contribution seems to be sound.\n\n**Clarity**. The paper is well organized.\n\n**Significance**. KANs have attracted much attention from the community lately and theoretical contributions would be very welcome.", "weaknesses": "**Clarity**. While the flow of the paper is clear, I wish technical results were complemented by an intuitive discussion. Specifically,\n- The third point under main contributions claims “The results shed new light on how the performance of KANs is affected by the underlying network structure and the network complexity.” However, this point is never presented or discussed in the paper.\n- Empirical results lack discussion completely.\n\n**Quality**. The empirical results should be more comprehensive and include more details.\n- Some very particular choices are made for the synthetic target functions, however, they are never motivated or explained.\n- The performance of the networks (loss/accuracy rather than excess loss) is never presented. This is important in order to evaluate whether the regime the authors present in their result is an interesting one or not.\n- The paper does not include empirical results for the generalization bounds. It would be of great interest to the reader to see whether the bound is tight.\n- The tightness of the bound is never discussed.\n- I would expect an appendix with details of the experiments. Specifically, the “normalization” procedure for the complexity measure is never explained. \n\n**Significance**. The significance of the result largely depends on the tightness of the bound, but it is never discussed or examined.\n\n*Nits*:\n- The appendices are never referenced. I would expect them to be referenced when relevant.\n- The text in the figures is barely readable at any reasonable “zoom” level.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yyVVaJIH4W", "reviewer": "Reviewer_geZs", "summary": "The paper presents generalization bounds for KANs, where the activation functions are learned from the datasets. The paper considers activation functions that are eighter represented by linear combinations of basis functions or lying in a low-dimensional RKHS. The paper presents generalization bounds based on covering number analysis under some assumptions on the Lipschitzness of the activation functions and the l1-norm on the coefficients. The paper further shows that the analysis can be extended to unbounded loss functions, which can be applied to general regression loss functions. Experimental results are also given to verify the theoretical findings.", "questions": "- It would be nice if the authors can included a detailed discussion on the technical challenge in the covering number estimates of the KANs. Especially, the authors should clarify the novelty of the theoretical analysis.\n\n\n\n- Is it possible to give high-probability generalization bounds for unbounded loss functions.\n\n- There is a lacking discussions of the generalization bounds for KANs with those for fully connected networks. Is there a clear advantage on the generalization behavior by theoretical analysis in the paper?\n\nMinor point:\n\n- $\\mathcal{F}$ in Proposition 2 is not defined.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "KANs have found various applications in many scientific and engineering problems. The paper gives solid generalization analysis for KANs. The derived generalization bounds clearly show how the parameters affect the generalization, including the Lipschitzness and l1-norm constraint on the coefficients.", "weaknesses": "- The technical contribution seems to be limited. As far as I see, the theoretical analysis follows from existing covering number estimates of fully-connected neural networks [R1]. The idea is to build a recursive relationship of covering number estimates for neural networks with different layers (Proposition 1), which shares similarity to Lemma A.7 in [R1]. Furthermore, Proposition 2 also essentially similar to Lemma 3.2 in R1.\n\n\n- The paper also gives generalization bounds for learning with unbounded loss functions. However, the results admit some drawbacks. First, the bound in Theorem 3 has a linear dependency $1/\\eta$, where $\\eta$ is a confidence parameter. This is much worse than the logarithmic dependency. Therefore, the results for unbounded loss functions are not with high probability. Indeed, they are just derived by truncating the prediction followed by Markov's inequality. Second, the bound involves $n^{(1-s)/(2s)}$ and would be much worse than the case with bounded loss function ($n^{-1/2}$). For example, if $s=2$, then the bound in Theorem 3 is of the order of $n^{-1/4}$.\n\n\n\n\n\nR1: Bartlett, P.L., Foster, D.J. and Telgarsky, M.J., 2017. Spectrally-normalized margin bounds for neural networks. Advances in neural information processing systems, 30.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)