Generalization Bounds for Kolmogorov-Arnold Networks (KANs) and Enhanced KANs with Lower Lipschitz Complexity
Authors: Pengqi Li
Lizhong Ding
Jiarun Fu
ChunhuiZhang
Guoren Wang
Ye Yuan
Venue: NeurIPS 2025 poster ()
Keywords: Generalization bounds
Kolmogorov-Arnold Networks
Model complexity.
Primary area: theory
TL;DR: This paper explores the generalization mechanism of KANs and designs more effective KANs with lower model complexity and better generalization.

Abstract: Kolmogorov-Arnold Networks (KANs) have demonstrated remarkable expressive capacity and predictive power in symbolic learning. However, existing generalization errors of KANs primarily focus on approximation errors while neglecting estimation errors, leading to a suboptimal bias-variance trade-off and poor generalization performance. Meanwhile, the unclear generalization mechanism hinders the design of more effective KANs variants. As the authors of KANs highlighted, they ``would like to explore ways to restrict KANs' hypothesis space so that they can achieve good performance''. To address these challenges, we explore the generalization mechanism of KANs and design more effective KANs with lower model complexity and better generalization. We define \textit{Lipschitz complexity} as the first structural measure for deep functions represented by KANs and derive novel generalization bounds based on \textit{Lipschitz complexity}, establishing a theoretical foundation for understanding their generalization behavior. To reduce \textit{Lipschitz complexity} and boost the generalization mechanism of KANs, we propose Lipschitz-Enhanced KANs ($\textbf{LipKANs}$) by integrating the Lip layer and pioneering the $L_{1.5}$-regularized loss, contributing to tighter generalization bounds. Empirical experiments validate that the proposed LipKANs enhance the generalization mechanism of KANs when modeling complex distributions. We hope our theoretical bounds and LipKANs lay a foundation for the future development of KANs.

Generalization Bounds for Kolmogorov-Arnold Networks (KANs) and Enhanced KANs with Lower Lipschitz Complexity
Pengqi Li Lizhong Ding" Jiarun Fu Chunhui Zhang Ye Yuan Guoren Wang
School of Computer Science and Technology, Beijing Institute of Technology (pengqi lii, jrfu, zhangch, yuan-ye)@bit.edd edu. cn lizhong ding@outlook. com wanggrbit0126.com
Abstract
Kolmogorov-Aroold Networks (KANs) have demonstrated remarkable expressive capacity and predictive power in symbolic learning. However, existing general- ization errors of KANs primarily focus on approximation errors while neglecting estimation errors, leading suboptimal varsnnariance trade-off and poor general- ization performance. Meanwhile, the unclear generalization mechanism hinders the design of more effective KANs. As the authors of KANs highlighted, they "would like to explore ways to restrict KANs' hypothesis space so that they can achieve good performance." To address these challenges, mechanism of KANs and design more effective KANs with lower model complex- ity and better generalization. We define Lipschitz complexity as the first structural measure for deep functions represented by KANs and derive novel generalization bounds based on Lipschitz complexity, establishing theoretical foundation for un- derstanding their generalization behavior. To reduce Lipschitz complexity and boost the generalization mechanism of KANs, propose Lipschitz-EEnhance KANs (LipKA Ns) by integrating the Lip layers and pioneering the Lu.srregularizatinn contributing to tighter generalization bounds. Empirical experiments validate that the proposed LipKANs enhance the generalization mechanism of KANs when modeling complex distributions. We hope our theoretical insights and proposed LipKANs lay a foundation for the future development of KANs.
explore the generalization
Introduction
KANs represent multivariate functions as layer-wise iterative compositions of univariate continuous functions [Liu al. 2025], indicating the strong expressive capacity to model complex distributions and possessing faster neural scaling laws. The fundamental difference between KANs and existing neural network architectures (He al.] 2016 Vaswani [017 Brown et al. 2020] is that KANs replace the single activation in each layer with a family of activations. KANs demonstrate strong accuracy and
interpretability on Al Science tasks (Shukla et al.) 2024 Wang et al. 20256), while also achieving promising results in socio-economic tasks Xu et al. 2024b). engineering-related tasks MMubarak 2024 Kundu et al.) 2024 Bresson al. 2024 Carlo al. 2024], and interpretability-diieee
al.
tasks (aalitsky] 2024]. While KANs demonstrate strong approximation capabilities and excel in symbolic learning, their generalization does not show significant advantage over existing models (Yu ctt al. 2024 Zheng et al. 2025). We suggest that the suboptimal generalization performance of KANs arises Trom an increased hypothesis space complexity, which introduces an approximation-esiiaaiio error trade-off: while lowering approximation error, it elevates sample complexity, thereby raising
"Corresponding Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

estimation error and ultimately harming generalization. Parameter count serves an insufficient proxy for hypothesis space complexity in KANs, and naively reducing parameter counts fails to improve generalization. Consequently, understanding KANs' generalization mechanisms and developing complexity measures beyond parameter count are critical for enhancing their applicability and advancing more effective KANs-based models.
Existing studies on the generalization of KANs primarily focus on bounding approximation error. For [2025] derive a more general approximation result extending beyond depth-2 Kolmogorov-Arool representations, and Qiu et al. [204] provide distance-award error bounds for KANs. Wang et al.
instance. Ataei et al.
[202aa] rigorously establish that KANs are at least as expressive as MLPs by showing that any MLP can be reparameterized as a KAN with degree it splines which only slightly larger and KANs can also be represented by MLPs, implying that KANs with large grid sizes may be more efficient approximating certain classes of functions. However, these studies largely overlook the impact of estimation error, limiting their contribution to improving KANs generalization performance. To our knowledge, the only existing generalization bounds (Zhang and Zhou, 2024) provide a foundation for KANs but overlook their structural complexity, yielding limited insight into their generalization mechanisms and offering little guidance for more effective models.
In this paper. we derive the generalization bounds for KANs and propose more effective Lipschitz- Enhanced KANs (LipKANs), with lower complexity and better generalization performance. We reformulate KANs in Eq. (I) by treating each layer combination of activation functions and linear transformation, highlighting that the core of KANs' expressive capacity arises from the family of activations in each layer. We define a structural measure, Lipschitz complexity, for the function family represented by KANs and, based on this, directly evaluate the generalization error of KANs by deriving their generalization bounds in Theorem 4.5] These bounds: (a) scale with Lipschitz complexity of KANs; (b) have no dependence on combinatorial parameters (e.g. number of layers or cardinality of activations family) outside of log factors: (c) regard the previous generalization bounds for MLPs as special cases; (d) accommodate the selection of various activations. We show that adopting activations with lower Lipschitz complexity results in tighter bounds, while incorporating complexity-relater regularization during training helps trade off bias and variance. Leveraging the theoretical analyses, design LipKANs, which incorporate the Lip layers and replace the L1 Le-reuularized loss [Liu et al. 2025) with the L1.s-regularize loss, where the Lip layers ensure the low Lipschitz complexity, and regularization helps balance the bias-variance trade-off. Significantly, we conclude that LipKANs lead to better generalization both theoretically and empirically.
Contributions This work establishes the generalization bounds for KANs and proposes LipKANs, with an enhanced generalization mechanism. The central contributions are as follows:
Theoretical Foundation: We are the first to rigorously define Lipschitz complexity as a measure of the structural complexity of deep functions represented by KANs, and to derive complexity-dependent generalization bounds.
Architectural Innovation: We introduce generalization-drives paradigm for model design that establishes clearer generalization mechanisms. Specifically, we introduce LipKANs, which insert Lip layers and apply L1.5 reggulaarizaiin to explicitly bound network complexity.
Broad Adaptability: We evaluate LipKANs on vision, text, and multimodal benchmarks, demonstrating that they consistently improve existing KANs, with significant accuracy gains when modeling complex distributions.
2
Related works
2.1 Generalization of Deep Models
Generalization error is defined as the gap between the expected risk (expected loss over the data distri- bution) and the empirical risk (observed loss on the training set). Therefore, smaller generalization error indicates better model generalization. A generalization bound a theoretical upper bound on the generalization error, taking the form: expected risk < empirical risk + complexity + confidencee, where the complexity term reflects the expressiveness of the hypothesis set, and the confidence term quantifies how likely the bound holds.
2

---

Most of the studies on the generalization of deep models are based on generalization bounds (Jang
al.
2019 Dziugaite et al. 2020). Depending on the selection of the complexity term, they can be
classified into: VC dimension bounds [Shalev-Shwarte and Ben-David, [014 Neal et al.) 2018 Ding
et al. 2020b Yang et al. 2023]. Rademacher complexity bounds [Bartlett and Mendelson
2002
Sachs et al.] 2023]. margin bounds [Bartlett et al. 2017 Neyshabur et al., 201 20.1
Ding et al.
2018
Barron and Klusowski 2019], NTK-based bounds AAroa et al. 2019], BaC-Bayes bounds [Ernngis
ct al. 2024]. and uniform stability bounds (Hardt et al. 2016 Mou et al. 2018 Attia and Koren
2022 However, many generalization bounds fail to reveal the underlying generalization mechanism
of the models, making them less helpful for guiding model enhancements.
2.2
Kolmogorov-Arnole Networks (KANs)
Since the emergence of KANs [Liu et al. 2025 J, future researches can be categorized into four
directions: (I) Explore the applications of KANs in fields such as graphs (Bresson et al. [022]] Carlo
al. 2024 Kiamari et al., 2024| Zhang and Zhang 2024], computer vision [Azam and Akhtar
2024
Li et al.) 2024 Cheon, 2024, |Syydi et al. [024]. kernel learning (Zinage et al., [024] and
quantum science [undu et al. 2024] Ahmed and Sifat 2024 These studies have expanded the
application scope of KANs, but lack theoretical analysis Liu et al., 2019]. (II) Explore various
activation functions, including Gaussian radial basis function [i 2024], Fourier series [Xu et al.
2024a Chebyshev polynomials (SS and R [024]. finite basis [Howard et al., 2024 Tal [024]]
wavelet [Bozorgasl and Chen 2024| Seydi et al. (024), Jacobi basis functions [Aghaci] 2024a
polynomial basis functions [Scydi] 2024), and rational functions [Aghaci 2024b). , These KANs
variants propose empirically more effective KANS but lack a theoretical foundation. (III) Combine
KANs with models like Convolutional KANs [Bodner al., 2024]. federated KANs [Zeydan et al.,
2024] and PowerMLP [Qiu et al. 2024). These works blend KANS with other deep networks, though they may sacrifice KANs inherent expressive capacity. (IV) Research the approximation error of KANs Qiu et al.) 2024 Ataei et al.) 2025). Our study on generalization error overlaps with this area, suggesting that focusing only on approximation error may reduce the model's reliability and stability when handling real-world data, thus limiting its practical utility. The trade-off between estimation and approximation errors is crucial for improving generalization performance.
3 Notations and Preliminaries
In the supervised learning scenario, we assume that data points (I,. E X x y for i E [m] are drawn independently from a distribution P. The sample is denoted as = {(11, V1), (22, v2)................. ... (F Un]} ~ Ipn The input sample can be represented in matrix form as X (I1, T2,,.. In.
Given a loss function L(.,)) y x y - R_ and hypothesis space H. the expected loss of f E H is defined as R(f) = [) [Z)f())))) and the empirical loss 1 E._, (J (x;) v:). In Section 4 we aim to derive upper bounds for R(f) that hold for all E H with high probability. These bounds will depend on both R(ʃ) and the complexity term. In Appendix C.I formally define the norms of vectors (e.g.. 2;). matrices (e.g., the input matrix X) and dimenniimenional tensors (e.g., the weight tensor W, of KANs). Our analysis focuses on sub-Gaussian distributions PP formal definitions of sub-Gaussian random variables and vectors are provided in Appendix C.2
of f E H on the set S is given by R(ʃ)
KANs. It was established by KART Koomogorov
1961 that, for continuous f 1]" R".
there exist continuous univariate functions D. : R - IR and Φg.n : [0, 1] + R such that:
2n+1
f(x) = f (I1, Fn) =
Σ Qqp (Ipp) 9=1 p=1
+ 4
To expand the family of functions that a two-layer KAN can represent, it is generalized to be both wider and deeper, and is parameterized by 1D functions using B-spline basis functions (Liu et al. 2025]. The underlying idea is that when using a two-layer KAN to represent functions, some of ID functions may be difficult to learn, while these functions become learnable when the network is
sufficiently deep. For example, ʃ (I1, 22, Uz, x34 = exp (sin (2] + +z)) + sin (rī + xx)) cannot be
smoothly represented by a two-layer KAN but can be smoothly composited by a three-layer KAN
with the width [4, 2, 1, 1]. We define the activations family of cardinality G as Σ {or R - R. k
3

---

[G]]. Generally, we consider a KAN of depth L and width [d1, .2..... ,d), where the [-th layer is defined by the vector-valued map 1 : R + dh with vy(x) = (x),1())..... Vi,d, (x)) and each neuron given by (x) =
d Ej Σ k=1 Wij w.jjao (xz)), where in w..jkk. indexes the neuron in the next layer, j refers to the neuron in the current layer, and corresponds the basis function from Σ.
Lipschitz Property. Let (X, dx) and (Y, dy) be metric spaces. A function f X + Y is called Lipschitz if there exists L E R. such that dy (f - L. dx v), Vu, u X. The infimum of all L in this definition is called the Lipschitz norm of f and is denoted |f As for binary functions (such loss functions L(:-)) y x y + +)), define their Lipschitz norm by fixing the parameter at one position, represented as a function ( =
The Lipschitz property
of various function families is discussed in Appendix
4 Generalization Analysis
In this section, we provide a theoretical framework for understanding model's ability to generalize from training data unseen data, which also plays a crucial role in guiding model selection and regularization strategies. We first formalize KANs in terms of activation layers and linear layers, then we define Lipschitz complexity as a measure for functions represented by KANs and derive uniform high-probabiliyy generalization bounds, thereby revealing the generalization mechanism of KANs.
4.1 Reformulation of KANs
For reformulation, we denote the family of non-linearities Σ applied element-wise to Rd as (x)) E _dx where each entry is given by (M(=)); = (x:)). We introduce the weight tensor Wi E BdxxdixxxG xG with (Wii.j. = Wi.j.-- Using the Einstein summation operation (einsum) "O" (defined in Appendix C.4). we compactly express V,, as: V. (a:) W E(x). By iterating
2 = WW o E(x1-1) with Zo = æ, we formally represent KAN as a composition of linear layers Wi and mon-lincassties Σ:
Definition 4.1 (Reformulation of KANs). Given an activation family Σ for R + R k E [G]]
a KAN with depth L and layer widths [dg, di....,dd dz] is defined as a composition of nonlinear activation layers Σ and linear layers WP- (W1, 1: W...... W....., WL):
KAN(z) ()) = Fw (z) = (W, o Vo W1-10--- WI)
(I)
For comparison, an MLP with linear weight matrices Wi Rdxdd and non-linearity a is expressed:
MLP(x) = (WL o WI-1 0...00 Wi) (z),
where a : R + R is an activation function (e.g., ReLU,, sigmoid, or Tanh). Thus, in each layer, KANs replace the single activation in MLPs with a family of activations, contributing to their strong expressive capacity.
4.2 Assumptions
For our research subject, KANs, we make the following assumptions:
Assumption 1 (Sub-Gaussian). We assume that each row a; E Rdd of the input data matrix X an independent and identically distributed sub-Gaussian random vector, with the corresponding sub-Gausse Sab-Gaussian norm K == max, ||z:||lz-
Assumption 1 is satisfied by a bounded dataset, and ensures the concentration of the data matrix ||||| with high probability. Indeed, we have VL > 0, Pr ([] 2 C(vák + ()) & e where C constant. Using the concentration bound for the sum of squared sub-Gaussian vectors, we obtain the following lemma:
Lemma 4.1. Under Assumption 1, we have
Pr
[|x]|2 2 CK (1d
(2)
where C' is a constant.

---

Lemma 4.1 demonstrates that, under assumption 1, |XX]] concentrates around OK Vnd) with exponential tail decay, ensuring that deviations beyond this bound are extremely unlikely. detailed discussion of the theoretical extension to other distributions [Ding et al. 2019 is provided in Appendix
for Definition 4.2 (Lipsehitz Norm of Functions Family). Consider a finite family of functions
R - IR, E [G]]. which corresponds one-to-one with a matrix-valued function My : IR4 + Rdxc and for given vector z (z1 .2. x.) given by (x;). The Lipschitz norm of this matrix-valued function is then defined as:
R the (i, k)-th element of the matrix My(#) is
|M
Σ lowlllip
For convenience, we write |SD|[ı : ||
Assumption 2 (Lipschitz). We assume the family of basis functions Σ applied to KANs has bounded Lipschitz norm | Σ C, with Ei[cap defined in Decinition 4.2
To check the consistency of this definition with the previous definition of the Lipschitz norm for single function, we note that the vector space and matrix space are endowed with the vector 2-norm and the matrix (2, 2o-mm, respectively. We cautiously point out that the notation [[F]|ra here involves slight abuse of symbol, as it does not represent norm in the formal mathematical sense.
Assumption 3. We assume that the loss function L(:,-) is Lipschitz continuous with respect its first positional argument, i.c. maxzey p(y) < oc. Further suppose that L(.,-) S M.
This assumption ensures the Lipschitz continuity of the mapping from the hypothesis space to the loss function space. Both cross-entropy loss (p(y) = 1) and margin loss (p(v) = 4 satisfy this property, as well as Lp-loss on a bounded domain.
4.3 Generalization Bounds for KANs
The primary goal of this section to derive uniform high-probability generalization bounds for KANs with auxiliary lemmas (please refer to Appendix
We outline the proof by first deriving layer-wise covering number bounds via induction (Lemma 4.2 then establishing tensor covering results for the linear transformations within each layer (Lemma 4.3 and finally obtaining the generalization bounds for KANs using standard generalization analysis Techniques (Theorem
We begin by decomposing the covering number for multi-layer function compositions into covering numbers for layer-wise functions. Our techniques are similar to Lemma A.7 [Bartlett et al. 2017].
Lemma 4.2. Let (t,....,(.)) ). be given. Under Assumption and 2, each as. is Lipschitz continuous with Pk = ||o Let p == (PI....,PG). Suppose the tensors W (WI..... Wi) lie within B, x ... x Br where B; are arbitrary classes with the property that each W E B, has [ 5 cs. Then, letting TI 61 and = El= cjllel12)e, the neural net images
Hx = { (Fw/ (X) Wt E B, ... Bi, have a covering number bound:
N (Hxx,T,,ll 112)) <
II sup
N ((W:oE (x))= (( W, E (e.],66l111
(W...... W
==1
VJCi-WEE,, EB,
The complete proof is provided in Appendix Lemma A.2 shows that it suffices to study the covering number of the (-th layer of the KAN, given that the first / layers are fixed.
To establish the tensor covering bounds for the linear transformation, we demonstrate Lemma that considers W o Σ(X), where W represents the weight tensor for given layer, and X denotes the data passed through all preceding layers.
5

---

Lemma 4.3. Let conjugate exponents (p. q) and (u, v) be given with S 2 as well as positive reals
(a,b,e) c) and positive integer m. Let data X be given with [Σ(X) 1lp S Then
d'
In (f X(X) W E Rd xdxG | S a -11111
In(2dd' G),
22
where [x] returns the smallest integer greater than or equal to x.
The complete proof provided in Appendix E.2 Now define Lipschitz complexity as a measure for the functions represented by KANs.
Definition 4.3 (Lipschitz Complexity). The Lipschitz complexity Rwt of KAN Fwz with weights
Wf (W1, W2...., WL) is defined as:
Rwf == (I>ILLE
II Wollo
Σ
E
2/3 2,2,11
where WT refers to permuting the 3D tensor W, from (diml, dim2, dim3) to (dim2, dim3, diml).
Remark on Lipschitz Complexity. Lipschitz Complexity acts as complexity for hypothesis space [Mohri 2018 Ding et al. 2020a] and is indeed distinct from algorithmic complexity. While the formal definition of Lipschitz complexity looks highly structured, it actually systematically accounts for both intuitive factors (e.g., network depth L and spectral weight norm [Waloo and non- non-intuitive factors (e.g., smoothness the basis family |اا|[ip and specific (2,2,1) weight norm [W] 1l2,2,1) that characterize KANs. Besides, the logarithmic dependence of generalization bounds and basis family size G ensures their marginal impact on complexity compared to the dominant term Ry Finally, under Assumptions & will strictly bound the leftover terms by constants. These designs make Lipschitz complexity sufficient measure higher Rwz makes generalization more challenging, whereas lower Ry facilitates generalization.
width
With the above results, we can provide bound on the covering number of Hxx specializing to the particular case of Lemma 4.3] where we take p = q = 2, u oo and 1.
Theorem 4.4 (Full-Network Covering Bounds). Under Assumption we can bound
R²
we log(2dG)
log (H,, 1122) S
(5)
22
where d := max, d,-
The whole proof of Theorem 4.4
provided in Appendix has no dependence on combinatorial parameters such as the number of nodes and the number of activations outside of the logarithmic factor. Built upon the above results, now derive bounds the generalization error R R(Fw+), ). The following theorem provides generalization bounds for KANs whose activations is fixed but Lipschitz complexity is bounded.
B.3 It worth noting that the above bound
Theorem 4.5 (Generalization Bounds for KANs). Under Assumption 2 and 3, let fixed Lipschitz
activations Σ {ax k E [G]) and weight tensors Wt- (W1, Wz...., WL) be given. Then for (x. y), [11 31), (Ia, Vn) drawn i.i.d. from distribution IP, we have with probability greater than 1
R(FwF) R(Fw4) SO (1ix max, (p(v))
1/c n
log(d") +
(6)
Remark Generalization Bounds. We acknowledge that the above bounds may achieve optimal tightness. Nevertheless, our bounds reveal KANs' generalization mechanisms through Lipschitz complexity, provide theoretical guidance for model design, and predict empirical behaviors like the O(n- error decay. While our current generalization bound does not make inter-layer dependencies explicit, it already captures layer-wise alignment through the operator norm products I'' Wallo which originate from the covering number bound of the entire network in Lemma 4.2

---

This implicitly reflects structural relationships across layers. We agree that making these dependencies more explicit could tighten the bound, and we consider this a valuable direction for future refinement.
Some studies [Qiu et al. 2024 Wang et al., 2025a] theoretically demonstrate stronger expressive power of KANs than MLPs bounded domains. Here, remark that special case of generalization bounds corresponds precisely to the generalization bounds of the MLPs [Bartlett et al. 2017]. When the activations family > contains only single activation function, we have G 1, then the result coincides with the classical margin bound in [Bartlett et al. 2017].
Based on Assumption 1, which states that the distribution IP satisfies the sub-Gausse Gaussian property. can bound || in Eq. (6) with high probability, leading to more applicable bounds in Corollary [E.] (See Appendix E.5). Based on the generalization bounds of KANs in terms of Lipschitz complexity, we gain the insight that adopting activations with lower Lipschitz complexity results in tighter bounds, while incorporating [W; 1 rrelated regularization during training helps trade off bias and variance in Eq. (6). Building upon this, in the next section, we propose LipKANs, which enjoy tighter generalization bounds.
5 LipKANs: Generalization-Driven Models
With our efforts in Section 4. in this section, we introduce LipKANs, which incorporate the Lip layer and replace the L1 rregularizedd loss (Liu al., 2025) with the L L1.-rregularized loss, where the Lip layer ensures the low Lipschitz complexity, and regularization helps balance the trade-off between bias and variance.
We first introduce the structure of LipKANs. From Theorem 4.5 it follows that adopting function families with greater Lipschitz smoothness leads to tighter bounds, while incorporating complexity- related regularization during training alleviates training costs. Therefore, we design the Lip layer in each KANs layer between each activation layer Σ and linear layer W, reduce the Lipschitz norm of the basis function family. The fixed Lipschitz function & + R satisfies ||E]|Tip S (e.g., g., tanh, Sigmoid, ReLU). Given activations family Σ for R IR k E [G]}, LipKAN with depth L and layer widths [do, di..... .......d,)) dz.] is defined as a composition of nonlinear activation layers E, Lip layers and linear layers W (W1, W 22. ..... Wy):
(ip = FwF (xx) = (,, ofo o WL-1 ....o W, o
where acts element-wise Next, define the L.ssregullriizdd loss address the imbalance between
Algorithm 1 Generalization LipKANs with L1.5 Regularization
Require: Sample S = (0,, y:) ) Activations family E, Function with |4]|μ I 1, Learning rate 17. Regularization coefficient X.
Ensure: Model weights WP = (W1,... W.).
1: Initialize weight tensors W/-
with Kaiming initialization HHe et al. 2016
2: Initialize AdamW optimizer with parameters:
8n f0=(W], 17, Awd (weight decay), 81 0.9, B3 0.999.
3: for each epoch t = 0 to T do
4: for each batch (Ir 9:) E S do
5: - (0)
6: for j = to L do
Forward propagation. 20) + olo E(2(i-1)) b Leading to lower |[E[[ii and Rw (Definition 4.3)
7:
8:
end for
Fwy (L)
9:
10: = Lentropy + x Σ/= W 222,1
> LI. enhances generalization (Corollary 5.1 Employ the AdamW optimizer.
Update: 0c+1 Adam W (0,, L, C, Awd)
12:
end for
end for
14: return W - 0r
// Integrate Lip layers and design L1.s regullaaaaiioiiii optimized via Adam W to enhance general- ization and training stability.

---

bias and variance. As illustrated in Algorithm [ we note that the design of L1.5 -loss originates from generalization bounds (Theorem 4.5) which regularize |W7 2,2,1 in Rwz leading an improved
bias-variance trade-off (Corollary E.1 Given dataset 5= { (æ1. Vn), (Z2, Vz),.... (In: yn)} ~ Ph",
basis function family Σ and weight tensors W (WI.
1. W W..... 2..... WL), we design the L1.5-
regularized loss to replace the Le-reuularized loss in [iu et al. 2025 based on Theorem[4.5 as:
LL {) Loe (S; W/. () Σ 12...11
where Lee is the eross-entropy loss and À controls the scale of regularization. Then we employ Ad- maW optimizer [Loshchilov and Hutter 2017] with the optimization problem of LipKANs designed as: Wif arg mi (S; WF: E). We base model selection on theoretical foundations, but empirically compare regularization norms (L,, Lts), in Section and and discuss their theoretical impli- utilize L--reuularization
cations Appendix B Both KANs (Liu et al., 2025) and Efficient MeffcientKKANs S
with the penalty term L |W lll2 to promote sparsity and interpretability. In contrast, we adopt re.-regularization, which enhances generalization performance. A promising future direction is to explore integrating L1, L1.s, and the entropy regularization to leverage their complementary advantages. We analyze the cost of Lip layers and regulaaauaoinotioon in Appendix D.3
Remark on LipKANs. Both components are principled and arise naturally from our formulation of Lipschitz complexity (Definition[4.2) Specifically, the Lip Layer constrains the smoothness of the basis family through the Lipschitz norm of E, while the L1.3 L...-regularizztte (2,2,1) norm of the linear weight tensors W. These two components regulate orthogonal axes of Lipschitz complexity and hence play complementary roles in promoting generalization.
penalizes the structured
Finally, we prove that I. LipKANS exhibit lower Lipschitz complexity, which tightens the generalization bounds and theoretically enhances the generalization performance.
Corollary 5.1 (Generalization Bounds for LipKANs). Under Assumption Lipschitz activations Σ = {ax A E [G]}, weight lensor Wt = (W W2...., W.. and Lipschitz function : R + IR with [[]]zzip S Then for (æ. (1119)))-- (mn, Un) drawn i.i.d from sub- Sab-GGussiann distribution P. we have with probability greater than
and 3, let fixed
8,
KR max; (((x))
1/e
R(Fwy: R((Fwy ,) SO
(aae(PGG)i log(d G)y poyydoog dogg
where the right-hand side of the inequality is also bounded by the upper bound in Theorem
4.5
The proof is deferred to Appendix E.5 Due to Corollary 5.1 adopting activations with lower Lipschitz complexity results in tighter generalization bounds, suggesting better generalization performance theoretically. In the next section, we will empirically evaluate the performance of LipKANs.
Bias- Variance Trade-Off. To clarify the bias-variance trade-off in our setting, we revisit the generalization bounds in Theorem 4.3 and Corollary|55.] [5.]] The Lipschitz complexity term reveals that the model variance-its sensitivity
training data perturbations-is closely related to the structured (2, 2, 1) ---oorm of the weight tensors, which controls the overall magnitude of the spline coefficients. A larger norm enables greater flexibility in fitting the data but also increases the risk of overfitting, thereby contributing to higher variance. In contrast, the bias stems from the approximation capacity of the basis functions (e.g.. B-spline, Fourier, and Chebyshev basis). These basis functions provide strong approximation capacity for smooth target functions, enabling the learned model to closely align with the underlying data distribution when appropriately regularized. To this end, the proposed le.ssregularization penalizes the structured norm, reducing variance while preserving sepressivity-therte thereby encouraging a favorable bias bars-nariance trade-off and enhancing generalization.
Experiments
6.1 Deep Learning Tasks
Baseline. We choose Original KAN Liu et al. 2025 Rational-KAN (Yang and Wang 2024].an ,and RBF-KAN Ta 2024 as our baselines. particular, Rational-KAN and RBF-KAN replace the
8

---

B B-spline function with their respective basis functions. For all baseline methods, a LayerNorm layer added before each KAN-Linear layer to enhance training stability. All KANs are set with grid size of 5, grid range of [-10, 10), and hidden layer width of 64. The Lip layer of LipKANs tried using tanh, sigmoid, and ReI.U, and selected the best-perforning Lip function l.
Datasets. We test the performance of the Lip structures on original KAN, Rational-KAN and RBF-KAN through benchmark datasets spanning image (MNIST (LeCun et al. 2010 CIFAR-10
Krizhevsky et al., 2 CIFAR-IO [Krizhevsky et al. 2009]. STL-10 10 [Coates al. 2011). text
(AG News [Zhang ct al. 2015]), and multimodal domains (AVMNIST [Liang et al. 2021J. MIMIC-
III (Johnson al., 2016 Harutyunyan et al. 2019) ) confirming LipKANs' robust generalization
improvements across broad applications.
All reported accuracy and F1 scores in Table are evaluated on held-out test sets using the officially predefined training/test splits for each dataset, consistent with the evaluation protocols established in prior works [Liu al. 2025 Zheng al., 2025].AA ..As shown in Table T. the design of LipKANs demonstrates significantly improved generalization across various KANs when evaluated on complex datasets (SLT-10. CIFAR- 100, AG News, and MIMIC-III). In contrast, simpler benchmark datasets (MNIST, CIFAR-10, AVMNIST) demonstrate statistically insignificant performance variation, even exhibiting marginal accuracy degradation in certain cases. This phenomenon stems from the inher- ently low- owwcomplexity distributions captured by KANs on such datasets, rendering hypothesis space complexity reduction unnecessary.
Table 1: Evaluation of KANS and LipKANs on broad domains including CV, NLP,, and MM (F1 score (%) for MIMIC-III; accuracy (%) for others) with corresponding 95% confidence intervals computed over five independent runs.
Model
SLT-10
MNIST
CIFAR-10
CIFAR-100
AG News
AVMNIST
MIMIC-III
KAN
25.94 + 0.53
95.62 0.08
46.01 = 0.50
14.77 + 0.33
65.58 + 0.45
71.91 + 0.28
66.39 + 0.00
LipKAN
38.77 1.39
07.71 0.32
50.40+00.88
20.51 + 0.57
68.12 + 0.52
72:23 + 0.35
70.18 + 0.75
Increase
+12.83
+1.09
+4.39
+5.74
+2.54
+0.32
+3.79
Rational-KAN KAN
35.92 + 0.85
96.79 + 0.15
50/23 0 0.62
20.84 + 0.42
71.56 4: 0.38
71.63 + 0.31
65.97 + 0.55
LipRational KAN
40.67 + 1.12
97.01
0.25
51.12 0.78
22.79
0.51
0.43 74.98
0.33 72.42
69.12 +
Increase
+4.75
+0.22
+0.89
+1.55
+3,42
+0.79
+3.15
RBF-KAN
37.24 + 0..2
96.15 + 0.18
47.77 + 0.58
18.46 + 0.39
62.85 + 0.50
70.58 + 0.30
67.52 + 0.65
LAP
40.93 + 1.25
96.63 0.22
49.15 0.72
20.05 + 0.48
06.27 0.56
71.04 0.36
74.25 + 0.85
Increase
+3.69
+0.48
+1.38
+1.59
+3,42
+0.46
+6.73
6.2 Ablation Study
In this section, we perform ablation experiments to disentangle the contributions of the Lip layer and L..-regulaarizaionn We train original KANs (with default L1 regularization), KANs with L1.5- regularization, KANs with Lip Layers, and LipKANs on datasets MNIST, CIFAR1O, CIFAR100, and STL10, with training dynamics visualized in Figure Following prior practices, we consistently used A = for regularization and & = tanh as the Lipschitz function. Experimental results demonstrated that L1.5 regulaaaaaiiii played critical role in enhancing the generalization performance of Lip- KANs. Besides, L1.5 regularizazooo shows stronger benefits in complex datasets (e.g., STL-10) due to its ability to regularize high-dimensional weight interactions. Therefore, we strongly recommend adopting legg-aagulri instead of conventional -e-reuularization in future research.
To rigorously assess generalization, we follow prior works (Hoffer al. 2017 Fu al. 2023 and report three additional metrics in Table 2 Final Gap, Average Gap, and Overfitting Ratio. Notably, on multiple datasets such CIFAR-10 and MNIST, we observe that KANs + Lip Layers outperform
KANs + L1.5 s regularization on most metrics, suggesting that basis smoothness control is particularly effective low-noise, low-complexity regimes where architectural inductive bias is more beneficial than weight norm constraints. Conversely, on other datasets (e.g., CIFAR- 100, STL-10), the L1.-" variancre regimes where structured norm constraints are better suited to mitigating overfitting. Overall, while both Lip Layers and T,, regulariaiii aim to control model complexity, they mitigate training variance through different mechanisms, leading to distinct sensitivities to sample-level noise.
regularization yields greater improvements, indicating its strength

---

Interestingly, some models achieve comparable validation performance yet differ markedly in their generalization gaps. This phenomenon reflects differences in the underlying bias-variance trade-off: models with high training variance may overfit the training data, resulting in deceptively strong validation accuracy poor generalization. In contrast, methods like those in LipKANs introduce mild bias through regularization while substantially reducing variance, yielding more stable and reliable generalization across datasets. Finally, we analyze the role of activation family size G
(a) MNIST
(b) CIFAR-10
(c) STL-10
(d) CIFAR-100 100
Figure 1: Test accuracy across datasets: (a)-(d) show convergence dynamics of different archi- tectures four datasets. LipKANs (red diamonds) consistently achieve the highest accuracy with fastest convergence. Comparative analysis reveals 71.s-eegularization (green triangles) contributes more generalization improvement than Lipschitz layers (orange squares).
Table 2: Comprehensive ablation analysis of generalization across datasets. Combining both compo- nents (i.e., LipKANs) consistently yields the lowest generalization gap and best validation accuracy,
indicating their complementary effects.
Dataset
Model
Final Gap (%)
Avg. Gap (%)
Overfit Ratio (%)
Val. Acc (%)
KANs (La-Reeulatizationn)
6.04
5.64
13.13
46.01
KANs Lip Luyers
4.45
4.92
9.47
46.99
CIFAR-10
KANs L1.5 Regularization
4.99
5,28
10.11
49.37
LipKANs
3.16
4.89
6.26
50.40
KANs (I.1- Regglllaarization)
6.55
5.13
44.35
14.77
KANs + Lip Layers
5.10
38.98
14.98
CIFAR-100
KANs + L1.5 Re.gllgaaaiiztiinnn
3.76
5.09
19.89
LipKANs
2.32
5.03
11.31
20.51
KANs (Ly-Regularization)
5.52
4.92
5.77
95.62
KANs + Lip Layers
3.17
4.77
3.30
96.06
MNIST
KANs + L1.5- Vus-Regularization
5.13
4.81
5.27
97.26
LipKANs
2.55
4.70
2.60
97.71
KANs (Ly-Regularization)
6.93
5.27
26.72
25.94
KANs + Lip Layers
4.14
5.10
15.54
26.64
STL-10
KANs + L1.5- Reggulagaaiiziiinnn
3.95
5,06
10.47
37.73
LipKANs
3.31
4.98
8.53
38.77
Our analysis shows that the activation family size alone does not determine generalization; rather, the internal structure of the basis functions plays crucial role.
in Appendix D.4
7 Conclusion
In this paper, we establish the foundation for understanding the generalization mechanism of KANs and introduces LipKANs, a new network architecture designed improve generalization. We define Lipschitz complexity as the first measure to quantify the structural complexity of deep functions represented by KANs. Based on this measure, we derive complexity-driven generalization bounds, providing theoretical framework for analyzing KANs' generalization properties. Furthermore, we develop LipKANs, which reduce Lipschitz complexity and enhance the generalization mechanism of KANs. Both theoretical analysis and empirical results confirm that LipKANs achieve tighter generalization bounds and superior performance, paving the way for more effective and theoreti- cally grounded KANs variants. We strongly advocate replacing -eerellaaizotio in KANs with Ieus-regulaiizztiio in future research and hope our bounds establish theoretical foundations for understanding the generalization mechanism of KANs.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by the National Key Research and Development Program of China under Grant 2022YFB2703 100, the Joint Funds of the National Natural Science Foundation of China under Grant U22A2099, the National Natural Science Foundation of China under Grant 62376028, the Excellent Young Scientists Fund (Overseas) of the National Natural Science Foundation of China, and the National Key Scientific Instruments and Equipment Development Project under Grant 62427808.
References
Alireza Afial Aghaci. Fkan: Fractional kolmogorov-armold networks with trainable jacobi basis functions, arXiv:2406.07455 07456, 2024a.
Alireza Afzal Aghaci. Rkan: Rational Kolmogorov-arroo networks. arXiv:2406.14499 2024b.
Tashin Ahmed and Md Habibur Rahman Sifat. Graphkan: Graph kolmogorov arnold network for small molecule-protein interaction predictions. In ICML, 2024.
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In ICML, 2019.
Masoud Ataei, Mohammad Javad Khojasteh, and Vikas Dhiman. Darek - distance aware error for kolmogorov networks. (ti::2501.04757,, 2025.
Amit Attia and Tomer Koren. Uniform stability for first-order empirical risk minimization, In ICLR, 2022.
Basim Azam and Naveed Akhtar. Suitability of kans for computer vision: A preliminary investigation. arXiv:2406. 2007087700000087 2024.
Andrew R Barron and Jason M Klusowski. Complexity, statistical risk, and metric entropy of deep nets using total path variation. arXiv:1902.00800 2019.
Peter I. Bartlett and Shahar Mendelson. Rademacher and gaussian complexitie: Risk bounds and structural results. Journal of Machine Learning Research, 3(12):463-48, 2002.
Peter L. Bartlett, Olivier Bousquet, and Shahar Mendelson. Local rademacher complexities. The Annals of Statistics, 2005.
PeterL. Bartlett DylanJ. Foster, and Matus Telgarsky. Spectrally-nnommaiized margin bounds for neural networks. In NewrlPs 31, 2017.
Alexander Dylan Bodner, Antonio Santiago Tepsich, Jack Natan Spolski, and Santiago Pourteau. Convolutional kolmogorov-arood networks. arXiv:2406. 13155, 2024.
Zavareh Bozorgasl and Hao Chen.
Wav-kan:
Wavelet kolmogorov-arnold networks.
arXiv:2405. /2832, 2024.
Roman Bresson, Giannis Nikolentzos, George Panagopoulos, Michail Chatzianastasis, Jun Pang, and Michalis Vazirgiannis. Kagnns: Kolmogorov-arnold networks meet graph learning. arXiv:2406. 18380, 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, and Jared D Kaplan. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33(():1877-1901, 2020.
Gianluca De Carlo, Andrea Mastropietro, and Aris Anagnostopoulos. Kolmogorov-armold graph neural networks. arXiv:2406. 18354, 2024.
Minjong Cheon. Demonstrating the efficacy of kolmogorov-arnold networks in vision tasks. arXiv:2406. 140.19466..111. 2024.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215-223. JMLR Workshop and Conference Proceedings, 2011.

---

Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pages 4171-4186, 2019.
Ming-Wei
Lizhong Ding, Shizhong Liao, and Yong Liu. Randomized kernel selection with spectra of multilevel circulant matrices. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, pages 11711-1117 2018.
Lizhong Ding, Zhi Liu, Yu Li, Shizhong Liao, Yong Liu, Peng Yang, et al. Linear kernel tests via empirical likelihood for sigh-dimensional data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 4222 4229, 2019.
Lizhong Ding, Shizhong Liao, Yong Liu, Approximate selection via matrix approximation. IEEE Transactions on Neural Networks and Learning Systems, 31(11):1-11, 2020a. doi: 10.1109) 2019...2019295892..
Lizhong Ding, Zhi Liu, Yu Li, Shizhong Liao, al. Two generator game: Learning sample via linear goodness-of-fit test. In Advances in Neural Information Processing Systems (NeurIPS), 2020b.
Gintare Karolina Dziugaite, Alexandre Drouin, Brady Neal, Nitarshan Rajkumar, Ethan Caballero, Linbo Wang, loannis Mitliagkas, and Daniel M Roy. In search of robust measures of generalization. In Advances in Neural Information Processing Systems, 2020.
Deividas Eringis, John Leth, Zheng-Hua Tan, Rafael Wisniewski, and Mihály Petreczky. Pac-bayes generalisation bounds for dynamical systems including stable rnns. In AAAI, 2024.
Jingwen Fu, Zhizheng Zhang, Dacheng Yin, Yan Lu, and Nanning Zheng. Learning trajectorics are generalization indicators. Advances in Neural Information Processing Systems, 36:71053-71077, 2023.
Boris A. Galitsky. Kolmogorov-arool network for word-level explainable meaning representation. Preprints, 5 2024.
Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In /CML, 2016.
Hrayr Harutyunyan, Hrant Khachatrian, David C Kale, Greg Ver Steeg, and Aram Galstyan. Multitask learning and benchmarking with clinical time series data. Scientific data, 6(1):96, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image
recogmition. CVPR, 2016.
Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generaliza- tion gap in large batch training of neural networks. Advances in neural information processing systems, 30, 2017.
Amanda A Howard, Bruno Jacob, Sarah H Murphy, Alexander Heinlein, and Panos Stinis. Finite basis kolmogorov-armold networks: domain decomposition for data-driven and physics-informed problems. arXiv:2406.1966 2024.
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. 19X12:19/2.02178. 2019.
Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, freely accessible critical care database. Scientific data, 3(1):199, 2016.
Mchrdad Kiamari, Mohammad Kiamari, and Bhaskar Krishnamachari. Gkan: Graph kolmogorov- amoll networks. 2xXiv22406006777 2024.
Andre Nikolaevich Kolmogorov. On the representation of continuous functions of several variables by superpositions of continuous functions of a smaller number of variables. American Mathematical
Society, 1961.
12

---

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Akash Kundu, Aritra Sarkar, and Abhishek Sadhu. Kanqas: Kolmogorov- network for quantum architecture search. EPJ Quantum Technology, 11(1):76, 2024.
Yann LeCun, Corinna Cortes, and Christopher J.C. Burges. The MNIST database handwritten digits. http://wwn..cccccco//eeee///eeee
2010.
Chenxin Li, Xinyu Liu, Wuyang Li, Cheng Wang, Hengyu Liu, Yifan Liu, Zhen Chen, and Yix- uan Yuan. U-kan makes strong backbone for medical image segmentation and generation. arXiv:2406.02918, 2024.
Ziyao Li. Kolmogonov-arnold networks are radial basis function networks. arXiv:2405.06721/ 2024.
Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Chen, Peter Wu, Michelle A Lee, Yuke Zhu, et al. Multibench: Multiscale benchmarks for multimodal representation learning. Advances neural information processing systems, 2021(DB1):1, 2021.
Yong Liu, Lizhong Ding, and Shizhong Liao. Approximate kernel selection with strong approximate consistency. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 4230-4237, 2019.
Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruchle, James Halverson, Marin Soljacic, Thomas Y. Hou, and Max Tegmark. KAN: Kolmogorov-aroold networks. In ICLR, 2025.
Hya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprini 17Xiv:1111.05100 2017.
Mehryar Mohri.
Foundations of machine learning, 2018.
Wenlong Mou, Liwei Wang, Xiyu Zhai, and Kai Zheng. Generalization bounds of sgid for non-convex learning: Two theoretical viewpoints. In COLT, 2018.
Auwalu Saleh Mubarak, Zubaida Said Ameen, Sagiru Mati, Ayodele Lasisi, Quadri Noorulhasan Naveed, and Rabiu Aliyu Abdulkadir. Quasi-newton optimised kolmogorov- networks for wind farm power prediction. Heliyon, 10(23), 2024.
Brady Neal, Sarthak Mittal, Aristide Baratin, Vinayak Tantia, Matthew Scicluna, Simon Lacoste- Julien, and Ioannis Mitliagkas. A modern take on the bias- variance tradeoff in neural networks. arXiv: 1810.08591, 2018
Behnam Neyshabur. Implicit regularization in deep learning. arXiv preprint arXiv:1709.01953, 2017.
Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A pac-bayesian approach to spectrally- normalized margin bounds for neural networks. ICLR, 2017.
Ruichen Qiu, Yibo Miao, Shiwen Wang, Lijia Yu, Yifan Zhu, and Xiao-Shan Gao. Powermlp: An efficient version of kan. arXiv:2412. 13571, 2024.
Sarah Sachs, Tim van Erven, Liam Hodgkinson, Rajiv Khanna, and Umut Şimşekli. Generalization guarantees via gegorithm-dependeer rademacher complexity. In COLT, 2023.
Seyd Teymoor Seydi. Exploring the potential of polynomial basis functions in kolmogorov-arnol networks: A comparative study of different groups of polynomials. arXiv:2406.02555 2024.
Seyd Teymoor Seydi, Zavareh Bozorgasl, and Hao Chen. Unveiling the power of wavelets: A wavelet- based kolmogorov-amold network for hyperspectral image classification. arXiv:2406.07880 2024.
Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.
Khemraj Shukla, Juan Diego Toscano, Zhicheng Wang. Zongren Zou, and George Em Kamiadakis. A comprehensive and fair comparison between mlp and kan representations for differential equations and operator networks. arXiv:2406.02911 2024.
13

---

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting The journal of machine learning research, 15(1):1929-1958 2014.
Sidharth SS and Gokul R. Chebyshev polynomial-based kolmogorov-armlld networks: An efficient architecture for nonlinear function approximation. arXiv:2405.07200, 2024.
Sidharth SS, Keerthana AR, Gokul R, and Anas KP. Chebyshev polynomial-based kolmogorov-arnold
networks: An efficient architecture for nonlinear function approximation. arXiv:2405.07200 2024.
Hoang-Thang Ta. Bsrbf-kan: A combination of b-splines and radial basic functions in kolmogorov- arnold networks. arXivv2406.11172 11133, 2024.
Silviu-Marian Udrescu, Andrew Tan, Jiahai Feng, Orisvaldo Neto, Tailin Wu, and Max Tegmark. Ai feynman 2.0: Pareto-optimal symbolic regression exploiting graph modularity. Advances in Neural Information Processing Systems, 33:4860-4871, 2020.
Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Yixuan Wang. Jonathan W. Siegel, Ziming Liu, and Thomas Y. Hou. the expressiveness and spectral bias of KANs. In ICLR, 2025a.
Yizheng Wang, Jia Sun, Jinshuai Bai, Cosmin Anitescu, Mohammad Sadegh Eshaghi, Xiaoying Zhuang. Timon Rabezuk, and Yinghua Liu. colmogorov-aroold-inffee
neural network: A physics-informed deep learning framework for solving forward and inverse problems based on kolmogorov-arnold networks. Computer Methods Applied Mechanics and Engineering, 433: 117518, 2025b.
Jinfeng Xu, Zheyu Chen, Jinze Li, Shuo Yang, Wei Wang, Xiping Hu, and Edith C-H Ngai. Fourierkan- gef: Fourier kolmogorov-arnold network-an effective and efficient feature transformation for graph collaborative filtering. arXiv:2406.0/034 2024a.
Kunpeng Xu, Lifei Chen, and Shengrui Wang. Kolmogorov-arnold networks for time series: Bridging predictive power and interpretability. arXiv:2406.02499 2024b.
Xingyi Yang and Xinchao Wang. Kolmogorov-armold transformer. arXiv:2409.10544 2024.
Yahong Yang, Haizhao Yang, and Yang Xiang. Nearly optimal vc-dimension and bounds for deep neural network derivatives. Advances in Neural Information Processing Systems, 36(1):21721-21756 2023.
Runpeng Yu, Weihao Yu, and Xinehao Wang. Kan or mlp: A fairer comparison. arXiv:2407.16674 2024.
Alireza Zaeemzadeh, Nazanin Rahnavard, and Mubarak Shah. Norm-preservation: Why residual networks can become extremely deep? IEEE transactions on pattern analysis and machine intelligence, (3(11):3980-3900 2020.
Engin Zeydan, Cristian Vaca-Rubio, Luis Blanco, Roberto Pereira, Marius Caus, and Abdullah Aydeger. F-kans: Federated kolmogorov-amold networks. arXivv2407.20100, 2024.
Fan Zhang and Xin Zhang. Graphkan: Enhancing feature extraction with graph kolmogorov arnold networks. 2406v:2406. 13597, 2024.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015.
Xianyang Zhang and Huijuan Zhou Generalization bounds and model complexity for kolmogorov- arold networks.
2024.
Liangwewi Nathan Zheng, Wei Emma Zhang, Lin Yue, Miao Xu, Olaf Maennel, and Weitong Chen. Free-knots kolmogorov-arnold network: On the analysis of spline knots and advancing stability. arXiv:2501.09288. 2025.
Shrenik Zinage, Sudeepta Mondal, and Soumalya Sarkar. Dkl-kan: Scalable deep kernel learning using kolmogorov- amoll networks. arXiv:2407.21176 2024.
14

---

Supplementary Material
A
Auxiliary Lemmas
This section contains a collection of results that are needed in the proofs, most of which are classical theorems in statistical learning.
Lemma A. uses high-probability upper bound on the local Rademacher complexity (Brtlett et al. 2005 control the maximal deviation between empirical means and means of bounded function, typically a loss function.
Lemma A.1. [(Theorem 2.1 in Bartlett et al. 2005)) Let F be class of functions that map X into [a, b). Assume that there some r 2 such that for every f F, (Xr) S r. Then, for every O, with probability at least 1 - 2e- over the data S = (2,, Inn
1 1+ I- B
2rz +
1
1+a
sup (Pf- - (Pf- & inf fEF
R
F
V
+ (b 10-00 a)
+ +
aE(0.1)
n
a
2a - a)
(8)
We denote that Pf = Ex~r]f(X)] and Paf = R =supfeF R. Besides, we represent empirical Rademacher complexity as Eg R, E. { supper fzz)) where os are Rademacher random variables.
f(ai). For a class F, set 1
Corollary A.1. Let F be class of functions that map X into [-M, M). Assume that there is some
r 2 such that for every f E FF, Varls(X)] S r. Then for every > O, with probability at least
1 over the data S [x1, D2,,,22.....0
2r ()g(
32M log(2)
sup (Pf - Paf) S R. FEF
(9)
3n
Proof. It suffices to take = 2,, 2e and b 2M in Eq. 8
Lemma A.2 is slight variant of the standard Dudley entropy integral bound on the empirical Rademacher complexity.
Lemma A.2 (Lemma A.5 in Bartlett et al. 2017)). Let F be real-valued function class taking
values in [0, M] and assume that F. Then we have
4a Vn
12
M.a
R(F(X)) I inf
Vlog (F(X), 112)de
(10)
n
Lemma A.3 is attributed to Maurey, and applied bounding the covering number generally. Lemma A.3. In Hilbert space (H, || II), let
be given with the representation U En awW, where Wi E H and ai 2 O. Then, for any positive integer k, there exists choice of nonnegative integers (ka kN) such that Eis kg k and
|al|
||l||
N
||all max ||Vi
V
Σ 1=1
a |1
<
(11)
where a = {(a,....,a,, and ||a||| = Σn an
Proof. Denote 3 = ||a]], Let WI,..., ......,W. be i.i.d. r.v. such that P(W, = 3V,) = -4 Assume
that W E, W: then EW EW, Σ, 8V, 2 U.
15

---

On the other hand,
Wlll W
U
Ell
U
W:)
1.2
Σ I wall?
+ Σ
UU - Ws, U W
2
(#)
E
Σ 11U will3
E
1
[IIU
will
1
EE [[] - -113113) (12)
k22
K
3
E
[[]
Σ
B²
|
Σ
Vill
8² max |V
Therefore there exists realization (i1,... ( jk) E [I, d} such that We - 3V W- = E.W and
| W|| - E [|[ U| Finally we conclude our result by taking k; = 2; 1 (i==).
B Comparsion with Different Regularization Techniques
In this section discuss the theoretical implications of different regularization techniques, such as standard L2 penalties for ResNet50 [He et al.) 2016), L₁ penalities for KANs [Liu et al. 2025
dropout for BERT [Devlin et al. 2019 and L1.5 penalities in our work.
L Iy-regularization For MLPs, L1 regularization of linear weights is used to favor sparsity. In KANs, linear weights replaced by learnable activation functions, so Liu et al. (2025) define the L1 norm of an activation function ø to be its average magnitude over its N, input, 1.c.
N₂ N2
|0/ =
Σ
Φ (2((1)
Then for a KAN layer & with Rin inputs and noout outputs, Liu et al. 2025 define the L1 norm of Φb be the sum of Li norms of all activation functions, i.c.,
Thun
The total training objective Leotal is the prediction loss Spred plus L1 regularization of all KAN layers:
I-1
Liotal Lpred + A
Σ
1el1,
1=0
where controls overall regularization magnitude. The problem is the sparsification which is claimed to be critical KAN's interpretability. For efficiency, Sfficient-KAN"" instead replaces the L1 regularization on samples with the LI regularization on the weights, which is more common in neural networks and is compatible with the reformulation:
Lrotal = Epred + A
Will.
Ig-regularization L2 regularization promotes smooth optimization landscapes and improves generalization by penalizing large weights. The training objective is formulated
Lrotal = Lpree +
W 1l
1=0
16

---

Theoretically, Ly regularization interacts with the implicit regularization induced by residual connec- tions, further constraining the optimization path smoother, more stable solutions. This synergy ensures weight shrinkage toward zero while preserving gradient propagation through skip connections /Zaeemzadeh et al. 2020 Neyshabur 2017
Dropout Dropout SSrivastava et al. 2014 is popular and effective heuristic for preventing large neural networks from overfitting. Indeed, dropout improves the stability bounds generically (Hardt et al. 2016]. From the point of view of stochastic gradient descent, dropout is equivalent to setting a fraction of the gradient weights to zero. In our paper, regguaaaariiiii also improve the bounds.
I..-regularization
The design of -oss-lss originates from generalization bounds (Theorem [WTll2,,1 in Rwf. leading to an improved bias- bias-variance trade-off (Corollary
which regularize
C Supplementary Definition
C.1 Norms of Vectors, Matrices and Three-Dimensional Tensors
In this section, we define the norms of vectors (e-g.: T:J), matrices (e.g., the input matrix X) and three-dimensional tensors (e.g., the weight tensor W, of KANs). Let
p,r oc.
Given a vector a = (a1, 02 ........, am) )" E Rm, the L norm of a is ||o|]p = ( m a²)² Given a matrix = (a1, .2,.
E Rm where the i-th column of A is a, E R", we use represent the (p,r) norm of matrix A, denoted as |Alpp.-- Especially, the standard L2 induced norm and Lx induced norm of matrix A can be repre- sented respectively by s and ||-A]] co. Besides, use ||A||, to represent the standard spectral norm (also called L2 induced norm) of A. To define norms for three-dimensional tensors similarly the matrix (p, r) norms, we can generalize the structure systematically. Let A E mm be three-dimensional tensor. Its elements are denoted by Aijk- The tensor A can be thought of as a stack of matrices, where the k-th matrix slice is A (i.e., fixing the third index of the tensor). Let
|Allp.r [I(1|a pp: ||oz|lpp..... I|,,, Ilp))l,
1/9
1 S P.r.q S 0o, we define the (p, r, 2) norm of A as: ||Al|p...g = (Ei- 11
where
|A = (11A.1 11 1112all,...,,AAAnn 1, for each slice A..k E RRmxn.
C.2 Sub-Gaussian Distribution
Our analysis focuses on lub-Gaussian distribution P, so we introduce sub-Gaussian random variables and sub-Gaussian random vectors for preparation. Given a random variable æ, if there exists K 0 such that the tail of x satisfies TP [z] 2 f) S exx (-t² / Vt 2 0, then we call x a sub-Gaussian random variable, where the quantity K is named the sub-Gaussian variance proxy. The sub-Gaussian norm of ær denoted | defined ||a]|e> == inf { 0 E Texp (x² 1(2) < 2] It can be
deduced by Markov's inequality that there exists >>>> such that P([x) 2 1 S 2 exp (-e1//1<=113)
indicating that the tail decays slower as the sub-Gaussian norm becomes larger. A random vector Rn is called sub-Gaussian if the one-dimensional marginals (X, x) are sub-Gaussian random variables for all x E R". The sub- Gaussian norm of X is defined as || ll = supyes--1 [| 8) || In particular, a random vector with independent bounded coordinates is a sub-Gaussian random vector.
C.3 Lipschitz Norm
In this section, we discuss Lipschitz properties of various function family. Definition C.1 (Lipschitz functions). Let (X, dx) and dy,y)) be metric spaces. A function f X + Y is called Lipschitz if there exists L E IR, such that
dy dy(f(u) f()) & L (),
The infimum of all L in this definition is called the Lipschitz norm of f and is denoted || Ulip.
17

---

In other words, Lipschitz functions may not blow up distances between points too much. Lipschitz functions with I I 1 are usually called contractions. Specifically, the layer normalization ã (anh(a) is contraction with I; -i-- -"
Lemma C.1. If R + R is p-Lipschitz along every coordinate, then it is p-Lipschitz according 1pp for any p 2
Proof. For any æ, x' E R2.
1/8 1
1/p
(((z))o(<'))))
Σ (o(x)) -
Σ pe x -241 = =p
x'11
(13)
Lemma C.2. Let E 1- + +R| e [G]} be a function family. If a is P--Liscchizz then
My : IR + RdxG EEe (i= -Lipschitz according to || for any p 2 1, where the (i.
element of the matrix Mx:(1)) is given by fz(wz).
Proof. For any x, x' Rdd.
[| My(x')1pp Σ [o((x)) -o,(x))13
p | x x'1
pe
||0 - x' 11p = ||l1ll| x lle.
where p= (PI: (P......P)) PG)
Therefore, we define |E| = |Mallup for brevity. We lead to Definition 4.2 when we take = 2.
Now we prove Lemma E.1
Lemma E. IR R and / R R are both Lipschitz continuous functions with Lipschitz norms | and [p]] respectively, then the composition 1 o (x) is Lipschitz continuous with Lipschitz norm ||| o ol|Lip I |لplo Lip-
Proof. By the definition of Lipschitz continuttty and Lipschitz norm, for any x, E R
[((((z)) ((a(2')) < [Mlμ)lo(() - a(x')l
< || ir - -x'l
Thus, the composition o a satisfies the Lipschitz norm || a afluup S ||4 |upllo||up
Below, we discuss the Lipschitz property of various basis function families.
When Σ set of univariate B-spline basis functions with degree and knots (5.).. have P: < A where A max; max;(Ex+p {x). By default of [Liu et al. (2225,,
2p
p with grid size 5. To ensure the continuity and smoothness of the interpolation, it is usually necessary extend p points each end of the original data points. We get the knots &
[-2.2 -18, -1.4
1.0, -066, -
-0,2, 0.2, 0.6, 1.0, 1.4, 1.8,
with A 1.2 and Pk < 3p
See Figure for for the visualization of B-spline basis functions. Finally, the Lipschitz norm of Σ can be bounded as ||E||rap < 10
When Σ (cos(ka)) sin (kx(k))) le a set of Fourier basis functions Xu al. 2024a we have Pk S l based on Lemma To ensure the consistency of the number of parameters,
we set l [4] and deduce that Lip S 2v15.
18

---

S-ader 3-ptre Bess Fuectirs
Figure 2: Visualization of the 3-order B-spline basis functions.
Lemma C.3. cos(kx) and sin (ker) are k-Lipschitz continuous.
Proof. We only prove the case of the cosine function; the sine function is similar. For any I, x² E E.
k: + ky
kr
ky
(kos(kx)) cos(ky))
2 sin
sin
2
kx +
ky
kx
ky
2 sim
sin
2
2
ky
(14)
< 2
sim
2
fr
ky
< 2
2
ks yl
where the last incquality is due to the fact that
sin
kx - ku
Jox ky
When Σ = {Tx) is set of Chebyshev polynomial SS et al. 2024 which defined (x)) cos & ar)cos(x)) (please see Figure 3 for the visualization) and also be
expressed using the explicit polynomial form:
To(x) =1
Ti (z = I
T2(x): = 2x2-1
T3(x) 4x 3r
TT(x) = 2x (I) - T, Tn-2(x)) for 2 2,
we can calculate the Lipschitz norm directly due its polynomial nature. These polynomials are sequence of orthogonal functions. so only smaller number of basis functions are
needed. We set |Σ] like [SS et al. 2024 and in this case, |_||Lip I v14.
C.4
Einsum
To define the product between tensors, first introduce the definition of matrix inner product.
The inner product of two matrices A, E Rmxn is defined as the sum of the products of their corresponding entries. Mathematically, it is given by:
m7
(A,
ΣΣ A Bij.
Alternatively, this can be expressed using the trace of the product of A and B
(A, B) tr (AT
19

---

Chelystes Niyoomes the Fins Kine
-022
.039
-639
Figure 3: Visualization of the Chebyshev polynomials of the first kind.
Specifically, the inner product induces the Frobenius norm:
|||||= (A, A)
with the Cauchy-Schwarz inequality stated as
K4, B) s ||A|| ||
(15)
Below we define the Einstein summation operation(einsum) "o" between W Rm.. and R
as
(W E); Σ Σ W Wi E 1,..., einsum (1n, E); W.E):
k=
We also represent W X R" (W Σ); (W,, X) where W; Wt: R
D Supplementary Experiments
D.I Function Approximation
Datasets Feynman (Udrescu et al. 2020 is symbolic regression and function approximation dataset of physical equations collected from Feynman's textbook. We implement a function approx- imation task using the Feynman dataset, suggested by Liu al. 2025 investigate whether
LipKANs can learn better activation functions compared to KANs.
To demonstrate that LipKANs preserves the powerful expressive capacity of KANs, first compare the function approximation capabilities LipKANs with baselines. We utilize the function generation script provided by PYKAN [Liu et al., 2025 and assess model performance We exclude some equations since the function generation script returns NaN in input and inf in label present the RMSE of each model several
the Feynman dataset.
(such "Jackson 1..38 (Doppler)"). In Table 3
Table 3: RMSE Comparison between KANs (the first row of each cell) and LipKANs (the second row of each cell) on Feynman Dataset.
Feynman Eq.
Original Formula
MLP
KAN
Fourier-KAN
Rational-KA
RBF-KAN
Cheby-KAN
1.6.0a
0.04859
0.00147
0.02359
0.00219
0.00082
0.04359
0,00039
0.02884
0.00482
1.9.18
0.42783
0.4]967
0.54147
6.41817
0.42203
0.41933
0.41965
1.24.6
im (w² + wg)
0.04384
0.00877
0.10813
0.02019
0.01101
0.04750
0.00871
0.07405
0.03187
0.03415
0.4821
1.13.4
1m (0² 2 + w?)
0.23311
0.01264
0,79906
0.01203
0.5021
0,761
0.01187
0.17017
0.07661
0.00352
0.23197
0.17830
11.55.18
expl
Hepp
0.13784
0.06220
0.14590
0.08346
0.07837
0.17846
0.18712
0.42472
0.21685
0.19196
0.33203
1I1.4.3
0,27705
Cap
0.18774
0.29129
0,20695
0.21895
0.35953
examples, where the second row each cell represents the Lip version of the corresponding model. It is clear in Table 3 that the Lip version of KANs not only does not diminish the expressive power
20

---

but even outperforms in many functions. Among all the baselines, the original KAN achieves the best fitting performance, while Fourier-KAl exhibits the worst fitting performance. and RBF-KAN perform closely in RMSE. MIP is not comparable to the KAN-based architecture in the experiment, reflecting the strong function approximation ability of KAN-based methods.
Mitoonal-KNN
D.2 Generalization Gap
In this section, provide comprehensive analysis of generalization performance, follow the prior works [Hoffer et al.] 2017 Fu et al. 2023 and quantify generalization gaps using the following three metries:
Final Generalization Gap: Difference between training and validation accuracy at conver- gence.
Average Generalization Gap: Mean gap across the entire training trajectory.
Overfitting Ratio: Final gap normalized by final training accuracy.
The results are summarized below.
Table 4: Analysis of generalization performance using four metrics across datasets: LipKANs
achieves consistent advantages in generalization gaps and overfitting ratio.
Dataset
Model
Final Gap (%)
Avg. Gap (%)
Overfit Ratio (%)
Val. Ace (%)
KANs (e--Regularization)
6.04
5.64
13.13
KANs Lip Layers
4.45
4.92
9.47
46.99
CIFAR-10
KANs Re.-RRegularization
4.99
5.28
10.11
49.37
LipKANs
3.16
4.89
6.26
50.40
KANs (La-Regularization)
6.55
5.13
44.35
14.77
KANs + Lip Layers
5.10
38.98
14.98
CIFAR-100
KANs + Le.s-Regulaiization
3.76
5.09
19.89
19.93
LipKANs
2.32
5.03
11.31
20.51
KANs (Lz-Regulaiization)
5,52
4,92
5.77
95.62
KANs + Lip Layers
3.17
4.77
3.30
96.06
MNIST
KANs + Iegg-Regularizzaionn
5.13
4.81
5.27
97.26
LipKANs
2.55
4.70
2.60
97.71
KANs Regulaaaariztiinn KANs + Lip Layers
6.93
5,27
26.72
25.94
4.14
5.10
15.54
26.64
STL-10
KANs 11.5- le.s-Regularization
3.95
5.06
10.47
37.73
LipKANs
3.31
4.98
8.53
38.77
Across all datasets, the LipKANs consistently achieve the lowest overfitting ratio and final generaliza- tion gap, suggesting stronger generalization. Interestingly, this advantage does not always manifest significantly higher validation accuracy-some models achieve similar validation accuracy, yet differ notably in their generalization gaps. This discrepancy reflects a more favorable birs-variance trade-off. Specifically, models with higher variance may fit the training data well yet generalize poorly, resulting in larger generalization gaps despite acceptable validation accuracy. In contrast, regularization methods such as those employed in LipKANs may introduce slight bias but effectively reduce variance, leading to more reliable generalization behavior.
D.3
Computational Cost
We would like to clarify that both the Lip layer and the regulaaguliiization are lightweight mechanisms that introduce minimal computational burden. The Lip layer imposes Lipschitz norm constraints to control the smoothness of basis functions, without modifying the network architecture or introducing auxiliary parameters. The regulaaauaiinna is direct substitute for the commonly used L1 penalty-it operates on the same weight tensors already present training and does not incur additional memory overhead.
Empirically, Table 5 we quantify efficiency using two widely adopted metrics: training time (in seconds) and peak GPU memory usage (in MB). Our measurements show that both the Lipschitz
21

---

Table 5: Comprehensive ablation study on computational cost (training time and memory) across
datasets.
Model
CIFAR-10
CIFAR-100
MNIST
STI-10
Time
Memary (MB)
Time
Mernery (MR)
Time (S)
Memory (MB)
Time
Memory (MB)
KANs
263.96
53.52
265.45
54.61
227.72
26.44
221.57
356.28
KANs Lip Layers
268.88
53.52
259.99
54.61
226.14
26.44
227.49
356.40
KANs (L1.- Regguaaiiiziztioon)
262.73
53.52
283.43
54.63
227.00
26.57
221.62
356.63
LipKANs
264.89
53.52
263.72
54.63
229.04
26.57
226.76
356.65
layers and the L1.5 -egguaaauaiiiiii incur only moderate overhead relative to baseline KANs, while delivering notable improvements in generalization performance.
Table Model performance across datasets with different grid sizes and spline orders,
Datasset
Grid Size
Spline Order
#Params
Final Gen Cap
Test
Overfit Ratio
Train Time
Peak GPU Mem
1,87.008
1.42
50.42
2.81
196.96
39.57
1.84,25
0.79
49.60
1.59
201.30
42.17
1,781
212.61
L.781.004
4,00
$2.39
201.16
45.05
CIFAR-10
1,978,752
3.06
$2.33
5.85
210.33
48.03
2.176,000
2.30
52.15
4.41
221.63
50.50
2,579.248
5,78
$2.94
10.92
215.97
53.51
2,570,196
4.50
53.23
$.45
228.08
56.85
1,27,328
6630,336
1665
21.08
7.82
198 90
42.90
883,3444
1.52
21,00
7.24
211.00
45.90
.833,344
197.96
CIFAR-100
2,036,352
2.57
23.57
10.91
204.95
48.93
2,249 360
2,48
22.59
10.98
220.06
$1.49
2,442.468
24.12
16.78
214.33
2,645,376
3.92
21.04
16.31
227.14
58.02
357.408
97.28
4.24
66
224
97.59
3.58
26.35
84
459,010
3.14
97.54
3.23
29.00
54.18
459,040
3.56
97.33
3,66
31.87
54.18
809,866
55.52
560.672
2.89
97.45
2.97
35.92
56.86
$60.672
3,36
97.41
3.45
38.63
56.88
662,304
2.91
97.47
2.99
43.99
59.54
1.587.08
0.78
28.54
2.73
98.73
58
L,584,256
1,781 504
0.98
27.50
3,56
96.58
46.39
L7881504
1.45
32.41
4.47
93.69
46.37
STL-10
2,176.000
1.01
31,25
3.23
97.00
51.81
2,176.000
1.68
34,40
4.88
93.14
$1.79
2.573.248
1.95
33.91
5.75
97.27
54.07
D.4 Activation Family Size
The activation family size G reflects the cardinality of the basis function family, treated as a constant that captures overall family size, while its internal structure (e.g., smoothness) remains tunable. For instance, KANs can employ different basis KAN), and rational functions (Rational-Kee -KAn) each with adjustable basis affect generalization without altering G.
(original KAN), Fourier (Fourier-
From practical perspective, conducted sensitivity analyses on B-spline basis functions by varying degree and grid size in Table 6 which together determine the shape and cardinality of the activation family. The key findings
1. Internal structure matters beyond size: Even when G Grid size + Spline order fixed, generalization can vary significantly. On MNIST, both (3,4) and (5,2) yield the same G, yet the Final Gen Gap differs by over highigglighting that basis structure, not just size, plays crucial role.
2. Opposing effects of grid size vs. spline order: With fixed grid size, increasing spline order improves generalization; conversely, with fixed spline order, increasing grid size degrades
22

---

generalization. This aligns with our view that smoother basis functions (higher order) help regulate model complexity.
3. Efficiency trade-offs: Larger G increases computational cost, with spline order contributing more to time and memory overhead than grid size.
These results reinforce theoretical claims and demonstrate the framework's 's robustness across different architectural settings.
E
Proof of Main Results
E.1 Proof of Lemma 4.2
As outlined in the text, constructing a whole-network cover through induction on layers requires minimal assumptions about the norms imposed the weight matrices. In this subsection, delve into a more general analysis namely, given tensors W (W1,... I....... W,, v,,, define the mapping Fw as Fw(x) KANw(a) in Eq. Tb. More generally for i S L define Wi = (WI,.. W.) and
this approach. The structure of the networks the same as before;
Fwj (x) (W, N.. W1 M)x
with the convention Fa( For brevity, write Fw! (z) Fw: when data is fixed.
Lemma 4.2 Let (E1, (fI,..,,EL) EL) be given. Under Assumption I and 2, each og is Lipschitz continuous with Pk = || Lipp. Let p == (Pi....,PG) PG). Suppose the tensors W = (WI...., W,) lie within B1 ... x BL where B, arbitrary classes with the property that each W; E B, has
[ S cz. Then, letting 71 E1 and n EL c)llll2)e, the neural net images
Hx = Fioy (X): :f E Bi x ... x B1 have covering number bound:
N (Hx, 112) - II
N ((W..O((Ww--(1)
(Fm- W, E B
(W W Wi-1) ,,Ci-WEBB
Proof. Inductively construct covers F1, Fi...., Fr. of Wz,.... WT-1 as follows,
Choose an proper ci-cover Fw of (Fw₁. W E B.), thus
| M[[Fy W1 B1), 61, || (12) =: N1
For every element F E Fi, construct an 5441-covrr Gi+ (F) of (Wit1 E(F) W B.+1).
The covers are proper, so F = Fwf for some (W....., W:) E B1 x that
B, follows
(9++1 (F) & sup (W W.) W,,,
N ((W.H1Fw;(2Z) W E BBI1 G41111 llx42) =: No41
Construct the cover
Fi == (F1A.)
(16)
with cardinality
Fit1l $ |Fl No4 < i+1
I Nr.
Below, we show that Fit1 is an -ovvrr of ( Wi+ (W1 ... W.+1) E Bi x.. x B
For any Fw (W+41 ..... W1 E) by construction, find F E Fi such that
| Fw;ll <1==n =
23

---

Now suppose we can find F; Fs such that
[|F, Fw; 75
By the construction of Fi+1 Eq. (6). we can find Fi+1 E Fit1 such that
|1+11 W441 o D(F)ll2 S 6++1
Then we use the triangle inequality of norm || 112. we have
|1+t1 F Fwftill2
=I| Wi+1 E(F.) + Wi+1 E(E)) Fwillz l
F||F W.+1 E(F + ( E(F.) Fu
2441 [Weilb|EE(R Fwy)ll2
S441 G+1llpp127 Ti+1
where the second < is due Cauchy-Schwarz inequality in Eq. (15)
E.2 Proof of Lemma 4.3
The proof relies upon the Lemma A.3 which stated in terms of sparsifying convex hulls, and in its use here is inspired by covering number bounds for linear predictors.
Lemma 4.3 Let conjugate exponents (p, q) and (u, v) be given with p S 2 as well as positive reals
(a, 6, e) and positive integer m. Let data X be given with [ - b. Then
dd 2
In ((Wo E(X): W E xdxx | S 5a) 11 -112) -
In(2dd G),
where [x] returns the smallest integer greater than equal to
Proof.
Let data X E Rxd be given, with weight tensor W E Rdxaxx" For brevity, we write tensor
= -(x) E Rnxdxo with Enj [xx(x,)), and W [xx) E Raxd', Set k == a2b2,'d
We obtain tensor Σ Rax by rescaling the first dimension of tensor to have unit p-norm:
Eigk = E.y Eyjl
Define E RdxGxd to be rescaling tensor such that for each E [d"].
121 ||
[ا21 llp
O.
|E Ip
|
which satisfies Wa VoE= o W) E where @ denotes element product. Note that
||allp.n.- =
(d')
la.. Ilp.p
(d')
Σ
(d')
Σ
Eigk
= (d') |لارك
24

---

Define W = a O W, whereby using conjugacy of || Ipp...u and || llq.0.v gives
| < (a, W) - |alpa Wlgggg
S (d') EElpa
Now we have
d
d' Σ5
G
W WoE Σ
W ijh Eija oF
k
(17)
d G
-|
ΣΣΣ
Wijk
Ligh 3F
[
where Eijle E RdxGxxd is tensor where the element at position (i,j. k) is 1, and all other elements are O.
To derive the desired cover, we define
(Vi.... VN} EgEigh E g
[d.. [G], e [d]}
and construct cover
N
N
C
à
V : k; 2
0,
kis
Vig (i....., is) E [M
where k 's are integers and N := 2dd G. By construction, we have |C] s N and
|E lle E.gtll2
max [VVll2 S max
J.k
where the last inequality is due to p S.
To use Lemma A.3 We view V/s elements in the Hilbert space Raxd equipped with the inner product (A, = trace (AT A) defined in Appendix C.4 and norm |||]|.. Following Eq. (1
W Σ lies in the convex hull of (Vi...., Vy), i.c.
W E = W o E a- conv (V,,..., VN}
Combining the preceding constructions with Lemma A.3 there exist nonnegative integers (ki,..., ,N) with E, such that
à kg VI = W o
N
/ max 11l12 <
add' 2 Σソ < 2.
W o -
<
k
k
The desired cover element is thus E kvV, and the result follows.
E.3 Proof of Theorem
The whole-netword. covering bound now follows by the general norm covering number in Lemma and the matrix covering lemma Lemma 4.3
4.2
Theorem 4.4 Under Assumption 2, we can bound
R2
log(2d? G)
No N e, 11-112))
22
where :== max; d,
Proof. Given tensors Wi-1 =
:-( (WI.... Wi-1), we define
x 1--1 = Fwg-1(X = (W,- Eo... W1 X
25

---

Set 2,u = 00,0 = in Lemma 4.3 define B, = { E R4 1x6 [
S
c, W 12,,,11 S a. then we can decompose the whole covering number based on Lemma 4.2 log.V (Her TL, 112)
sup log N ((W, (X)) W, E B.) eis l12)
(Wi..,,,... Wi-1) vjei-WEB,
logN ({ [W [EFw; (IFm W. W, E(0) W, E B. 12,11-112 l111)
sup
WW W
(18)
VEE
log N ([W; (E(a't)) E(0)) W, E B.J.c..ll l1a)
sup (W W1 ,EE;
< 000 log (d
Below, we the Lipschitz property of E to bound the term |E(xl-1)- E(0))l2 as follows:
E(x'-1) - E(0)]l2 < |>||ppp
1=22
< 2 1[ W o E [ 1|12
<
> Lip
Σ
Lip
W |E('' [() E(0)]l2
Lip
W
(IE(('' E(0)]l2 + |E()))
>
tpp
W
(1 lo'"""11 2/22 + >11|>>
1-1 X||2
1-1
))
|
[
+
Σ
II |-|-
[ =lo
w
t=1
L-1
1-1
c
C
+
C'E
II
c
j=1
where | =
|1g l1' represents the oc norm of function family E. Now we suppose Σ no S E, then we have
Will S a. and
a))
H
[-1
1 - (0)12 < C'D cs
+
OhE
II c
(19)
i=1
j=1
Plugging Eq. 19 into Eq. 18
we obtain that
E x 1)
log.N N (Ha, TL, -ill-1 l1l) s
log (d,
e
(20)
C''
E
O [hi
cx) )² (2)(G)
Define
)
0 a
πI Cc
CDD 9
I
Σ
C'
II k=iーj
ck
s=i+1
j=1
j=1
and à DL ag For any > 0, set
G =
à |1
L
Ccÿ
26

---

Then we have 7s = = (Tlizita Ccs) e, and hence
a log (2dC)
log (Ha, e, Il l12) s
2²
E.4 Proof of Theorem 4.5
We prove our main result by integrating Theorem 4.4 with standard properties of Rademacher complexity in Section A
Theorem 4.5 (Generalization Bounds for KANs) Under Assumption and 3, let fixed Lipschitz activations Σ = fox k E [G]] and weight tensors Wi/- (W,, W Wr) } be given. Then for (ar y), (2), (,,),,..,, nn Un) drawn iid from distribution P, we have with probability greater than 1 1-6
144/[
4M2 log(2/c)
32M 12/c)/c)
R(Ewç) - R(FwF)) v
n |X [[X]]2Rwe max; (p(vx)
n
3n
1/c
00
log log(d (c) +
where ( - 12 log (2d2 G) max; p² (y1).
Proof.
Define the loss class L(S) ((C(f(21). ((f(x))) (.))))))))))))))))))))) 83 log(d2d (2d G) max, p² (y:) S
H). By theorem 4.4
we have
log N (C(S),,,11-1lb
2²
2
Lemma
A.2 implies that
4a
12
R (L(S))
< inf
a20
vn
4a
12v5
inf a>0
log log(M VT/(M)
Vn
n
12vC
12 12V(log(mMM) (3v()
+
n
24 (/1log((MM////)))
Using Lemma A.I with 2M we have with probability greater than 1-
14N/C1og(nM//3VV)))
E[C(f(r) S
Σ c (f())
n
4M log(2/e)
32M log(2/c)
a
3n
a (a>Vin(de))
Σ [((()),)))
n
for any f E H. which completes our proof.
E.5 Other Theoretical Results
Based on Assumption 1, which states that the distribution satisfies the sub-Gaussian property. we can bound || [|2 in Eq. (6) with high probability, leading to more applicable bounds in Corollary E.
27

---

Corollary E.1 (Sub-Gausee Gaussian Generalization Bounds for KANs). Under Assumption 1, 2 and 3, let fixed Lipschitz activations given. Then for (x, w) (01, (1),,,)..... (In, Un) drawn iid from sub- sab--aussian distribution IP, we have with probability greater than 6--
fox [G]} and weight tensor W (W,, 2..... W r. be
K KB max; (p(v)))
log (og(PG)) dlog T
1/e
R(Pw;) (R(wp) So
(21)
where was defined in Assumption 1.
Proof. We conclude the result immediately by plugging Eq. (2) into Eq. 6
We introduce Lemma E.I that ensures the Lipschitz property of the composition of Lipschitz functions. Lemma E.1. I R and R + are both Lipschitz continuous functions with Lipschitz norms |zop and [[[]]zip- respectively, then the composition (a:)) is Lipschitz continuous with Lipschitz norm ||e o Lip S [11pppplll
The proof of Lemma E.1 is provided in Appendix C.3
Lemma E.1 demonstrates that composing with a univariate function having a small Lipschitz norm reduces the Lipschitz norm of the activations family, thereby reducing the complexity of the functions represented by LipKANs.
Corollary 5.1 (Generalization Bounds for LipKANs) Under Assumption 2, and 3, let fixed Lipschitz activations Σ fox & E [G]), weight tensor W {W,, W2.... W, and
Lipschitz function / + IR with ||/]]zip I 1. Then for (æ: y), (æ1, (1),...... ..., (In, Un) drawn i.i.d. from sub-Gausee Gaussian distribution P. we have with probability greater than -
K KR ( max; (p(x))
1/e
R(FwF.) 1(FwF))
log( (og(ddo G) doo
n
n
where the right-hand hand side of the inequality is also bounded by the upper bound in Eq.
21
Proof. By Lemma E.
Rw
[le Eli Ilwill. d=1
Σ
W
T
222.1
2/3
Iuwalle)
| T
2/3 2,2,1
where the right hand side of the inequality represents the complexity of KANs, thereby proving the conclusion.
Limitations and Future Work
Our analysis currently uses the sub- Gaussian assumption control input tail behavior and derive generalization bounds via standard concentration tools. However, the framework is not inherently restricted aab-Gausiian inputs. principle, can extend to broader distribution Jlasses-such as sub-exponential or heavy-tailed distributions by leveraging tools from empirical process theory, including Bernstein-type inequalities, truncation methods, or Orlicz norm bounds. We view this as natural direction for future theoretical extensions. For visualization of complexity metrics and empirical study covering bounds, we leave these directions for future work, viewing them as promising avenues to further connect our theoretical analysis with practical model behavior.
28

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] pprovided proper justification is given (e.g., "error bars are not reported because would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly state the contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section 6, this paper analyzes some failure cases specifically and identifies when our approach is most beneficial.
29

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faireess
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Theoretical results are stated with explicit assumptions, and proof' are provided in Appendix E.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Full architecture details, hyperparameters, and training procedures are de- scribed.
Guidelines:
The answer NA means that the paper does not include experiments.
30

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
All datasets are released.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7 /ttppp.cc/pbllc/guudeeee http://oddeuubiiinoooooooooooooe
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
31

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer:
[Yes] Justification: Section and Section 6 include all the training and test details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported in our paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [NA]
Justification:
It's not the main focus of our paper.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
32

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The work adheres to ethical guidelines, including transparency in methodology and reproducibility.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Section 6 discusses potential positive/negative societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The work focuses on theoretical improvements to KANs; no high-risk models datascts released.
33

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Publicly available datasets are cited with licenses.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperssithcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: It's not the main focus this paper.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
34

---

Answer:
[NA]
Justification:
The work does not involve human subjects or crowdsourcing experiments.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: No human subjects research was conducted.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: LLMs were not used in the development of the core methodology.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non standardd components.
Please refer to LLM policy (https://neurip:: htt.://pps/pee/ooeeeeenn//////// for what should or should not be described.
35


Reviews:
[{"review_id": "08AP5wzt4y", "reviewer": "Reviewer_Zm6r", "summary": "This paper establishes generalization bounds for Kolmogorov-Arnold Networks (KANs), and proposes a new architecture for improving generalization, called LipKANs. Bounds for KANs, unlike MLPs, are complexity independent, and a mathematical measure, ‘Lipschitz complexity’, is introduced to represent the structural complexity of functions represented by KANs. Then, an architectural innovation is introduced where Lip layers are inserted in the KAN with explicit L1.5 regularization to bound network complexity. Finally, the proposed framework is evaluated on multiple benchmarks for vision, text, and multimodal data.", "questions": "1.\tIs the F1/accuracy score shown in Table 1 on a held-out test set? If you haven’t, can you specifically mention this somewhere? Also, an explicit quantification of the generalization gap would be useful.\n2.\tCould you provide confidence intervals for the output metrics shown in Table 1?\n3.\tFrom the ablation study, it seems that the L1.5 generalization is the biggest driver of improvements in generalization, and the Lip layers only help for MNIST. I wonder if this effect is just result of optimization/training splits? Can you show that LipKANs consistently show an improvement for MNIST? This for me is the biggest weakness in the paper, and any additional experiments to improve confidence in this would be useful.", "limitations": "NA", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1.\tThis paper explicitly addresses what is called the ‘approximation-estimation’ trade-off for KANs. This is important because naively reducing parameter count does not improve generalization in KANs proportionately, unlike MLPs. By representing the capacity of KANs in terms of properties of their activations, specifically in terms of the ‘Lipschitz complexity’, it no longer has a dependence on the number of parameters resulting in tighter bounds. \n2.\tWhile the paper provides theoretical arguments, this is also converted into a relatively simple, practical implementation for improving generalization in KANs.\n\nWeaknesses:\n1.\tThe results of the ablation study make me question the value of the Lip layers in improving generalization. Improvements seem to be mainly driven by the L1.5 regularization. I would want to see more experiments showing that Lip layers reliably contribute to generalization for MNIST and other datasets to be convinced that the proposed practical architecture genuinely improve generalization in line with theoretical results.", "was_revised": true, "final_justification": "Thank you to the authors for providing additional information. I and justification.\n\nI have updated my scores accordingly.", "rebuttal": "We would like to thank the reviewer for the careful reading and constructive suggestions. We respond to each point below.\n>**Q1**: Is the F1/accuracy score shown in Table 1 on a held-out test set? If you haven’t, can you specifically mention this somewhere? Also, an explicit quantification of the generalization gap would be useful.\n>\nThanks for your valuable suggestion, and we apologize for missing the details! We confirm that all reported accuracy and F1 scores in Table 1 are evaluated on held-out test sets using the officially predefined training/test splits for each dataset, consistent with the evaluation protocols established in prior work [1,2]. The test sets are strictly excluded from the training process, preventing any form of data leakage. To make this explicit, we have updated the caption of Table 1 and added a clarifying note in the Experiments section.\n\nIn addition, to provide a more comprehensive analysis of generalization performance, we follow the prior work [3,4] and quantify **generalization gaps** using the following three metrics:\n\n- **Final Generalization Gap**: Difference between training and validation accuracy at convergence.\n- **Average Generalization Gap**: Mean gap across the entire training trajectory.\n- **Overfitting Ratio**: Final gap normalized by final training accuracy.\n\nThe results are summarized below.\n\n| Dataset      | Model                              | Final Gap (%) | Avg. Gap (%) | Overfit Ratio (%) | Val. Acc (%) |\n|--------------|-------------------------------------|----------------|---------------|--------------------|---------------|\n| **CIFAR-10** | KANs ($L_1$-Regularization)         | 6.04           | 5.64          | 13.13              | 46.01         |\n|              | KANs + Lip Layers                  | 4.45           | 4.92          | 9.47               | 46.99         |\n|              | KANs + $L_{1.5}$-Regularization     | 4.99           | 5.28          | 10.11              | 49.37         |\n|              | **LipKANs**                        | **3.16**       | **4.89**      | **6.26**           | **50.40**     |\n| **CIFAR-100**| KANs ($L_1$-Regularization)         | 6.55           | 5.13          | 44.35              | 14.77         |\n|              | KANs + Lip Layers                  | 5.84           | 5.10          | 38.98              | 14.98         |\n|              | KANs + $L_{1.5}$-Regularization     | 3.76           | 5.09          | 19.89              | 19.93         |\n|              | **LipKANs**                        | **2.32**       | **5.03**      | **11.31**          | **20.51**     |\n| **MNIST**    | KANs ($L_1$-Regularization)         | 5.52           | 4.92          | 5.77               | 95.62         |\n|              | KANs + Lip Layers                  | 3.17           | 4.77          | 3.30               | 96.06         |\n|              | KANs + $L_{1.5}$-Regularization     | 5.13           | 4.81          | 5.27               | 97.26         |\n|              | **LipKANs**                        | **2.55**       | **4.70**      | **2.60**           | **97.71**     |\n| **STL-10**   | KANs ($L_1$-Regularization)         | 6.93           | 5.27          | 26.72              | 25.94         |\n|              | KANs + Lip Layers                  | 4.14           | 5.10          | 15.54              | 26.64         |\n|              | KANs + $L_{1.5}$-Regularization     | 3.95           | 5.06          | 10.47              | 37.73         |\n|              | **LipKANs**                        | **3.31**       | **4.98**      | **8.53**           | **38.77**     |\n\n\nAcross all datasets, the **LipKANs** consistently achieve the lowest overfitting ratio and final generalization gap, suggesting **stronger generalization**. Interestingly, this advantage does not always manifest as significantly higher validation accuracy—some models achieve similar validation accuracy, yet differ notably in their generalization gaps. This discrepancy reflects a more favorable **bias–variance trade-off**. Specifically, models with higher variance may fit the training data well yet generalize poorly, resulting in larger generalization gaps despite acceptable validation accuracy. In contrast, regularization methods such as those employed in LipKANs may introduce slight bias but effectively reduce variance, leading to more reliable generalization behavior.\n\nWe will include these clarifications and additional generalization metrics in the updated camera-ready version to improve transparency and completeness.\n\n---\n\n>**Q2**: Could you provide confidence intervals for the output metrics shown in Table 1?\n>\nWe appreciate your question and apologize for missing these important details. We have conducted five independent runs with different random seeds and now report the corresponding 95% confidence intervals for each validation accuracy. These intervals quantify the variability due to optimization randomness and demonstrate the **robustness and reproducibility** of our results. We have included these confidence intervals in Table 1 in the revised submission.\n| Model | SLT-10 | MNIST | CIFAR-10 | CIFAR-100 | AG News | AVMNIST | MIMIC-III |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| KAN | 25.94 ± 0.53 | 95.62 ± 0.08 | 46.01 ± 0.50 | 14.77 ± 0.33 | 65.58 ± 0.45 | 71.91 ± 0.28 | 66.39 ± 0.60 |\n| LipKAN | 38.77± 1.39 | 97.71 ± 0.32 | 50.40 ± 0.84 | 20.51 ± 0.57 | 68.12 ± 0.52 | 72.23 ± 0.35 | 70.18 ± 0.75 |\n| **Increase** | +12.83 | +1.09 | +4.39 | +5.74 | +2.54 | +0.32 | +3.79 |\n| Rational-KAN | 35.92 ± 0.85 | 96.79 ± 0.15 | 50.23 ± 0.62 | 20.84 ± 0.42 | 71.56 ± 0.38 | 71.63 ± 0.31 | 65.97 ± 0.55 |\n| LipRational-KAN | 40.67 ± 1.12 | 97.01 ± 0.25 | 51.12 ± 0.78 | 22.79 ± 0.51 | 74.98 ± 0.43 | 72.42 ± 0.33 | 69.12 ± 0.82 |\n| **Increase** | +4.75 | +0.22 | +0.89 | +1.95 | +3.42 | +0.79 | +3.15 |\n| RBF-KAN | 37.24 ± 0.92 | 96.15 ± 0.18 | 47.77 ± 0.58 | 18.46 ± 0.39 | 62.85 ± 0.50 | 70.58 ± 0.30 | 67.52 ± 0.65 |\n| LipRBF-KAN | 40.93 ± 1.25 | 96.63 ± 0.22 | 49.15 ± 0.72 | 20.05 ± 0.48 | 66.27 ± 0.56 | 71.04 ± 0.36 | 74.25 ± 0.88 |\n| **Increase** | +3.69 | +0.48 | +1.38 | +1.59 | +3.42 | +0.46 | +6.73 |\n\n---\n\n>**Q3**: From the ablation study, it seems that the L1.5 generalization is the biggest driver of improvements in generalization, and the Lip layers only help for MNIST. I wonder if this effect is just result of optimization/training splits? Can you show that LipKANs consistently show an improvement for MNIST? This for me is the biggest weakness in the paper, and any additional experiments to improve confidence in this would be useful. \n>\n>**W1**: The results of the ablation study make me question the value of the Lip layers in improving generalization. Improvements seem to be mainly driven by the L1.5 regularization. I would want to see more experiments showing that Lip layers reliably contribute to generalization for MNIST and other datasets to be convinced that the proposed practical architecture genuinely improve generalization in line with theoretical results.\n\nThanks for the great question. We would like to clarify that both components are principled and arise naturally from our formulation of Lipschitz complexity (**Definition 4.3**). Specifically, the Lip Layer constrains the smoothness of the basis family through the Lipschitz norm of $\\Sigma$, while the $L_{1.5}$-regularization penalizes the structured (2,2,1) norm of the linear weight tensors $\\mathbf W$. These two components regulate **orthogonal axes** of the Lipschitz complexity and hence play **complementary roles** in promoting generalization.\n\nTo rigorously assess generalization beyond final validation accuracy, we report three additional metrics in the Table of Q1: Final Gap, Average Gap, and Overfitting Ratio. Notably, on multiple datasets such as CIFAR-10 and MNIST, we observe that **KANs + Lip Layers** outperform **KANs + $L_{1.5}$-regularization** on most metrics, suggesting that basis smoothness control is particularly effective in low-noise, low-complexity regimes where architectural inductive bias is more beneficial than weight norm constraints. Conversely, on other datasets (e.g., CIFAR-100, STL-10), the **$L_{1.5}$-regularization** yields greater improvements, indicating its strength in high-variance regimes where structured norm constraints are better suited to mitigating overfitting. Overall, while both Lip Layers and $L_{1.5}$-regularization aim to control model complexity, they mitigate training variance through different mechanisms, leading to distinct sensitivities to sample-level noise.\n\nFinally, to ensure that the reported gains are not due to specific training splits or optimization variance, all experiments are **averaged over 5-fold cross-validation and multiple random seeds**, ensuring statistical reliability and reproducibility. We will include these clarification tables and extended results in the camera-ready version to fully address the reviewer’s concern.\n\n## References\n[1] Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruehle, James Halverson, Marin Soljacic, Thomas Y. Hou, and Max Tegmark. KAN: Kolmogorov–arnold networks. In ICLR, 2025.\n\n[2] Liangwewi Nathan Zheng, Wei Emma Zhang, Lin Yue, Miao Xu, Olaf Maennel, and Weitong Chen. Free-knots kolmogorov-arnold network: On the analysis of spline knots and advancing stability. arXiv:2501.09283, 2025.\n\n[3] Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. In NeurIPS, 2017.\n\n[4] Jingwen Fu, Zhizheng Zhang, Dacheng Yin, Yan Lu, and Nanning Zheng. Learning trajectories are generalization indicators. In NeurIPS, 2023."}, {"review_id": "sNj3DwBnW9", "reviewer": "Reviewer_kpEt", "summary": "This paper points out that existing research on KANs generalization error mainly focuses on approximation error, but ignores estimation error, resulting in a poor bias-variance trade-off and poor generalization performance. Therefore, this paper introduces Lipschitz complexity as a novel structural complexity measure for Kolmogorov-Arnold Networks (KANs), derives generalization bounds based on this measure. Based on Lipschitz complexity, this paper proposes LipKANs, which enhanced architecture with Lipschitz layers and L1.5 regularization to improve generalization.", "questions": "- From the ablation study results in Figure 1, the main performance improvement comes from the L_{1.5}-regularization. Does this mean that it is not important to keep low Lipschitz complexity using Lip layer in practical applications?\n- This paper starts with the lack of KAN's generalization, but only verifies the performance on several small-scaled datasets. Discussing the transfer learning performance of KAN (for example, evaluating the performance of a model trained on the MNIST dataset on CIFAR-10) should better verify the generalization ability of the model.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- The paper provides a comprehensive theoretical foundation, including formal definitions of Lipschitz complexity (Definition 4.3), covering number bounds (Theorem 4.4), and generalization bounds (Theorem 4.5). Proofs are detailed in the appendix.\n- Experiments across vision (MNIST, CIFAR), text (AG News), and multimodal (MIMIC-III) tasks demonstrate consistent improvements in generalization for LipKANs over baselines (Table 1). Ablation studies (Figure 1) disentangle the contributions of Lipschitz layers vs. L1.5 regularization.\n- The paper is well-organized, with clear sections for reformulation (Sec 4.1), theory (Sec 4.3), and architecture (Sec 5). Figures and tables effectively summarize results.\n- The core theoretical contribution—defining Lipschitz complexity for deep compositional functions—is novel and distinct from existing measures (e.g., VC dimension, Rademacher complexity).\n\nWeaknesses\n\n- While experiments cover multiple domains, the gains on simpler datasets (e.g., MNIST, CIFAR-10, AVMNIST) are marginal (<1% in Table 1), suggesting limited utility for low-complexity tasks.\n- The cost of Lipschitz layers and L1.5 regularization is not analyzed, leaving practical efficiency unclear.", "was_revised": true, "final_justification": "This paper proposes LipKANs, which incorporate Lipschitz layers and L1.5 regularization to improve the generalization of KANs. The rebuttal provides good answers to the issues, thus I raise the score.", "rebuttal": "We thank the reviewer for the constructive suggestions and will further revise the manuscript accordingly.\n>**W1**: While experiments cover multiple domains, the gains on simpler datasets (e.g., MNIST, CIFAR-10, AVMNIST) are marginal (<1% in Table 1), suggesting limited utility for low-complexity tasks. \n>\n>**Q1**: From the ablation study results in Figure 1, the main performance improvement comes from the L_{1.5}-regularization. Does this mean that it is not important to keep low Lipschitz complexity using Lip layer in practical applications?\n>\nThanks for the insightful question. We confirm that the Lip Layer remains an important and complementary component for controlling generalization. Both mechanisms stem from our unified **Lipschitz complexity** definition (**Definition 4.3**), but target different terms: the **Lip Layer** directly constrains the smoothness of the basis functions $\\|\\Sigma\\|_{\\mathrm{Lip}}$, while the **$L_{1.5}$-regularization** controls the structured (2,2,1) norm of the weight tensors $\\mathbf W$. These two components regulate **orthogonal axes** of the Lipschitz complexity, thereby acting in a **complementary fashion** to improve generalization. \n\nEmpirically, to provide a more comprehensive analysis of generalization performance, we report three additional generalization metrics:\n\n- **Final Generalization Gap**: Difference between training and validation accuracy at convergence  \n- **Average Generalization Gap**: Mean gap across the entire training trajectory  \n- **Overfitting Ratio**: Final gap normalized by final training accuracy  \n\nThe results are summarized below. We observe that the Lip Layer consistently reduces the generalization gap and overfitting ratio across multiple datasets, particularly on MNIST and CIFAR-10, where the basis smoothness plays a more critical role than weight sparsity. This supports the practical value of maintaining **low Lipschitz complexity** via basis smoothing. Overall, while both Lip Layers and $L_{1.5}$-regularization are designed to reduce variance, they operate via distinct mechanisms—basis-level smoothing versus structured weight penalties—resulting in different sensitivities to sample noise.\n\nMoreover, as shown in our full ablation table, combining both components (i.e., **LipKANs**) consistently yields the **lowest generalization gap** and **best validation accuracy**, indicating their **complementary effects**. Interestingly, this advantage does not always translate into significantly higher validation accuracy—some models achieve comparable validation performance yet differ markedly in their generalization gaps. This phenomenon reflects differences in the underlying **bias–variance trade-off**: models with high training variance may overfit the training data, resulting in deceptively strong validation accuracy but poor generalization. In contrast, methods like those in LipKANs introduce mild bias through regularization while substantially reducing variance, yielding more stable and reliable generalization across datasets. \n\nWe will clarify this intuition and add further discussion in the camera-ready version.\n| Dataset      | Model                              | Final Gap (%) | Avg. Gap (%) | Overfit Ratio (%) | Val. Acc (%) |\n|--------------|-------------------------------------|----------------|---------------|--------------------|---------------|\n| **CIFAR-10** | KANs ($L_1$-Regularization)         | 6.04           | 5.64          | 13.13              | 46.01         |\n|              | KANs + Lip Layers                  | 4.45           | 4.92          | 9.47               | 46.99         |\n|              | KANs + $L_{1.5}$-Regularization     | 4.99           | 5.28          | 10.11              | 49.37         |\n|              | **LipKANs**                        | **3.16**       | **4.89**      | **6.26**           | **50.40**     |\n| **CIFAR-100**| KANs ($L_1$-Regularization)         | 6.55           | 5.13          | 44.35              | 14.77         |\n|              | KANs + Lip Layers                  | 5.84           | 5.10          | 38.98              | 14.98         |\n|              | KANs + $L_{1.5}$-Regularization     | 3.76           | 5.09          | 19.89              | 19.93         |\n|              | **LipKANs**                        | **2.32**       | **5.03**      | **11.31**          | **20.51**     |\n| **MNIST**    | KANs ($L_1$-Regularization)         | 5.52           | 4.92          | 5.77               | 95.62         |\n|              | KANs + Lip Layers                  | 3.17           | 4.77          | 3.30               | 96.06         |\n|              | KANs + $L_{1.5}$-Regularization     | 5.13           | 4.81          | 5.27               | 97.26         |\n|              | **LipKANs**                        | **2.55**       | **4.70**      | **2.60**           | **97.71**     |\n| **STL-10**   | KANs ($L_1$-Regularization)         | 6.93           | 5.27          | 26.72              | 25.94         |\n|              | KANs + Lip Layers                  | 4.14           | 5.10          | 15.54              | 26.64         |\n|              | KANs + $L_{1.5}$-Regularization     | 3.95           | 5.06          | 10.47              | 37.73         |\n|              | **LipKANs**                        | **3.31**       | **4.98**      | **8.53**           | **38.77**     |\n\n---\n\n>**W2**: The cost of Lipschitz layers and L1.5 regularization is not analyzed, leaving practical efficiency unclear.\n>\nThanks for the constructive suggestion. We would like to clarify that both the **Lip layer** and the **$L_{1.5}$-regularization** are lightweight mechanisms that introduce minimal computational burden. The **Lip layer** imposes Lipschitz norm constraints to control the smoothness of basis functions, without modifying the network architecture or introducing auxiliary parameters. The **$L_{1.5}$-regularization** is a direct substitute for the commonly used $L_1$ penalty—it operates on the same weight tensors already present in training and does not incur additional memory overhead.\n\nEmpirically, we quantify efficiency using two widely adopted metrics: **training time (in seconds)** and **peak GPU memory usage (in MB)**. Our measurements show that both the Lipschitz layers and the $L_{1.5}$-regularization incur only moderate overhead relative to baseline KANs, while delivering notable improvements in generalization performance.\n| Dataset | CIFAR-10 |  | CIFAR-100 |  | MNIST |  | STL-10 |  |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Model | Time(s) | Memory(MB) | Time(s) | Memory(MB) | Time(s) | Memory(MB) | Time(s) | Memory(MB) |\n| KANs ($L_1$-Regularization) | 263.96 | 53.52 | 265.45 | 54.61 | 227.72 | 26.44 | 221.57 | 356.28 | \n| KANs+Lip Layers | 268.88 | 53.52 | 259.99 | 54.61 | 226.14 | 26.44 | 227.49 | 356.40 | \n| KANs+$L_{1.5}$-Regularization | 262.73 | 53.52 | 283.43 | 54.63 | 227.00 | 26.57 | 221.62 | 356.63 | \n| LipKANs | 264.89 | 53.52 | 263.72 | 54.63 | 229.04 | 26.57 | 226.76 | 356.65 | 820.66 | \n\n---\n\n>**Q2**: This paper starts with the lack of KAN's generalization, but only verifies the performance on several small-scaled datasets. Discussing the transfer learning performance of KAN (for example, evaluating the performance of a model trained on the MNIST dataset on CIFAR-10) should better verify the generalization ability of the model.\n>\nThanks for the good suggestion! While our current framework does not address out-of-distribution (OOD) scenarios, investigating LipKANs’ behavior under distribution shifts is an interesting direction for further work. We would like to clarify that our theoretical and empirical analyses in the current draft focus on in-distribution generalization under the **standard i.i.d. setting** (as formalized in **Section 3**), which is commonly adopted in statistical learning theory. Specifically, our generalization bounds (e.g., **Theorem 4.5** and **Corollary 5.1**) quantify the expected risk gap between the training and test distributions drawn from the **same underlying data distribution**, with controlled model complexity via **Lipschitz complexity**. This notion of generalization differs from OOD generalization or transfer learning, which instead assesses the model’s ability to adapt or perform on a different target domain. As a preliminary exploration, we conducted transfer experiments from MNIST to CIFAR-10, but both KANs and LipKANs achieved relatively low target-domain accuracies (19.37% and 23.28%, respectively). Returning to the in-distribution setting, we have evaluated our models across multiple datasets as reported in Table 1, where LipKANs consistently yield higher validation accuracy than the original KANs, indicating enhanced generalization performance under standard i.i.d. settings."}, {"review_id": "PqKQUulIo6", "reviewer": "Reviewer_yRDY", "summary": "The paper investigates the generalization mechanisms of Kolmogorov-Arnold Networks (KANs), and, building on this understanding, introduces LipKANs that is more efficient owing to lower model complexity and yields better generalization. For the methodology, they derive a uniform high-probability generalization bound and introduce the concept off Lipschitz complexity accordingly. It is shown that the generalization bound with high probability for KANs is proportional to the Lipschitz complexity proposed. Lipschitz complexity is then regarded as a measure of structural complexity of a KAN model. LipKANs leverages this insights by encouraging the training process to be favor low-complexity models, replacing the conventional L1-regularized loss with the proposed Lipschitz complexity. Through extensive experiments, it demonstrates enhanced generalization performance compared to other variants of KAN and highlights the critical role of the Lipschitz through ablation study.", "questions": "1. Is it common that assumption 2 holds?\n2. Is it possible that we take the relation among weights into consideration so as to refine Lipschitz complexity?\n3. The theory is based on sub-Gaussian distribution. Can it extend to other distribution?", "limitations": "1.The theory relies on sub-Gaussian distribution assumption.\n2. Although the paper presents strong empirical results, it does not deeply discuss potential discrepancies between theoretical bounds and observed performance.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strength:\n1. Technically Solid Methodology:\nThe paper is technically rigorous, offering a comprehensive and logical flow that articulates the significance of generalization bounds in deep learning. The authors clearly identify the gap in understanding the generalization behavior of Kolmogorov-Arnold Networks (KANs), then systematically derive a high-probability generalization bound for KANs. The rationale for introducing Lipschitz complexity as a meaningful measure of model complexity is convincingly argued, connecting the theoretical analysis to practical implications.\n\n2. Well-Motivated Introduction of LipKANs:\nBuilding on the theoretical insights, the transition to proposing LipKANs is natural and well-justified. The authors make a strong case for why optimizing over Lipschitz complexity can lead to better generalization, and they effectively motivate the replacement of standard L1 regularization with their new complexity measure.\n\n3. Thorough and Consistent Experimental Validation:\nThe experimental section is comprehensive, covering various benchmarks and scenarios. The results consistently demonstrate the superior performance of LipKANs compared to baseline models, and the ablation studies effectively highlight the crucial role of Lipschitz complexity in improving generalization.\n\n4. Clarity in Theoretical Development:\nAlthough detailed proofs are relegated to the appendix, the main paper offers clear explanations of the core ideas and proof techniques, allowing readers to understand the theoretical underpinnings without being bogged down in technicalities. This clarity strengthens the paper’s overall accessibility.\n\n\nWeakness:\n1. Loose Generalization Bound:\nWhile the proposed generalization bound successfully captures key factors influencing KANs’ generalization, it does not exploit dependencies or relationships among different weight matrices. As a result, the bound can be quite loose. For instance, for two KAN models where each weight matrix has the same norm, the calculated complexity—and thus the bound—will be identical, regardless of the relative directions of the matrices. However, the actual Lipschitz constant, and thus the true generalization capability, can be significantly affected by these directional relationships. As a consequence, the proposed bound may fail to distinguish between models with genuinely different complexities, potentially affecting model selection.\n\n2. Insufficient Discussion of Bias-Variance Trade-Off:\nThe paper mentions the bias-variance trade-off, but does not clearly explain how it relates to their results or the broader context of estimation versus generalization. Clarifying this relationship could help readers understand how LipKANs balance model expressiveness and generalization, and why the trade-off is enhanced compared to the conventional method.\n\n3. (minor) It is better to move definition 4.2 before assumption 2 as the definition is mentioned in assumption.", "was_revised": true, "final_justification": "I really enjoyed reading this paper and honestly couldn’t find any flaws.", "rebuttal": "We thank the reviewer for raising important points, and we will incorporate the corresponding revisions accordingly.\n>**W1**:  Loose Generalization Bound: While the proposed generalization bound successfully captures key factors influencing KANs’ generalization, it does not exploit dependencies or relationships among different weight matrices. As a result, the bound can be quite loose. For instance, for two KAN models where each weight matrix has the same norm, the calculated complexity—and thus the bound—will be identical, regardless of the relative directions of the matrices. However, the actual Lipschitz constant, and thus the true generalization capability, can be significantly affected by these directional relationships. As a consequence, the proposed bound may fail to distinguish between models with genuinely different complexities, potentially affecting model selection.\n>\n>**Q2**: Is it possible that we take the relation among weights into consideration so as to refine Lipschitz complexity? \n> \n\nThanks for highlighting this limitation. While our current generalization bound does not make inter-layer dependencies explicit, it already captures layer-wise alignment through the operator norm products $\\prod_{i=1}^{L}\\|\\mathbf W_i\\|_\\sigma$, which originate from the covering number bound of the entire network in **Lemma 4.2**. This implicitly reflects structural relationships across layers. We agree that making these dependencies more explicit could tighten the bound, and we consider this a valuable direction for future refinement.\n\n---\n\n>**W2**: Insufficient Discussion of Bias-Variance Trade-Off: The paper mentions the bias-variance trade-off, but does not clearly explain how it relates to their results or the broader context of estimation versus generalization. Clarifying this relationship could help readers understand how LipKANs balance model expressiveness and generalization, and why the trade-off is enhanced compared to the conventional method.\n>\nThanks for raising this insightful point. To clarify the bias-variance trade-off in our setting, we revisit the generalization bounds in **Theorem 4.5**. The Lipschitz complexity term reveals that the model **variance**—its sensitivity to training data perturbations—is closely related to the structured $(2,2,1)$-norm of the weight tensors, which controls the overall magnitude of the spline coefficients. A larger norm enables greater flexibility in fitting the data but also increases the risk of overfitting, thereby contributing to higher variance. In contrast, the **bias** stems from the approximation capacity of the basis functions (e.g., B-spline, Fourier, and Chebyshev basis). These basis functions provide strong approximation capacity for smooth target functions, enabling the learned model to closely align with the underlying data distribution when appropriately regularized. To this end, the proposed $L_{1.5}$-regularization penalizes the structured norm, reducing variance while preserving expressivity—thereby encouraging a favorable **bias-variance trade-off** and enhancing generalization. \n\nFurthermore, this trade-off aligns closely with the decomposition of uncertainty. The complexity-dependent term reflects epistemic uncertainty, which encompasses both bias and variance as it arises from limited data and model capacity, and can be reduced by regularization or more training data. In contrast, the second term, $\\sqrt{{1}/{n\\epsilon}}$, derived from statistical concentration inequalities, quantifies aleatoric uncertainty, reflecting intrinsic randomness in the data that persists regardless of the model's capacity. In this way, our generalization bounds—interpreted via uncertainty decomposition—offer a complementary perspective on the bias-variance trade-off: epistemic uncertainty encompasses both bias and variance and can be effectively reduced via $L_{1.5}$-regularization, while aleatoric uncertainty sets the irreducible limit. Our method improves generalization by explicitly targeting this trade-off in a principled manner.\n\nEmpirically, we validate this in the following experiments, where LipKANs consistently show lower overfitting ratios and generalization gaps than KANs with standard $L_{1}$-regularization. This confirms that our design not only aligns with the theory but also improves generalization in practice by facilitating a better bias-variance trade-off.\n\nWe will clarify this connection in the revised version to improve the presentation.\n| Dataset      | Model                              | Final Gap (%) | Avg. Gap (%) | Overfit Ratio (%) | Val. Acc (%) |\n|--------------|-------------------------------------|----------------|---------------|--------------------|---------------|\n| **CIFAR-10** | KANs ($L_1$-Regularization)         | 6.04           | 5.64          | 13.13              | 46.01         |\n|              | KANs + Lip Layers                  | 4.45           | 4.92          | 9.47               | 46.99         |\n|              | KANs + $L_{1.5}$-Regularization     | 4.99           | 5.28          | 10.11              | 49.37         |\n|              | **LipKANs**                        | **3.16**       | **4.89**      | **6.26**           | **50.40**     |\n| **CIFAR-100**| KANs ($L_1$-Regularization)         | 6.55           | 5.13          | 44.35              | 14.77         |\n|              | KANs + Lip Layers                  | 5.84           | 5.10          | 38.98              | 14.98         |\n|              | KANs + $L_{1.5}$-Regularization     | 3.76           | 5.09          | 19.89              | 19.93         |\n|              | **LipKANs**                        | **2.32**       | **5.03**      | **11.31**          | **20.51**     |\n| **MNIST**    | KANs ($L_1$-Regularization)         | 5.52           | 4.92          | 5.77               | 95.62         |\n|              | KANs + Lip Layers                  | 3.17           | 4.77          | 3.30               | 96.06         |\n|              | KANs + $L_{1.5}$-Regularization     | 5.13           | 4.81          | 5.27               | 97.26         |\n|              | **LipKANs**                        | **2.55**       | **4.70**      | **2.60**           | **97.71**     |\n| **STL-10**   | KANs ($L_1$-Regularization)         | 6.93           | 5.27          | 26.72              | 25.94         |\n|              | KANs + Lip Layers                  | 4.14           | 5.10          | 15.54              | 26.64         |\n|              | KANs + $L_{1.5}$-Regularization     | 3.95           | 5.06          | 10.47              | 37.73         |\n|              | **LipKANs**                        | **3.31**       | **4.98**      | **8.53**           | **38.77**     |\n\n---\n\n>**W3**: It is better to move definition 4.2 before assumption 2 as the definition is mentioned in assumption.  \n\nSincerely thanks for the thoughtful suggestion. We will make this revision in the camera-ready version to enhance readability.\n\n---\n\n>**Q1**: Is it common that assumption 2 holds?  \n\nYes, as discussed in **Appendix C.3**, we provide concrete examples to support Assumption 2. For instance:  \n- B-spline basis (degree 3, grid size 5): $|\\Sigma|_{\\text{Lip}} \\leq 10\\sqrt{2}$;\n- Fourier basis (frequency 4): $|\\Sigma|_{\\text{Lip}} \\leq 2\\sqrt{15}$;\n- Chebyshev polynomial basis (order 4): $|\\Sigma|_{\\text{Lip}} \\leq \\sqrt{14}$.  \n\nThese values confirm that the Lipschitz constants remain bounded in practical basis families, validating the assumption.\n\n---\n\n>**Q3**: The theory is based on sub-Gaussian distribution. Can it extend to other distribution?  \n\nYes, and thanks for the thoughtful question. Our analysis currently uses the **sub-Gaussian assumption** to control input tail behavior and derive generalization bounds via standard concentration tools.  However, the framework is not inherently restricted to sub-Gaussian inputs. In principle, it can extend to broader distribution classes—such as sub-exponential or heavy-tailed distributions—by leveraging tools from **empirical process theory**, including Bernstein-type inequalities, truncation methods, or Orlicz norm bounds. We view this as a natural direction for future theoretical extensions."}, {"review_id": "1vT7BOOSyR", "reviewer": "Reviewer_9hVp", "summary": "This paper investigates the generalization properties of Kolmogorov-Arnold Networks (KANs), a recently proposed architecture known for strong expressiveness in symbolic and scientific learning tasks. The authors address a critical gap in prior analyses, which largely focused on approximation error, by rigorously formulating generalization bounds for KANs through a novel complexity measure—Lipschitz complexity. Building on these theoretical insights, they introduce LipKANs, a variant that integrates Lipschitz-bounded layers and L1.5-regularization, leading to provably tighter generalization bounds and improved empirical performance on benchmark datasets.", "questions": "Questions for the Authors\n\nRole of Activation Family Cardinality:\nWhile the bounds depend only logarithmically on the activation family size 𝐺, in practice how sensitive is model performance to the choice of \nG? Have you observed performance trade-offs when varying the number or type of basis functions?\n\nInterpretability Impacts:\nOne often-cited feature of KANs is their symbolic interpretability. Do LipKANs (with additional Lipschitz layers and L1.5 regularization) preserve this interpretability, or does the added complexity obscure it?\n\nSuggestions for Improvement\n\nVisualization of Complexity Metrics:\nA plot showing how the Lipschitz complexity evolves during training across different variants (KAN, LipKAN, etc.) would offer intuitive insight into how the regularization actually impacts model complexity and generalization in practice.\n\nEmpirical Study of Covering Bounds:\nAlthough theoretical, it would be illuminating to empirically explore how covering numbers (or their proxies) behave in practice for different architectures. For example, measuring empirical Rademacher complexity or sharpness may help support the theoretical framework.", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\nStrong Theoretical Foundation:\n\nThe paper makes a significant theoretical contribution by defining Lipschitz complexity as a structural complexity measure for deep KANs. The derived generalization bounds are carefully constructed, with a clear connection to covering numbers and tensor norms.\n\nThe generalization bounds are both novel and insightful, shedding light on how the architecture’s functional richness impacts its statistical behavior. The connection to known results for MLPs (e.g., Bartlett et al. 2017) is well-executed and positions the contribution in context.\n\nArchitectural Innovation – LipKANs:\n\nThe introduction of Lip layers and the shift from L1 to L1.5 regularization are elegant and well-motivated by the theory.\n\nThe reformulated architecture adheres closely to the theoretical framework, and the algorithmic instantiation (Algorithm 1) is clear, reproducible, and practically useful.\n\nClarity and Rigor:\n\nThe paper is written with excellent clarity, especially for a technically dense topic. Definitions, assumptions, and lemmas are well organized.\n\nThe proofs (particularly Theorems 4.4 and 4.5) are carefully structured and leverage standard but non-trivial tools in statistical learning theory (e.g., tensor covering bounds, sub-Gaussian concentration).\n\nEmpirical Validation:\n\nThe experiments, though not extensive, support the key claims. Notably, the improvements on more complex datasets (e.g., STL-10, CIFAR-100, MIMIC-III) convincingly demonstrate the generalization benefits of LipKANs.\n\nThe ablation study on L1.5 regularization is especially useful, highlighting the nontrivial gain over L1 or standard Lip layers alone.\n\nWeaknesses\nLimited Experimental Breadth:\n\nWhile the theoretical and algorithmic contributions are substantial, the empirical section could be strengthened. The paper evaluates a modest set of tasks and models; additional baselines (e.g., Transformer-based models, state-of-the-art MLPs, or regularization baselines like dropout) would provide a more complete picture of LipKANs' practical competitiveness.\n\nExperiments on regression tasks or out-of-distribution generalization could further validate the claims about improved generalization due to lower Lipschitz complexity.", "was_revised": true, "final_justification": "Thanks for the rebuttal and it helps clarifies things. great paper.", "rebuttal": "We would like to thank the reviewer for their careful reading and constructive suggestions.  We respond to each point below.\n>**Q1**: Role of Activation Family Cardinality.\n\nYes, we thank the reviewer for the thoughtful question.\n\nThe activation family size $G$ reflects the cardinality of the basis function family, treated as a constant that captures overall family size, while its **internal structure** (e.g., smoothness) remains tunable. For instance, KANs can employ different basis families—B-splines (original KAN), Fourier (Fourier-KAN), and rational functions (Rational-KAN)—each with adjustable basis affect generalization without altering $G$.\n\nFrom a practical perspective, we conducted sensitivity analyses on B-spline basis functions by varying **degree** and **grid size**, which together determine the shape and cardinality of the activation family. The key findings are:\n1. Internal structure matters beyond size: Even when $G = \\text{Grid size} + \\text{Spline order}$ is fixed, generalization can vary significantly. On MNIST, both (3,4) and (5,2) yield the same $G$, yet the Final Gen Gap differs by over 7×—highlighting that **basis structure**, not just size, plays a crucial role.\n2. Opposing effects of grid size vs. spline order: With fixed grid size, increasing spline order improves generalization; conversely, with fixed spline order, increasing grid size degrades generalization. This aligns with our view that smoother basis functions (higher order) help regulate model complexity.\n3. Efficiency trade-offs: Larger $G$ increases computational cost, with **spline order** contributing more to time and memory overhead than grid size.\n\nThese results reinforce our theoretical claims and demonstrate the framework’s robustness across different architectural settings. We will include this analysis in the revised appendix.\n| Dataset      | Grid Size | Spline Order | #Params   | Final Gen Gap (%) | Test Acc (%) | Overfit Ratio (%) | Train Time (s) | Peak GPU Mem (MB) |\n|--------------|-----------|---------------|-----------|-------------------|---------------|--------------------|----------------|--------------------|\n| **CIFAR-10** | 3         | 2             | 1,387,008 | 1.42              | 50.42         | 2.81               | 196.96         | 39.57              |\n|              | 3         | 3             | 1,584,256 | 0.79              | 49.60         | 1.59               | 201.86         | 42.17              |\n|              | 3         | 4             | 1,781,504 | 0.59              | 49.51         | 1.19               | 212.61         | 45.08              |\n|              | 5         | 2             | 1,781,504 | 4.00              | 52.39         | 7.64               | 201.16         | 45.05              |\n|              | 5         | 3             | 1,978,752 | 3.06              | 52.33         | 5.85               | 210.33         | 48.03              |\n|              | 5         | 4             | 2,176,000 | 2.30              | 52.15         | 4.41               | 221.63         | 50.50              |\n|              | 7         | 2             | 2,176,000 | 6.94              | 53.19         | 13.05              | 203.14         | 50.47              |\n|              | 7         | 3             | 2,373,248 | 5.78              | 52.94         | 10.92              | 215.97         | 53.51              |\n|              | 7         | 4             | 2,570,496 | 4.50              | 53.23         | 8.45               | 228.08         | 56.85              |\n| **CIFAR-100**| 3         | 2             | 1,427,328 | 1.47              | 22.00         | 6.68               | 202.60         | 40.22              |\n|              | 3         | 3             | 1,630,336 | 1.65              | 21.08         | 7.82               | 198.90         | 42.90              |\n|              | 3         | 4             | 1,833,344 | 1.52              | 21.00         | 7.24               | 211.00         | 45.90              |\n|              | 5         | 2             | 1,833,344 | 3.09              | 23.29         | 13.27              | 197.98         | 45.87              |\n|              | 5         | 3             | 2,036,352 | 2.57              | 23.57         | 10.91              | 204.95         | 48.93              |\n|              | 5         | 4             | 2,239,360 | 2.48              | 22.59         | 10.98              | 220.06         | 51.49              |\n|              | 7         | 2             | 2,239,360 | 4.66              | 24.63         | 18.93              | 201.39         | 51.47              |\n|              | 7         | 3             | 2,442,368 | 4.05              | 24.12         | 16.78              | 214.83         | 54.59              |\n|              | 7         | 4             | 2,645,376 | 3.92              | 24.04         | 16.31              | 227.14         | 58.02              |\n| **MNIST**    | 3         | 2             | 357,408   | 4.12            | 97.28         | 4.24             | 25.06          | 50.66              |\n|              | 3         | 3             | 408,224   | 3.49            | 97.59         | 3.58             | 26.35          | 52.84              |\n|              | 3         | 4             | 459,040   | 3.14            | 97.54         | 3.22             | 29.00          | 54.18              |\n|              | 5         | 2             | 459,040   | 3.56            | 97.33         | 3.66             | 31.87          | 54.18              |\n|              | 5         | 3             | 509,856   | 3.31            | 97.40         | 3.40             | 33.82          | 55.52              |\n|              | 5         | 4             | 560,672   | 2.89            | 97.45         | 2.97             | 35.92          | 56.86              |\n|              | 7         | 2             | 560,672   | 3.36            | 97.41         | 3.45             | 38.63          | 56.86              |\n|              | 7         | 3             | 611,488   | 2.88            | 97.47         | 2.96             | 41.45          | 58.20              |\n|              | 7         | 4             | 662,304   | 2.91            | 97.47         | 2.99             | 43.99          | 59.54              |\n| **STL-10**   | 3         | 2             | 1,387,008 | 0.78              | 28.54         | 2.73               | 98.73          | 39.58              |\n|              | 3         | 3             | 1,584,256 | 1.20              | 28.10         | 4.27               | 95.53          | 42.73              |\n|              | 3         | 4             | 1,781,504 | 0.98              | 27.50         | 3.56               | 96.58          | 46.39              |\n|              | 5         | 2             | 1,781,504 | 1.45              | 32.41         | 4.47               | 93.69          | 46.37              |\n|              | 5         | 3             | 1,978,752 | 1.62              | 31.20         | 5.19               | 95.50          | 48.03              |\n|              | 5         | 4             | 2,176,000 | 1.01              | 31.25         | 3.23               | 97.00          | 51.81              |\n|              | 7         | 2             | 2,176,000 | 1.68              | 34.40         | 4.88               | 93.14          | 51.79              |\n|              | 7         | 3             | 2,373,248 | 1.95              | 33.91         | 5.75               | 97.27          | 54.07              |\n|              | 7         | 4             | 2,570,496 | 1.73              | 33.73         | 5.13               | 98.40          | 57.10              |\n\n---\n\n>**Q2**: Symbolic Interpretability.\n\nYes, we appreciate the reviewer’s attention to symbolic interpretability, which is an important strength of KANs. We confirm that **LipKANs preserve the symbolic interpretability** of standard KANs.\n\nSpecifically, the **core architecture** of KANs remains unchanged: each layer computes explicit linear combinations of parameterized basis functions (e.g., B-spline, Fourier, Chebyshev), which can be symbolically extracted and visualized. The **Lip layer** is applied outside the basis compositions and acts as a global scaling operator—it does **not interfere with the symbolic structure** of the basis expansions. Likewise, the $L_{1.5}$-regularization promotes generalization via weight norm control, without altering the **functional form** or symbolic interpretability of the model.\n\nTo empirically verify this, we follow the symbolic regression setup from the original KANs paper and train **LipKANs** to fit the same target function: $f(x, y) = \\exp(\\sin(\\pi x) + y^2)$, which combines periodic and polynomial components. We use soft pruning (threshold $10^{-4}$ from epoch 50). The results show that LipKANs achieve a 96.66% sparsity ratio, indicating that only a small subset of basis functions is retained, which is sufficient to represent the symbolic structure. Besides, training and test MSE are identical, confirming that the learned function captures the true symbolic form.\n\nTherefore, LipKANs enhance generalization while preserving symbolic transparency. In addition, **Appendix D.1** presents further function fitting results on more complex symbolic targets, reinforcing LipKAN’s interpretability. We will clarify this in the final version.\n\n---\n\n>**W**: Limited Experimental Breadth. **&** OOD Generalization.\n\nWe sincerely thank the reviewer for the valuable suggestions. While our current focus was on establishing the theoretical foundation and demonstrating initial empirical benefits, we fully intend to pursue these extensions in future work, and we will mention them as part of our roadmap in the final version.\n\n---\n\n>**Suggestions**: Visualization of Complexity Metrics. **&** Empirical Study of Covering Bounds.\n\nThanks for the constructive suggestions. We leave these directions for future work and view them as promising avenues to further bridge our theoretical analysis with practical model behavior."}]

Author rebuttal:

Meta-review: The paper introduces Lipschitz complexity as a principled measure of structural complexity for Kolmogorov–Arnold Networks (KANs), deriving complexity-dependent generalization bounds. Based on this theory, the authors propose LipKANs, which integrate Lipschitz-bounded layers and L1.5 regularization to improve generalization. Experiments across vision, text, and multimodal benchmarks support the claims with ablation studies.

This paper shows strong theoretical contribution and well-motivated architectural innovation derived directly from the theory and deficiency of previous work. Empirically, the new model has shown advantages across multiple domains, with ablations and generalization gap analysis. It has clear writing and strong connection between theory and practice. Rebuttal provided additional experiments, generalization metrics, efficiency analysis, and interpretability results.

Despite some drawbacks of this work, including:
- Improvements on simpler datasets are modest.
- Theoretical bounds are somewhat loose, as they do not fully capture inter-layer dependencies.
- Empirical breadth could be expanded.
the submission is technically solid, original, and well-presented.
Decision: Accept (poster)