On the Convergence of Two-Layer Kolmogorov-Arnold Networks with First-Layer Training
Authors: Seyed Mohammad Eshtehardian
Mohammad Hossein Yassaee
Babak Khalaj
Venue: ICLR 2026 Poster (2026)
Keywords: Kolmogorov-Arnold Networks (KANs)
Overparameterization
Neural Tangent Kernel (NTK)
Gradient Descent
Primary area: learning theory

Abstract: Kolmogorov-Arnold Networks (KANs) have emerged as a promising alternative to traditional neural networks, offering enhanced interpretability based on the Kolmogorov-Arnold representation theorem. While their empirical success is growing, a theoretical understanding of their training dynamics remains nascent. This paper investigates the optimization of a two-layer KAN in the overparameterized regime, focusing on a simplified yet insightful setting where only the first-layer coefficients are trained via gradient descent.

Our main result establishes that, provided the network is sufficiently wide, this training method is guaranteed to converge to a global minimum and achieve zero training error. Furthermore, we derive a novel, fine-grained convergence rate that explicitly connects the optimization speed to the structure of the data labels through the eigenspectrum of the KAN Tangent Kernel (KAN-TK). Our analysis reveals a key advantage of this architecture: guaranteed convergence is achieved with a hidden layer width of $m=\mathcal{O}(n^2)$, a significant polynomial improvement over the $m=\mathcal{O}(n^6)$ requirement for classic two-layer neural networks using ReLU activation functions and analyzed within the same Tangent Kernel framework. We validate our theoretical findings with numerical experiments that corroborate our predictions on convergence speed and the impact of label structure.

Published as a conference paper at ICLR 2026
ON THE CONVERGENCE OF TWO-LAYER KOLMOGOROV-ARNOLD NETWORKS LAYER TRAINING
WITH FIRST-
Seyed Mohammad Eshtchardian Department of Electrical Engineering
Sharif University of Technology Tehran, Iran
mohammad www.teeherdiieeshariff.c.
Mohammad Hossein Yassaee
Babak Khalaj
Department of Electrical Engineering
Department of Electrical Engineering
Sharif University of Technology
Sharif University of Technology
Tehran, Iran yassaee@sharif..e
Tehran, Iran
kha la aallaj@saaa jesharif.edde
ABSTRACT
Kolmogorov-Arool Networks (KANs) have emerged promising alterna- tive to traditional neural networks, offering enhanced interpretability based on the Kolmogorov-Aroole representation theorem. While their empirical success is growing, a theoretical understanding of their training dynamics remains nascent. This paper investigates the optimization of two-layer KAN in the overparam- eterized regime, focusing a simplified yet insightful setting where only the first-layer coefficients are trained via gradient descent.
Our main result establishes that, provided the network is sufficiently wide, this training method is guaranteed converge global minimum and achieve zero training error. Furthermore, we derive a novel, fine-grained convergence rate that explicitly connects the optimization speed to the structure of the data labels through the eigenspectrum of the KAN Tangent Kernel (KAN-TK). Our analysis reveals key advantage of this architecture: guaranteed convergence achieved with hidden layer width of m = On²²), ) significant polynomial improvement (nz)) requirement for classic two-layer neural networks using
over the ReLU activation functions and analyzed within the same Tangent Kernel frame- work. We validate our theoretical findings with numerical experiments that cor- roborate predictions convergence speed and the impact of label structure.
INTRODUCTION
Neural networks have become the cornerstone of modern machine learning. Despite their success, the nested composition of linear transformations and fixed nonlinearities (e.g., ReLU) creates an opaque structural complexity. often relegating these models to black boxes. This opacity makes it difficult interpret their decision-makinn processes, posing significant barrier in high-stakes domains where trust and transparency paramount. Kolnogonov-Arnold Networks (KANs) Liu jet al. (2025) offer a fundamentally different approach, with an architecture inspired by the Kol- mogorov-Amold representation theorem (Kolmogorov [961] Braun & Griebel, 2009). This theo- rem establishes that any continuous multivariate function can be decomposed into a nested sum of univariate functions, which are far easier to interpret.
Although the idea of building networks upon this theorem is not new, early attempts based directly on its two-layer structure struggled due to the potentially non-smooth and complex nature of the inner functions, making them difficult to learn in practice (Sprecher & Draghici] 2002) Köppen 200p)) Lin
& Unbehauen 1993
Lai & Shen 2021 Leni al. 2013 Fakhoury et al. 2022). The key innovation

---

Published as a conference paper at ICLR 2026
of modern KANs was to extend this shallow structure into a deep, multi- architecture, analogous to MLPs. This design mitigates earlier learning difficulties and shifts the paradigm: whereas MLPs place fixed nonlinearities at nodes, KANs place learnable univariate activation functions the edges. This architectural choice not only improves interpretability but also enhances parameter efficiency. These learnable edge functions are typically parameterized as linear combinations of basis functions, such as B-splines (de Boor] [000] [Schumaker] (007). More recent approaches have expanded this idea using alternative basis families, including Rational Polynomials Aghaei (2024b),
Chebyshev Polynomials SS et al. (2024), and Radial Basis Functions (RBFs) Li (2024). In addition, recent works such as Delis (2024); Huu et al.] (2025); Zhao al. (2025); Bozorgasi & Chen (2024);
Seydi 2024); Aghaei (2025) have introduced new classes of basis functions, further broadening the expressive power and adaptability of KANs.
The rapid emergence of KANs has led to exploration across diverse application domains. In com- puter vision, KAN-based convolutional architectures have demonstrated superior performance com- pared to traditional CNNs Bodner et al. 2
and have been successfully integrated into U-Net models for medical imaging (Li et al. 2025). For sequential data, Temporal KANs were introduced in Genet & Inzirillo (202)) where KANs replace the standard neural components in RNNs, yielding improved accuracy on complex time-series tasks (Han et al. 2024] Xu & Wang, 2024]. KANs have also been applied in reinforcement learning, achieving higher accuracy and per- kich
formance with significantly fewer parameters Guo & Liu 2024):] Kich et al. (2024), as well as in
time-series analysis tasks (Huang et al.) 2025 Zhou et al. 2025). Similar performance gains have
been reported in graph neural networks (Zhang & Zhang 2024 Fang et al. 2025 GuoguoAi et al.)
2025 Beyond these. KANs have shown strong potential scientific machine learning, particularly for solving partial differential equations, where they outperform physics- physics-informed neural networks (PINNs) (Wang et al. 2025b] TTocano et al., 2024 Aghaei (024a). The architecture has also been adapted for Transformers, showing promise for large language models (Yang & Wang 2025). Fur- thermore, Yu et al. (2024) demonstrated that KANs outperform MLPs on datasets constructed from
symbolic formulas. Comprehensive surveys and further results are available in (Ji et al. 2024 Rigas
cct al. 2024 Howard et al. 2024 Cheon 2024
Qiu et al. 2024 Polar& & Polucktov 202 Lee ct al.
2025
Alongside these empirical successes, a growing body of theoretical work has begun to establish a rigorous foundation for KANs. Several works have investigated the role of initialization, including interpolation- based, random random-based, and hybrid schemes designed to reduce the computational cost
of KAN initialization and ensure stable training across different basis functions (Rigas al. 2025). On the expressiveness side, Wang et al. (2025a) showed that KANs are at least as expressive as MLPs and may exhibit reduced spectral bias. Generalization properties have also been studied Zhang &
Zhou (225),, and other works explore deep learning alternatives to the classical Kolmogorov-Amol)
representation theorem itself (Guilhoto & Perdikaris 2025
Laczkovich 2021).
On the optimization side, wide range of algorithms have been proposed for training machine learning models Kingma & Ba (205)); Carmon et al.) (2018), with convergence guarantees typically relying on smoothness, Lipschitzness, or convexity assumptions (Li & Orabona (0099] Nesterov & Polyak 2006 Duchi et al. 2011 Reddi et al. 2019, Ji & Telgarsky| 2019). For MLPs, Zhang et al. (2021) observed that gradient descent (GD) and stochastic gradient descent (SGD) often reach nearly global minima in practice, driving the mean squared error toward zero. However, understanding why simple gradient-base based methods succeed in optimizing highly non-convex models such as MLPs and
KANs remains a central challenge.
Substantial progress has been made in the overparameterized regime Du al. (2019): Jacot et al. 2018): Arora et al.] (2019); CChiatt & Bach (2018a); Soudry & Carmon (2016): Sootano kotabi Soltanolkotabi et al. (2018) Vaswani al. (2019) et al. (2019); Polaccyk & Cyranka (2023), where neural langent kernel (NTK) type analyses yield convergence guarantees for sufficienlly wide networks.
2017 Xie et al. (2017) | Chizat & Bach
Oymak& & Soltanolkotabi
More recently, overparameterization requirements for two-layer networks have been sharpened: Po- laczyk & Cyrankal (2023) derive improved width bounds that ensure global convergence of GD through a refined analysis of the empirical Gram matrix. Extending this line of work to the KAN setting, Gao & Tan (2025) prove that a two-layer KAN converges to a global minimum when all parameters are jointly trained.
2

---

Published as a conference paper at ICLR 2026
In this paper, analyze the training dynamics of two-layer KAN under more constrained setting: only the first-layer coefficients are trained, while the second-layer coefficients are fixed after a random initialization. This setup, previously studied for standard neural networks Du et al. 2019
Arora et al. (2019), allows for a clearer analysis. Our contributions are as follows:
We prove that for two-layer KAN with only first-layer training, gradient descent con- verges to a global minimum, driving the training error to zero, provided the hidden layer sufficiently wide.
We derive novel, label-dependent bound on the convergence rate, showing that the speed of convergence is determined by the projection of the label vector onto the eigenvectors of the corresponding KAN Tangent Kernel (KAN-TK).
We show that the required width of the hidden layer for guaranteed convergence in KAN setup is significantly smaller than that required for standard two-layer neural net- works Du al. 2019), highlighting key parameter-efficiency advantage.
We provide empirical evidence that corroborates our theoretical findings, demonstrating the faster convergence for wider networks and the impact of label structure.
2 PRELIMINARIES AND SETUP
2.1 KOLMOGOROV-ARNOLD NETWORKS (KANS)
KAN'S architecture is inspired by the Kolmogorov- Arnold representation theorem, which states that any continuous multivariate function f: [0, 1] + R can be written as:
2d+1
f(x) Σ 4
Σ @pp((Tp)
where 4 and op.q are continuous univariate functions. While early attempts to build networks based on this theorem struggled|Sprecer & Draghici (2002); (200)) Kööpen (2002), the key innovation of modern KANs was to extend the two-layer structure of the theorem into a deep network, analogous to MLPs (Liu et al. 2025). In this architecture, learnable univariate functions, often parameterized as splines, are placed on the edges of the computation graph, while nodes simply perform summation. This in stark contrast to MIPs, where linear transformations occur on the edges and fixed non-linear activations are applied at the nodes.
The learnable edge functions are typically represented as a linear combination of basis functions, 6(z) E; c, (z)),, where the coefficients c; trainable parameters. A common choice for the basis functions B, (z) is B-splines, which are piecewise polynomials with favorable mathematical properties such as local support and controllable smoothness, making them well-suited for func- tion approximation (Schoenberg & Whitney 1953 de Boor 2001 [001] [Schumaker (007). The original KAN architecture, for instance,
cubic B-splines by default (Liu et al. 2025). To improve computational performance and explore different inductive biases, various alternatives have been proposed, including Radial Basis Functions (RBFs) Li 2024). Reflectional Switch Activation Func- tions (RSWAF) Delis (2024), Chebyshev Polynomials SS et al. (2024), Rational Polynomials Aghaci (202b)), and Fractional Jacobi basis functions (Aghaci 2025).
2.2 THE TWO-LAYER KAN ARCHITECTURE
We focus on a two-layer KAN with a d-dimensional input hidden layer of width m, and scalar output. The output f(x) is defined as:
f(x)
1
ΣΣ
m 9
Bpp))(Ep))
where Zp
g
mm
Here, {0,))== are a set of g basis functions (e.g., RBFs), Ayjk are the learnable coefficients for first layer, and But are the coefficients for the second layer. The 1 factor is standard scaling term used in overparameteriiztion analysis (Jacot al. 2018).
Vm
schematic illustration of this two-layer KAN architecture is provided in Figure
3

---

Published as a conference paper at ICLR 2026
INNER FUNCTIONS
OUTER FUNCTIONS
Σ (znd
Ei= Qudi(a))
Fb
f(x)
2 (1Te)
M Sraadel (2m)
Zie
Architecture of a two-layer Kolmogorov-Amold Network (KAN). The inner functions (blue) map inputs to intermediate latent variables 2, which are then processed by outer functions (orange) and aggregated.
Figure 1:
2.3 TRAINING DYNAMICS IN OVERPARAMETERIZED MODELS
Our analysis situated in the overparameterized regime, where the number of model parameters far exceeds the number of training data points. In this regime, neural networks trained with gradient descent often exhibit phenomenon known as "lazy training" Chizat Bach (2018a), where the network weights remain close to their initial values throughout training. This allows the network's output to be well-apprrximmted by first-order Taylor expansion around its initialization.
This lincarization gives rise to the Neural Tangent Kernel (NTK) Jacot et al. (2018), deterministic kernel that governs the training dynamics the network. For a two-layer MIP, it has been shown that if the network width is polynomially large in the number of data points n, gradient descent finds a global minimum, and the training dynamics are equivalent to kernel regression with the NTK (Du al. 2019: Arora al., 2019). Our work applies similar analytical framework to the two-layer KAN architecture.
2.4 TRAINING SETUP AND PROBLEM FORMULATION
We analyze the network under the following training protocol:
1. Initialization:
The first-layer coefficients Opjk are initialized independently from a Gaus- sian distribution (O, o²). The second-layer coefficients 3pt initialized independently and uniformly from the set {11,
2. Training:
Only the first-layer coefficients a = {apsk] are updated using full-batch gra- dient descent. The second-layer coefficients 8 (3pt) remain fixed throughout training (See Appendix E for more information).
Given a dataset ((x; U: H the goal is to minimize the mean squared error loss function:
1 y 2
L
Σ (vr f(x:))²
where is the vector of network outputs for all data points.
THE KAN TANGENT KERNEL.
Our analysis relies the concept of the KAN Tangent Kernel (KAN-TK), which characterizes the training dynamics of our two-layer KAN in the infinite-width limit. For a general model fo fo(), the tangent kernel is defined Hij (Vafo (m.), Volo Volo(w))): In the lazy training regime, this kernel
4

---

Published as a conference paper at ICLR 2026
remains nearly constant throughout training. Consequently, the complex, non-linear dynamics of the network can be accurately described by the much simpler process of kernel regression with this fixed kernel (Jacot et al. 2018 Additional explanations and details about tangent kernels are provided in Appendix A.I
For our specific two-layer KAN with ID input and RBF basis functions, we derive closed- form expression for the KAN-TK in the infinite-widhh width limit (m + oo). Since we only train the first-layer coefficients a, the kernel is computed with respect to these parameters. this section, we assume the basis functions (x) are Radial Basis Functions (RBFs), defined as:
xソ(+) exp (- (x- (- 12)²
(-~ 2n²
Proposition 3.1 (KAN Tangent Kernel with RBF basis). For two-layer KAN with RBF basis functions and fixed second-laye layer coefficients, the tangent kernel with respect to the first-layer weights the infinite-width limit is given by H The entry (H )ar (for S q, S is:
05(29) (o exp (-
H
gr
Σ th (pp)))))))))))) Xo + +ni +
Σ 6 y
04
31=1
where the auxiliary lensors are defined as follows:
Agi 61(2") (x "i(x"))'
by" - (()) (x()))) (I
Avr
G
)-
0²
Mi
T?" exp
(bary
G
8g
2
del (Gar)T""
Y
V det (GTr) (GT
= 20
X psl
det (GT")
μi det(tet) (G b"",, (G'' (G "2" 4
+
The derivation of this kernel is provided in Appendix A.2 The expression is highly complex and computationally intensive, scaling polynomially with the number samples n. This makes it im- practical for direct use in large-scale applications but provides powerful tool for theoretical analysis. Despite this complexity, we can use the kernel to perform regression and empirically ver- ify its expressive power. Moreover, in experiments relied this proposition specifically because it provides access to the eigenvalues and eigenvectors of the KAN-TK, which are essential for analyzing label alignment and convergence behavior.
THEORETICAL ANALYSIS
In this section, we present our main theoretical results. We first prove that gradient descent on our two-layer KAN converges global minimum with zero training error. We then refine this result by deriving a label-dependent convergence rate. Our analysis relies on few standard assumptions.
Assumptions.
We assume the following conditions hold:
1. Basis Functions:
The basis functions di are bounded, [x)(x) < 1, twice differentiable with bounded first and second derivatives, (1),, 10 (x)) S 1, and satisfy si(0) = 0.
2. Positive Definite Kernel: The infinite-wind winth KAN Tangent Kernel H is positive definite, meaning its minimum eigenvalue Ao is strictly positive (Ap 0).
5

---

Published as a conference paper at ICLR 2026
3. Bounded Data: The training data labels are bounded, |3s| S for all i.
The assumption of a Positive Definite Kernel is standard the analysis of overparameterized neural networks (Du al. 2019 Arora al., 2019). In particular, [Gao & Tan (2025) shows that this assumption holds for KANS equipped with appropriate polynomial basis functions. Their Lemma states:
Lemma 4.1 (Positive Definite Kernels). Assume that the basis functions are polynomials of degree
less than and the transformation functions are hyperbolic tangent or sigmoid. Then X >> O holds when all training samples distinct. If no transformation is used, Xo holds when the training
samples are linearly independent in the g-degree polynomial space:
{
Li,ds
1n
where g = (g- 1)2.
The transformation d (e-g.. tanh or sigmoid) ensures the first-layer outputs lie within the domain of the polynomial basis, so KAN variants using such nonlinearities satisfy the lemma when samples are distinct. In the no-transformation case (1/(2) = 2), the lemma only requires linear independence in the relevant polynomial space. Empirically, using FastKAN Li (024), we observe strictly positive minimum eigenvalues of the infinite width KAN-TK across several input distributions (c.g., 3.29 x 10 for inspace [-1, 1)), supporting this assumption in practice.
4.1 GLOBAL CONVERGENCE
We first establish that under sufficient overparameterization, the training loss converges to zero. Theorem 4.2 (Convergence to Global Minimum). Suppose the hidden layer width m sufficiently large and the initialization variance o² is sufficiently small, i.e.,
d c2 A
n1 8) :n), ) ))).
o= o=0 (
m
max
log
(
Then, with probability at least 06) over the random initialization, the gradient descent updates satisfy a linear convergence guarantee:
17,00 ((1),
C(++1)
where 17 = -( Aa gz is the learning rate and Ag Amia (H ) is the minimum eigenvalue of the
feffnite-widthe kernel.
Proof Sketch. The proof of Theorem 4.2, detailed in Appendix B proceeds by induction. The
core idea is to show that the network operates in the "lazy training" regime where the tangent kernel remains stable. We first expand the loss at step t + 1:
||y - u(t 1)1 bly (t) 2(y u(t))² (u(t + -(t)) || + 1) -1))))((11
u(t)ll
can be approximated by first-order Taylor series, which relates the tangent kernel at time t, H Jacot et al. 20. Using stability Lemmas below, 4.3
The change in the output,
4.4 and 4.5) we show that H(t) remains close to the deterministic, infinite-win width kemel H This stability allows us bound the terms the expansion and demonstrate a consistent linear decrease in the loss at each step.
Lemma 4.3 (Coefficient Stability). Under the assumptions of Theorem 4.2 the first-layer coeffi- cients remain in small neighborhood of their initialization values throughout training. That is, (os)k(t) (0)) < R. where R = O Xamm
gV=
[|u(0) - yll2).
Lemma 4.4 (Kernel Stability over Time). With high probability, the distance between the tangent kernel time and at initialization is bounded: (t) H dn d2 R R.
Lemma 4,5 (Initial Kernel Concentration). With high probability, the distance between the initial
langent kernel and the infinitewiddth kernel is bounded: || (())) H H0 <
do mm
log
2n
(

---

Published as a conference paper at ICLR 2026
4.2 LABEL-DEPENDER CONVERGENCE RATE
Next, we refine the convergence rate to show its dependency on the structure of the data labels. Theorem 4.6
Convergence Bound). Under the same conditions as Theorem 4.2 let the eigendecomposition of the KAN-TK be H Ei=1 A.a,oe Then the error vector I can be bounded as:
y y--ua utt S
Σ (1 (1-m))))> (1)
where is a small error term that vanishes m +
Proof Sketch. To prove Theorem 4.6 we start with the gradient descent update rule and show
that the change in the output can be approximated as u(t + 1) u(t) -n (u(() - y) This allows to express the error vector at step t + 1 (I n ) (u((t) y). Unrolling this recurrence yields u(t) - y ~ -(1 Iu (
recurrence relation: (u(t + 1) y) s
y). By assuming small initialization variance o², the initial output [|u[0]]|2 is negligible compared to || Taking the norm and applying the eigendeconposition of H 00 gives the desired label- dependent bound. The full proof is deferred to Appendix C
Remark (Eigenstructure and Convergence Speed). Theorem 4.6 demonstrates that the compo- nents of the error aligned with eigenvectors (w:) correspording to large eigenvalues (Ac) decay the fastest. Consequently, the label vector y has strong projection onlo these top eigenvectors (i.e., the labels have a structure that the kernel is well-suited to learn), the overall convergence will be much faster than if the labels were random or aligned with eigenvectors of small eigenvalues.
5 EXPERIMENTS
We conduct series of experiments using a two-layer KAN with RBF basis functions to validate our theoretical claims. Our implementation based on the FastKAN architecture (Li 2024). In all experiments, train only the first-layer coefficients using full-batch gradient descent, keeping the second-layer coefficients fixed after their random initialization. Additional experimental results are provided in Appendix D
5.1 CONVERGENCE RATE VS. NETWORK WIDTH
To validate Theorem 4.2 and the underlying "lazy training" phenomenon, we study how the hidden layer width m influences convergence.
Setup. We generate synthetic dataset with n 100 samples in d = 100 dimensions, where each feature is drawn from a standard normal distribution. Labels are drawn independently from (0, 1) create challenging learning task. We train KANs with varying hidden widths (m E (500, 1000, 2000, 4000, 8000, 16000, 32000}) for 5000 epochs. Each configuration evalu- ated over three independent runs, and the resulting training errors reported.
Results. Figure 2a reports the training error across epochs. As predicted by Theorem 4.2 larger widths m yield faster convergence. Figure 2b shows the maximum distance of the weight coeffi- cients from initialization, |[ox(() a< oo. As increases, the weights travel shorter distances. empirically confirming the "lazy training" assumption in Lemma 4.3
5.2 IMPACT OF LABEL STRUCTURE ON CONVERGENCE
We empirically evaluate Theorem 4.6 which predicts that the convergence rate of gradient descent is determined by how the label vector y aligns with the eigenspectrum the KAN-TK.
Setup for Figure 3a We generate a one-dimensional dataset with n = 50 points sampled uni-
formly from [1, J..
After computing the infinite-width KAN-TK H we project several label

---

Published as a conference paper at ICLR 2026
(a) Training error
Weight distance from initialization
Figure 2: Convergence behavior across hidden widths m. (a) Training decreases faster for wider networks. (b) Wider networks exhibit smaller deviations from initialization, consistent with the lazy training regime.
configurations onto its eigenvectors. We compare structured labels of the form
sin (0.7x/)))))
sin2(x/2)
with random labels drawn independently from N/O,,), 1).
Setup for Figure 3b We conduct second experiment similar one-dimensional dataset with n 30 uniformly spaced points in |-1, 1]. We evaluate three label configurations:
Structured, given by Eq. equation !
2. Random, sampled i.i.d. from N/ 1);
3. Anti-structured, defined as the eigenvector of H associated with its smallest eigenvalue.
For all settings, we train two-layer RBF-based KAN with hidden width 5000, updating only the first-layer coefficients for 3000 epochs using butl-batch gradient descent.
Results. Figure 3a illustrates the projections of the structured and random label vectors onto the eigenbasis of H The structured labels concentrate most of their energy on the top eigenvectors, whereas random labels distribute their mass more uniformly across the spectrum. Figure 3h shows the resulting optimization dynamics: networks trained on structured labels converge the fastest, random labels converge at a moderate rate, and anti-structured labels converge the slowest. Together, these observations provide strong empirical support for the label-dependent convergence behavior predicted by Theorem 4.6
COMPARISON AND DISCUSSION
We now compare the complexity of our proposed training scheme with two key benchmarks: (1) standard two-layer neural networks Du al. 2019), and (2) two-layer KANs where both layers trained (Gao & Tan 2025). This analysis highlights the trade-offs hetween parameter efficiency, stability, and convergence speed.
6.1 PARAMETER AND WIDTH COMPARISON
Table summarizes the asymptotic requirements hidden layer width and the number of trainable parameters needed to guarantee convergence to a global minimum.
Our method substantially reduces the required network width (m) compared to standard ReLU- activated Neural Networks (NNs) while employing the same Tangent Kernel (TK) stability analysis methodology used for examining neural networks in the overparameterized regime (Du et al. 2019). Specifically, classical two-layer NNs (often using ReLU) require a width m = O(n") to guar- antee convergence, whereas firt-layer-only Aoldo Network (KAN) achieves this
8

---

Published as a conference paper at ICLR 2026
Convergence Rates of KAN with Different Label inttalications
6.000
Label Random Label
100
30
250
Lagtthhen wow.onnnllooa Tuitiing lokal
-0.001
Mom Pojerttors
0.0
200
130
350
-0.003
custom function randem
200
400
600
300
1000
Epochs
(a) Projection of labels onto the eigenspectrum.
(b) Convergence label structure.
Figure 3: Effect of label structure on convergence. (a) Structured labels align with top eigenvectors. whereas random labels distribute across the spectrum. (b) Training converges fastest for structured labels, slower for random labels, and slowest for anti-structured labels.
Table 1:
Comparison of required hidden layer width and number of trainable parameters for global convergence guarantees.
Network Type
Hidden Layer Width (m)
Trainable Parameters
n
ndd
Neural Network (Du al. 2019)
An8s
1150
g n
10 n
KAN (Both Layers) (ao & Tan. 2025)
A
A
d nn
d g n
KAN (First Layer Only) (Ours)
X
XB
guarantee with m On?)), highlighting a parameter-efficiency advantage. This enhanced effi ciency stems directly from the superior expressive power of the learnable basis functions (such as polynomials) inherent in KAN architectures, which alleviate the need for extremely wide layers.
Compared to training both layers of a KAN, our method achieves improved stability with respect to Ag. particular, the dependence on the minimum eigenvalue of the tangent kernel, λo. improves from A, to Xo² This weaker dependence is advantageous because λo can be very small in prac- tice, and guarantees that are less sensitive to its value are therefore more robust. For instance, An decreases by factor of k. neural network would require k times more width to maintain convergence, whereas a KAN would require only 122 times more width. The trade-off is that our bounds introduce a stronger dependence Nonetheless, since the dataset size n typically dominates in practical settings, we regard this as favorable trade-off between stability and parameter scaling.
the input dimension d and the number of basis functions
Remark 2 (Why KANs Achieve Better Width Scaling than MLPs). KANs require only O(n²) width for kernel concentration and convergence, whereas two-layer ReLU networks typically need O(n The fundamental reason is the smooth and stable nature of KAN features during training. As empha- sized in the original KAN paper, KANs replace neuron-level activations with learnable univariate spline functions along edges. As a result, intermediate representations are compositions of smooth one one-dimensional functions rather than brittle, sign-dependent ReLU activations. This smoothness ensures that the Neural Tangent Kernel (NTK) of a KAN depends only on bounded derivatives of these splines and involves at most pairwise interactions between samples, yielding concentration with width scaling that is only quadratic in the dataset size. In contrast, classical ReLU networks must maintain stability of discrete activation patterns during training. NTK analyses (e.g.. (Dul lt al. 2019) show that preventing activation pattnrraatennnntt patternffiis flips requires controlling higher-order interac- tions among samples, which amplifies into the ( width requirement. Thus, the structural design

---

Published as a conference paper at ICLR 2026
Table 2: Comparison of required learning rates for guaranteed convergence.
Network Type
Learning Rate (1))
Ap
Neural Network (Du et al. 2019)
n
KAN (Both Layers) (Gao & Tan 2025)
Ao d&
KAN (First Layer Only) (Ours)
smooth functions edges, aligned with the Kolmogorov-Arrold representa- tion con-ielminates the combinatorial instability inherent ReLU networks and leads directly to the improved (n²²) scaling.
of
Remark (More Advanced Methods). Recent advances by Polaczyk Cyranka (2023) provide refined analysis of overparameterized networks using Clarke subdiffernntials and differential in- clusions, yielding a tighter width bound of (n for global convergence. While adapting these non-linear tools to KANs offers promising path for tighter theoretical guarantees, maintain benchmark against Du et al. (2019) to ensure methodological consistency with foundational results in the field.
6.2 CONVERGENCE RATE COMPARISON
While our approach is more parameter-efficient and stable, requires a smaller learning rate, which in turn leads to slower convergence. The key difference lies in the allowable step size 77.
As shown in Table 2 our method requires smaller step size than either the benchmarks. Since the linear convergence rate scales with иyλ, this smaller results in slower learning. This trade-off expected: by simplifying the optimization to only the first layer, we obtain stronger guarantees on parameter efficiency and stability, at the expense of convergence speed.
7 CONCLUSION
This work provides theoretical analysis of the optimization dynamics of two-layer Kolmogorov- Arnold Networks in the overparameterized regime. By focusing on a simplified setting where only the first layer is trained, zero training error. We also provide fine mnn-grddddd, label-dependent convergence rate that connects the optimization speed the intrinsic structure of the learning task. Our results demonstrate that KANs are not only more interpretable but also significantly more parameter-efficien than classical neural networks with ReLU activations, requiring a polynomially smaller hidden layer width (n (nn²) vs. m (nn)) to guarantee convergence.
prove that gradient descent converges to global minimum, achieving
Our analysis opens several promising avenues for future research. An immediate next step is to extend this theoretical framework to deep KANs to understand the role of depth in the training dy- namics and convergence rates. Another important direction is analyze the behavior of KANs under more practical, stochastic optimization algorithms like Adam. Furthermore, exploring alter- native theoretical methodologies beyond the tangent kernel framework, like the one in Polaczyk & Cyranka (2023), is crucial for deriving tighter convergence bounds. We can also examine the interpretability of KANs specifically within the overparameterized regime, connecting theoretical guarantees with explanatory power. Additional research should focus deriving closed-form ex- pressions for the KAN Tangent Kernel for multi-dimensional inputs and other basis functions, which would provide deeper insights into different KAN architectures. Finally, must also examine the impact various initialization techniques on the performance and theoretical guarantees of KANs in the overparameterized setting.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Aghaei. Kantrol: A physics- kolmogorov-arnold network framework for solving multi- dimensional and fractional optimal control problems. arXiv preprint (tt:::20090066444 2024a.
Alireza Afzal Aghaei. rkan: Rational kolmogorov-amold networks.
arXiv preprint
arXiv:2406. 14495, 2024b.
Alireza Afzal Aghaci. fkan: Fractional kolmogorov-arnold networks with trainable jacobi basis functions. Neurocomputing, 623:129414, 2025.
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- paramcterization. International conference on machine learning, pp. 242-252. PMLR, 2019.
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine- Fine-grained analysis of op- timization and generalization for overparameterizee two-layer neural networks. In International conference on machine learning, pp. 322-332. PMLR, 2019.
Rajendra Bhatia. Matrix analysis, volume 169. Springer Science & Business Media, 2013.
Alexander Dylan Bodner, Antonio Santiago Tepsich, Jack Natan Spolski, and Santiago Pourteau. Convolutional kolmogorov-arrold networks. arXiv preprint wwX11:2400613155, 2024.
Zavareh Bozorgasl and Hao Chen. Wav-kan: Wavelet kolmogorov-armold networks, 2024. arXiv preprint arXiv:2405. 2024.
Jürgen Braun and Michael Griebel. On constructive proof of kolmogorov's superposition theorem. Constructive approximation, 30:653-675, 2009.
Yair Carmon, John Duchi, Oliver Hinder, and Aaron Sidford. Accelerated methods for nonconvex optimization. SIAM Journal Optimization, 28(2):155-11772, 2018.
Minjong Cheon. Demonstrating the efficacy of kolmogorov-arnold networks in vision tasks. arXiv preprint arXiv:2406. 14916 2024.
Lenaic Chizat and Francis Bach. note on lazy training in supervised differentiable program- ming (2018). arXiv preprint 11Xiv:1812.079955. 2018a.
Lenaic Chizat and Francis Bach. On the global convergence of gradient descent for over- parameterized models using optimal transport. Advances neural information processing sys- tems, 31, 2018b.
Carl de Boor. A Practical Guide to Splines, volume 27 of Applied Mathematical Sciences. Peringer-Verlage New York, revised edition, 2001. ISBN 978-0-387-95366-3. doi: 10.1007/ 978-1-4612-0099
Athanasios Delis. Fasterkan. http:////ta..aaac////aa/aaaaaaaa 2024.
Ivan Drokin. Kolmogorov-arnold convolutions: Design principles and empirical studies. arXiv preprint arXiv:2407.01092, 2024.
Simon S. Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterizze neural networks. In International Conference Learning Representations. 2019. URL https: http:att:://ppeeeiiiii
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
Daniele Fakhoury, Emanuele Fakhoury, and Hendrik Speleers. Exsplinet: An interpretable and expressive based neural network. Neural Networks, 322:32--466, 2022.
Taoran Fang, Tianhong Gao, Chumping Wang, YihaoShang, Wei Chow, Lei CHEN, and Yang Yang. KAA: Kolmogorov-arnde attention for enhancing attentive graph neural networks. In The Thirteenth International Conference on Learning Representations, 2025. URL https:
//openrevies net. net.fform? d=atxC2VSXT3

---

Published as a conference paper at ICLR 2026
Yihang Gao and Vincent YF Tan. On the convergence of (stochastic) gradient descent for kolmogorov- arnol networks. IEEE Transactions on Information Theory, 2025.
Remi Genet and Hugo Inzirillo. Tkan: Temporal Kolmogorov-arroke networks. arXiv preprint UrXiv:2405.07344, 2024.
Leonardo Ferreira Guilhoto and Paris Perdikaris. Deep learning alternatives of the kolmogorov superposition theorem. In The Thirleenth International Conference on Learning Representations, 2025. URL httph:tttt//peeeeeee
ew net/ fentt/forunne fowummi StwpemmmmmmSyyyyssseeeeeee
Li Guo, and Liu. Kan vs mlp for offline reinforcement learning. arXiv preprint arXi::2499.006655. 2024.
GuoguoAi, Guansong Pang, Hezhe Qiao, YuanGao, and Hui Yan. Grokformer: Graph fourier kolmogorov-amood transformers. Forty-second International Conference on Machine Learn- ing. 2025.
Xiao Han, Xinfeng Zhang, Yiling Wu, Zhenduo Zhang, and Zhe Wu. Are kan and kan-based models effective for time series forecasting? arXiv preprint arXiv:2408. 1306, 2024.
Amanda Howard, Bruno Jacob, Sarah Murphy, Alexander Heinlein, and Panos Stinis. Finite basis kolmogorov-arnole networks: domain decomposition for data-driven and physics-informed problems. arXiv preprint arXiv:2406. /9662, 2024.
Lexiang Hu, Yisen Wang, and Zhouchen Lin. Incorporating arbitrary matrix group equivariance into KANs. In Forty-second International Conference on Machine Learning, 2025. URL https:
Copenzeview.net/Eoee http://orrom1111ttttCoCC000
Songtao Huang. Zhen Zhao, Can Li, and LEI BAI. TimeKAN: KAN-based frequency decomposi- tion learning architecture for long-term time series forecasting. In The Thirteenth International Conference on Learning Representations, 2025. URLhttps:/ idd=WLc799NNbh
oppnrevieew / net/ forum?
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and gen-
eralization in neural networks. Advances neural information processing systems, 31, 2018.
Tianrui Ji, Yuntian Hou, and Di Zhang. A comprehensive survey on kolmogorov amold networks (kan), arXiv preprint 24X 2024.
Ziwei Ji and Matus Telgarsky. Polylogarithmic width suffices for gradient descent to achieve arbi- trarily small test error with shallow relu networks. arXiv preprint 19X.:19909..2292, 2019.
Victor A Kich, Jair A Bottega, Raul Steinmetz, Ricardo B Grando, Ayano Yorozu, and Akihisa Ohya. Kolmogorov- arnold networks for online reinforcement learning. In 2024 24th International Conference on Control, Automation and Systems (ICCAS), pp. 958-963. IEEE, 2024.
Kingma and Ba. Adam: A method for stochastic optimization. In International conference for Learning Representations, 2015.
Andrei Nikolaevich Kolmogorov. On the representation of continuous functions of several variables by superpositions of continuous funtinuussuunctionsssscooc of smaller number of variables. American Mathemat- ical Society, 1961.
Mario Köppen. On the training of a kolmogorov network. In Artificial Neural Networks-ICAL 2002: International Conference Madrid, Spain, August 28-30, 2002 Proceedings 12, pp. 474 479. Springer, 2002.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Miklós Laczkovich. A superposition theorem of kolmogorov type for bounded continuous functions. Journal of Approximation Theory, 269: 105609, 2021.
12

---

Published as a conference paper at ICLR 2026
Ming-Jun Lai and Zhaiming Shen. The kolmogorov superposition theorem can break the curse of di- mensionality when approximating high dimensional functions. arXiv preprint arXiv:2112.09963, 2021.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-22) 1998.
Jin Lee, Ziming Liu, Xinling Yu, Yixuan Wang, Haewon Jeong, Murphy Yuezhen Niu, and Zheng Zhang. Kano: Kolmogorov-anne neural operator. arXiv preprint arXiv:2509. 2025.
Pimman-E Leni, Yohan D) Fougerolle, and Frédéric Truchetet. The kolmogorov spline net- work for image processing. In Image Processing: Concepts, Methodologies, Tools, and Applica- tions, pp. 54-78. IGI Global, 2013.
Chenxin Li, Xinyu Liu, Wuyang Li, Cheng Wang, Hengyu Liu, Yifan Liu, Zhen Chen, and Yix- uan Yuan. U-kan makes strong backbone for medical image segmentation and generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 4652-4660, 2025.
Xiaoyu Li and Francesco Orabona. On the convergence of stochastic gradient descent with adaptive stepsizes. The 22nd international conference on artificial intelligence and statistics, pp. 983- 992. PMLR, 2019.
Ziyao Li. Kolmogorov-arnold networks are radial basis function networks arXiv preprint arXiv:2405.06721, 2024.
Ji-Nan Lin and Rolf Unbehauen. On the realization of a kolmogorov network. Neural Computation, 5(1):18-20 1993.
Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruehle, James Halverson, Marin Soljacic, Thomas Y. Hou, and Max Tegmark. KAN: Kolmogorov-arnold networks. In The Thirteenth International Conference on Learning Representations, 2025. https:::::/ //tps//evepeee.ee.ee net/
A. M. Mathai and Serge B. Provost. Quadratic Forms in Random Variables: Theory and Applica- tions. Marcel Dekker, 1992.
Yurii Nesterov and Boris T Polyak. Cubic regularization newton method and its global perfor- mance. Mathematical programming, 108(1):177-222) 2006.
Samet Oymak and Mahdi Soltanolkotabi. Toward moderate overparamettrrizatoo:: Global vergence guarantees for training shallow neural networks. IEEE Journal on Selected Areas in Information Theory. 1(1):84-105, 2020.
Bartlomiej Polaczyk and Jacek Cyranka. Improved overparamettizzaioon bounds for global conver- genee of SGD for shallow neural networks. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL http:httss//pprreeeeeee
Andrew Polar and Michael Polucktov. deep machine learning algorithm for construction of the cotmogorov--moll representation. Engineering Applications of Artificial Intelligence, 99: 104137, 2021.
Qi Qiu, Tao Zhu, Helin Gong, Liming Chen, and Huansheng Ning. Relu-kan: New kolmogorov- arnold networks that only need matrix addition, dot multiplication, and relu. arXiv preprint arXiv:2406.02075, 2024.
Sashank Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. arXiv preprint /904 09237, 2019.
Spyros Rigas, Michalis Papachristou, Theofilos Papadopoulos, Fotios Anagnostopoulos, and Geor- gios Alexandridis. Adaptive training of grid-dependent thysics-informed kolmogorov-arnold net- works. IEEE Access, 2024.
Spyros Rigas, Dhruv Verma, Georgios Alexandridis, and Yixuan Wang. Initialization schemes for kolmogorov-amold networks: An empirical study. arXiv preprint 2xXiv:5509.03417, 2025.
13

---

Published as a conference paper at ICLR 2026
I. J. Schoenberg and A. Whitney. On Pólya frequency functions. III: The positivity of translation determinants with an application to the interpolation problem by spline curves. Transactions of the American Mathematical Society, 74(2)246-255, 1953. doi: 10.1090/
$0002-9947-1155--05533399
Larry L. Schumaker. Spline Functions: Basic Theory, Cambridge University Press, 3rd edition, 2007. ISBN 978-0-521-70512-7.
Seyd Teymoor Seydi. Exploring the potential of polynomial basis functions in kolmogorov-
arnold networks: comparative study of different groups of polynomials. arXiv preprint arXiv:2406.02588, 2024.
Mahdi Soltanolkotabi. Learning relus via gradient descent. Advances in neural information pro- cessing systems, 30, 2017.
Mahdi Soltanolkotabi, Adel Javanmard, and Jason D Lee. Theoretical insights into the optimization landscape of over-parameterized shallow neural networks. IEEE Transactions on Information Theory, 65(2):742-766 2018.
Daniel Soudry and Yair Carmon. No bad local minima: Data independent training enror guarantees for multilayer neural networks. arXiv preprint
2016.
David A Sprecher and Sorin Draghici. Space-filling curves and kolmogorov superposition-basel neural networks. Neural Networks, 15(1):57-67, 2002.
Sidharth SS, Keerthana AR, Anas KP, et al. Chebyshev polynomial-based kolmogorov-arnoll networks: An efficient architecture for nonlinear function approximation. arXiv preprint arXiv:2405.07200, 2024.
Juan Diego Toscano, Theo Käufer, Zhibo Wang, Martin Maxey, Christian Cierpka, and George Em Karniadakis. Inferring turbulent velocity and temperature fields and their statisties from la- grangian velocity measurements using physics-informed kolmogorov-arnold networks. arXiv preprint arXiv:2407.15722 2024.
Sharan Vaswani, Francis Bach, and Mark Schmidt. Fast and faster convergence of sgd for over- parameterized models and an accelerated perceptron. The 22nd international conference on artificial intelligence and statistics, pp. 1195-1204. PMLR, 2019.
Yixuan Wang, Jonathan W. Siegel, Ziming Liu, and Thomas Y. Hou. On the expressiveness and spectral bias of KANs. In The Thirteenth International Conference on Learning Representations, 2025a. URL https:
net/ Eorum SotrrummiidyyylDRUuu /pwp//pppeeeeeeeeeeeeeeeeeoo /toLttts//peeeeeelleee
Yizheng Wang, Jia Sun, Jinshuai Bai, Cosmin Anitescu, Mohammad Sadegh Eshaghi, Xiaoying Zhuang. Timon Rabezuk, and Yinghua Liu. Kolmogorov- wwwpoov--anoooonnoonnoeoee neural network: A physics-informed deep learning framework for solving forward and inverse problems based on kolmogorov- arnold networks. Computer Methods in Applied Mechanics and Engineering, 433: 117518, 2025b.
Bo Xie, Yingyu Liang, and Le Song. Diverse neural network learns true target functions. In Artificial Intelligence and Statistics, pp. 1216-1224. PMLR, 2017.
Chen Xu and Wang. Kolmogorov-armold networks for time series: Bridging predictive power and interpretability. arXiv preprint D.XXi::2406022996, 2024.
Xingyi Yang and Xinchao Wang. Kolmogorov-arroke transformer. In The Thirteenth International Conference Learning Representations, 2025.
[d==Ceock533t]
Runpeng Yu, Weihao Yu, and Xinchao Wang. Kan or mlp: A fairer comparison. arXiv preprint arXiv:2407. 16674, 2024.
Zhang and Zhang. Graphkan: Enhancing feature extraction with graph kolmogorov amold networks. arXiv preprint arXiv:2406. 2024.
14

---

Published as a conference paper at ICLR 2026
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107- 2021.
Xianyang Zhang and Huijuan Zhou Generalization bounds and model complexity for kol- mogorov- arnold networks. In The Thirteenth International Conference on Learning Representa- lions, 2025. URL
net oegpeaneeeeeeeeetetoooorroooooed
Zhangchi Zhao, Jun Shu, Deyu Meng, and Zonghen Xu. Improving memory efficiency for training KANs via meta learning. In Forty-second International Conference on Machine Learning, 2025.
URL
net/ forum? id= 9bi Cm 3Mnd
Quan Zhou, Changhua Pei. Fei Sun, HanJing, Zhengwei Gao, Haiming Zhang, Gaogang Xie, Dan Pei, and Jianhui li. KAN-AD: Time series anomaly detection with Kolmogorov-arnokk net- works. In Forty-second International Conference on Machine Learning. 2025. URL https:
//openreview http://wwweee/oooommddddd0/-0000
A TANGENT KERNELS
A.1 BACKGROUND TANGENT KERNELS
The tangent kernel is key concept for analyzing the training dynamics of overparameterized net- works. Formally, for model (x) with parameters 0, the tangent kernel defined as
H,j (Vofo(z,), Vofo(z;))
where Ti Ry are data samples. Intuitively, H measures how similarly parameter updates induced by different data points affect the model output.
In the so-called lazy training regime, which arises when the network sufficiently wide, the tangent kernel remains nearly constant throughout training. This stability means that the nonlinear training dynamics of the network can be closely approximated by a linear model whose evolution is governed by this fixed kernel. As a consequence, gradient descent on the network is equivalent to performing kernel regression with the tangent kernel (Jacot et al. 2018).
For standard neural networks, this leads to the well-known Neural Tangent Kernel (NTK). In case, where we focus on two-layer Kolmogorov-Arool Networks (KANs) with only the first-layer coefficients trained, the analogous object is the KAN Tangent Kernel (KAN-TK), The KAN-TK cap- tures the interaction between input features and learnable basis- fuinctfnn coefficients. In the infinite- width limit (m + oe), we can derive deterministic closed-form expression for KAN-TK when using RBF basis functions, which we employ throughout our experiments.
Finite- and infinitewwidt kernels. If run an optimization algorithm, then the parameters evolve with time, making the tangent kernel time dependent. We denote the kernel at step t by
((1) = (1))
which is computed from the gradients at that point in training. If the network is initialized randomly. then (0) is itself random matrix. Its expectation over random initialization defines the infinite- width tangent kernel, denoted by H
Networks Act Like Kernel Ridge Regression. To see why wide neural networks effectively be- have like kernel methods, note that in the lazy training regime the features Vefo(x;; remain nearly constant during training. This means that the model output at time can be approximated by a linear expansion around initialization:
(x) & (x) + Veloco (x) (((() 0(0)).
Since the gradient features are fixed, learning reduces to finding linear coefficients on this (very high- dimensional) feature map. By the representer theorem, this is equivalent to solving kernel ridge regression problem with kernel matrix H, where each entry Hij measures the similarity between features induced by samples a; and x.j-
15

---

Published as a conference paper at ICLR 2026
INNER FUNCTIONS
OUTER FUNCTIONS
Σi= (udi(as)
f(x)
x
2 (m)
Zee
Figure 4: Architecture of two-layer Kolmogorov-Armold Network (KAN) with one dimensional input. The inner functions (blue) map inputs to intermediate latent variables 2, which are then processed by outer functions (orange) and aggregated.
In other words, training an infinitely wide neural network with gradient descent mathematically the same performing kernel regression with its tangent kernel. The nonlinearity of the original network is thus captured entirely through the structure of H while the optimization itself is no more complicated than linear regression in feature space.
Connection to training dynamics. One of the main advantages of tangent kernels is that they al- low us to describe the network's s dynamics explicitly. For example, under gradient flow optimization Du al. )0199, the output vector evolves according
du(t) dl
H()) (y u(t).
This shows that the convergence behavior of the network is governed entirely by the spectral prop- erties of (t) (or H in the infinite-width case).
Thus, the tangent kernel viewpoint bridges the gap between the nonlinear training of KANs and a tractable kernel regression framework, offering both analytical insights and practical tools for understanding their optimization behavior.
A.2 PROOF OF PROPOSITION 3.1
Figure illlssrrates the two-layer Kolmogorov-Arnold Network (KAN) in the special case of dimensional input. This schematic clarifies the roles of the Oij and 3a coefficients, the intermediate
activations 2,, and the final scaled aggregation - Σ. (a))(z;) that produces the network out-
mm
put f(r). The subsequent analysis in this appendix derives the infinite-width kernel H associated with this architecture by decomposing it into the contributions from the parameters (H1²) and the parameters (Hz))
By definition, we have:
(H )ar
(or
0f(a9)
0f(x"),
0s(x9) Of(x)) da
H = gr
(2)
03 03
da
16

---

Published as a conference paper at ICLR 2026
First, compute the H term. From the definition of our network, know that 0f()) v (z(). From this, we can conclude:
(H gr
f= (z)
(2(2)
E
3 ( (2)))))))))
(3)
where the second line follows from the law of large numbers as m - co. We can simplify this as:
-1g)))} (= (=9 115)2 + (z" - 12/)3 13)2)
HHi Arr =
E
exp
202
2p + (29)2 (2r)² 24/ + 2²)
exp
20²
/=1
(29)² + (x))² 2//((2 + +2²
exp
E
exp
o²
202
i=1
Since z" Ei=1 (10((2 and a ~ NO0, Ig), we can write ST = T A + (4; where
Agi (x <1(x") (xx) and bs"" = - + ()()))
(5)
Using the moment generating function for a quadratic form Gaussian random variables Mathai & Provost 1992), we get:
C'p 2
(bar) JT (I 2t b0rr
exp
E Texp (1.ST))
del (I - -1
Setting t 11/(20²) gives:
exp
o²
m (bar)
1
(H1)) 0r
Σ
exp
8o
-- --T
brr
J=1
det
I +
1
Agr
0
Next, we compute H.
The derivative with respect to ai is:
0f(z)
(z)
(8)
Dag
v
This leads to:
(H )rr E
s,1j=1
E (2) ()))))))()))))))
(9)
where the second line follows because [[[3,3,] = òrz- Since 21(2) - -n we have:
(155)
[22 11)(z" - -H))((=")(((²"))
(10)
04
j1=11
17

---

Published as a conference paper at ICLR 2026
The expectation term can be written
E (=9 (=)=-))(="= pc) exp (- (_(39- -m)-mm)22 mi)² + (=" -mme
(11)
20
Let (t) E(( )]. We can relate the expectation to derivatives of 21" (t) with respect to the components of b"
0 (1) Bla, B(xx(t,x))((5 exp((S5f))
(12)
O6g
8 7 (t) E(a,) cxp(ts())
(13)
06, 0b.
And we know that (2 - pi) - pi) E., agapo (0 (x') + pt + p4 Σ, a. Putting these pieces together, express the expectation in terms of 21" (t) and its derivatives:
00(x"))((2') (x") 0 ZT (1)
E E1 - mI)(z" - -4) exp (tS
Σ
(tpz)² Obp 06
3.P
byT OZ (t)
+ 11 (t) +
(14)
06
By defining G (I 2 and T7 exp Pu (bar) 2tt"a)- we can find closed forms for the derivatives of Z (t). Substituting these back gives the final expression for (H22)g which completes the proof. We have the following:
(t)
PMP((I - 2t bar)
TT
(15)
06
det (1 2tArr
O² 24" (t)
PAT((1 - -21. -1.A9)--1))
06 06
/det (1 - 2tAar
(4 21A""))-bbr))dl 21 -- -bbrr TTo
(16)
det (I 2ttAar)
By defining:
VI" 102 (1)
144 / det (G9) bor TTT
(17)
t. 06g
X 1 727 (1)
det (Ger) (G )xp T + +2
det (GT b*) (G bar TT
(G
&u Obg 06g
(18)
we can write:
E [22 - (2) pi) exp [tSf" }]= = Σ "e + +ZZZ +
Σ bar yor
(19)
Substituting this back into the expression for (H gr gives the final result:
(x))))) exp (-#+)
(H,0) gr
Σ (x())) + i + Σ
p yr
0+
(20)
18

---

Published as a conference paper at ICLR 2026
B PROOF OF THEOREM 4.2
We begin by recalling the two-layer Kolmogorov-Amold Network (KAN) architecture analyzed in this appendix (see also Figure
1 Vm
[(x)
2p=1 Ei=1 (ptc
(21)
Σ k= Σj= Opjkóy (xTe)
Fp
This formulation makes explicit the dependence of the network output ((x) the coefficients Opjk and Spt. which will be central in the stability analysis that follows.
B. PROOF OF LEMMA 4.3 (COEFFICIENT STABILITY)
By the induction hypothesis we have
C(t) S (1- 11200
L(t 1).
2
Hence
[1((t) -11 (1-
17.0
||e(t 1) -y11.
2
which implies
11
[i((t)
U112 -
1
|lu(t - 1) 2112
M7A0
|eeet
vll2
(since v1 S 1 for < 1)
2
70
1
|2((0)
yl12.
(22)
Now consider the gradient descent update for single coefficient Aija:
(tjj(() -iiz - = -17 OC(L- -
daijk
(ug(t 11 ya)
Σ 1=1
Spr di ( (2B)
(23)
Daujk
Taking absolute values and using (0((-) < 1 from the assumptions,
02
(t) acya(t -1]1
(ug(t 1) 3gg
m
T.P.!
(x(م) (0,(x1) dip | - 1) - Ug
(24)
m
a.P.!
By (t 1) V0
m
ng
<
|ua(t - 1) - 8112.
(25)
m
where in equation 24 we used 6,p = lli p}. and equation 25 the incquality |a|||| S /n |x|||
for a E T
19

---

Published as a conference paper at ICLR 2026
Summing these updates over to 1,
(1)))(1)) Oiisk 2
Oijjk(7 +1) - (isk(TT)
1-1
S n9
[2((T) y1l2
S 11
71
1720)
) |e((0) - 2l1
(by equation 22
4
r=0
1 (1 1770
)
V
[(((0) 2112
4
1 1 (1 17110
4g
Iu(0) 8ll..
(26)
Aoymm m
Defining
R 4g
[(((0) - y112,
Aay m
we conclude that (oa5):(1) (1) Obigh (0) S R for all t, completing the proof.
B.2 PROOF LEMMA 4.4 KERNEL STABILITY OVER TIME)
By definition, the (q) entry of the tangent kernel time is
(Bgg()) (t)(() do
Hor
(27)
da
From Section B.I we have already computed
(tg(t)
Σ Ba (((()))))
(28)
Daizk
Vm
Substituting equation
into equation 27 gives
Bis ( (1
Σ
(t) m Σ (1
Σ
Ba Bi (llo0(((0000)))))((00)000)000000
(29)
m kj..kl,
Therefore,
Har (0)
m i.j.kdl b(fe()00((((000=6633)))404000000
Σ m k.k.
Σ
(30)
k.k.l.s
where we used (,,(,) s 1 and (+((-) $ and we now for a, b, c, & we have lab cd -
la cl + 1b-66 d)
20

---

Published as a conference paper at ICLR 2026
From the network definition equation 21
9
(t) 20) S
(x (tyy aijk (0)] - gdR,
(31)
where the last inequality follows from Lemma 4.3 and the bound [0) S 1.
By Assumptions, the second derivative of di bounded, hence
(0)(={(0) - (0)= s (=5(1) - =0) s gdR.
(32)
Substituting equation 32 into equation 30
we obtaín
HHr (t) - Hqr -
Σ
2gdR 9d
(33)
i...k,,s
Finally, taking matrix norms gives
(1) (O))) & [[H(()) - H(O) s
Σ
(1) Hgr( S 2n R.
(34)
g,r=1
This completes the proof.
B.3 PROOF OF LEMMA 4.5 (INITIAL KERNEL CONCENTRATION)
We begin by observing that
Hap (0) = Σ
o,(x))
(35)
Since the coefficients Oijkk are independent across different the expression above can be written as the average of m i..d. random variables
X.""
Σ j.k.s
(x(x)
(36)
By our assumptions, each variable is bounded in absolute value by
XX S dg²
Applying Hoeffding's ' inequality, we obtain
mc²
P -0
(0) H He < exp
(37)
2d2
Taking union bound over all n² entries of the kernel matrix, it follows that
mc²
P Plyg.r E [n] (0) H S cl 2 2n² exp
(38)
2d
Equivalently, setting
dg
2n2
log
Vm
obtain that with probability at least
6,
& Lg
2n
(())) -
(())
00
log
(39)
m
which establishes Lemma
4.5
21

---

Published as a conference paper at ICLR 2026
B.4 PROOF OF THEOREM 4.2
Step I: Bounding the initial error. By the triangle inequality,
We begin by establishing an upper bound on the initial error.
2 (O)1 ك +2
(40)
From our assumptions, we have 11 12 S n. We now derive bound for the second term.
Step 2: Distribution of the initialization. By the definition of Zp in equation 21 know that
zp(0) ~N 0, [o((zN)
Therefore,
E 2pp (0)
2
Σ j.k
09 (xx) S gd.
π
Using assumptions, we obtain
1
Bl,,(0)) S
Elcb (z (0))
vm
p.t
S 5md g0/20.
By Markov's inequality, with probability at least
8, we have
v mm n/ o.
|ex()]l2 S [1)())] 2 8
(41)
Substituting equation
41 into equation 40 we obtain
md
y (0)) S n
82 n²
02
(42)
Step 3: Error recursion.
We now analyze the error step 1. Expanding the loss, we have y (t + 1) = 1y -u(t) (u(t + 11)-
y -tt1l - 2 (01-2((uu))) u(t)) (u(t+ (mn()) () u(t)) Ilu(t+ 1))(())
(43)
Step 4: Defining the error term. To control the update, define the error term
n
(L) = (t+++ ut(() + 1 Σ Har (1) (1x - yr).
(44)
By expanding Hyr(t) (t), obtain
7 H (t) - vr) = Σ (wr(t)
m
Σ 3a P.i.l
(45)
Substituting equation 45 into equation
we find
(() m
1
Σ Bpt 1)) - -)) -
p.l
22

---

Published as a conference paper at ICLR 2026
Step 5: Bounding the Taylor remainder.
By Taylor's theorem and assumptions, obtain
+==g((++1) ()(=)((()) di ((((())()(()()+))))))))))))))))) =x(1)) (2)(1+ -)(()))))
Moreover, from equation 25 one can bound
(t) =0(()) -
Σ
Apja (t + appj (t)ll,,
j.k
dup m
g [(()) - yyl2.
(46)
Step 6: Bounding the size of the update.
Using equation 46 we find
|1g(1 1) u,((1] S
koh (= + 1)) (x(((()))
V
1=8(1++ 00)
m
S ndg Vn|lu(() - 2211
(47)
Thus,
(u(t + - -u(t)l2 2 ||u(t + - u()ill S ndgg 3/2 (|((t) - 2l12
Substituting into equation 43
we obtain
ly ull 1)113
|yy =u()) 2(y - (()) - + +(() + ||u(1 1))(ll
[1 27Amiu (H)) + - + 11 u(t)ll
-
(48)
Step 7: Lower bounding the minimum eigenvalue.
We now lower bound Amin (H())). By
Weyl's perturbation inequality Bhatia (2013) and Lemma
4.5 if
d d
n
m O
log
X
3
then [H(0) H S 0o/4, and hence Amin (H())
Ag- Furthermore, by Lemma 4.4 if
R = (000o((n22 (2 ()), then
[H()) - H(t)ll2 S 0o/4.
Together these imply
Amin (H())) 0o/2.
Step 8: Final convergence bound. Substituting this into equation 48 and using the induction hypothesis |y u(t)lla yy 2 we obtain
ly +t < [ - ni + colly u(0)llz
Vm + g"n") 11 u(t)ll
n $ [1 - nıo + cinn"
n2 + ) ut))l. (49)
m
Finally, suppose that a = (8) (0/V mnngdd
and m = m=
On)). If we choose the learning rate
do
S n L2 gD
then it follows that
u(t + 1) < (1- 1120 11y u(t)
ly
which establishes the desired linear convergence rate.
23

---

Published as a conference paper at ICLR 2026
C PROOF OF THEOREM 4.6
We prove Theorem 4.6 by relying on the lemmas established in Appendix B tion 44 we have, for each coordinate,
Starting from equa-
Ug (t 1) (1((1) -1 (1)(((((r()) v-) (t).
(50)
In vector form this yields
( + 1) - u(1) = (m()) - y) + =(();
(51)
where e()) (1)
the coordinate-wise Taylor remainder.
Using Lemmas 4.5 and 4.4 we decompose (t) H 00 (H H00) and rewrite equation 51
as
u(t + 1) - u(t) = - (u(t) - y) (H H 22) y) + <(t).
(52)
Define
x(1) = - - (u(t) - y)
By the triangle inequality and the lemmas controlling H(O) H and H(I) H obtain the high-probability bound
l1x(t) S H (tt)la|lu(() yl
I 1 (0) |H H(t)l2)) [(() gll2
< 2n 2 d g ((t)) 122
(53)
where the last inequality uses the concrete bounds from Lemmas 4.5 and 4.4 (see main text for the precise dependence on m and R).
Set ((t) (t) + <t)). Then the one-step recursion becomes
u(t 1) y (I n (u(t) - -(().
(54)
Unrolling this recursion for steps gives
(t) - y--yy- = (I - -
Σ (( T=0
=
( +
(55)
7=0
The first term -(I n remaining two terms are negligible.
0 l'y is the label-dependent main term in the theorem: we must show the
Bounding the initialization term. From equation 41 we have (with high probability)
- ( (1 n/ (())a22 s (1 n/ V md
3/2
(56)
8
Thus for small initialization variance x² the initialization term decays exponentially and negligi- ble.
Bounding the accumulated error term. Using equation 53 and the bound on E(t) from Ap-
pendix we obtain for the accumulated error
Σ H(t 7=0
<
- -1- " -1
noo Cn²ignn [((()) - v112
Sn g R (1 muu ) [ v112.
(57)
T=0
24

---

Published as a conference paper at ICLR 2026
KAM Karnal
KAN Karnal Niynomial
coining
1302
-05
00
07
96
64
1.
-156
-0.75
-1.50
-0.35
0122
0.05
076
035
180
1spa
Linear data
(b) Polynomial data
Figure 5: KAN-TK regression results on synthetic datasets. Figure (a) shows performance on linear data, while Figure (b) shows polynomial data.
where an absolute constant absorbed into X and we used the induction bound ||u(t) y z (() yll2 in the last line. Substituting the bound equation 42 for [u(0) yl1a yields
n md 7)1 S n2ddg ' RV
20
n2 n²2g00
Σ (1-mnhn n7n
82
n2 n²d2
md n
82 n² 02
(58)
n7
Parameter choices and conclusion. as
If we choose the initialization variance and stability radius
0
MiÃo
a=
R
my ng
n5/2dd
then both the initialization equation and the accumulated error above can be made arbitrarily small. Under these choices the dominant term in equation 55 is -(I oo) and the label- dependent bound of Theorem 4.6 follows.
D ADDITIONAL EXPERIMENTS
D.1 ADDITIONAL REGRESSION EXPERIMENTS WITH KAN-TK
As shown in Figure kernel regression with our derived KAN-TK effectively fits both a simple linear function (Figure Sap and more complex polynomial function (Figure This demonstrates that the induced kernel captures the expressive function space of the underlying KAN. To avoid overfitting, apply Kernel Ridge Regression with regularization parameter of 0.1.
Setup. For both experiments, construct datasets of n 50 samples with inputs drawn uni- formly from the interval [1,, The linear dataset is generated from
= 5x +e, E~~ N(0,0.0).
while the polynomial dataset is generated from
0.5x 3.62. + 1.32x² + 2x +e, e N(0,0.01...
Results. Figure shows that the KAN-TK regressor recovers the linear function almost perfectly despite the additive noise. Figure 5b further illustrates that the kernel can fit a substantially more complex nonlinear target function with high accuracy. These results highlight the flexibility of KAN- TK: even with a modest number of samples, it adapts effectively to functions of varying complexity while maintaining robustness through regularization.
25

---

Published as a conference paper at ICLR 2026
(a) Training error
(b) Weight distance from initialization
Figure 6: Convergence analysis on the MNIST dataset. (a) Training error and (b) loo distance of weights from initialization.
38
(a) Training error
(b) Weight distance from initialization
Figure 7: Convergence analysis the CIFAR-10 dataset. (a) Training error and (b) Eoo distance of weights from initialization.
D.2 EXAMINING RESULTS ON MORE COMPLEX DATASETS
In this section, we evaluate our convergence and distance eestanc-frrominiine intttmiiiitialzzationn results standard image classification benchmarks: MNIST | et al. (1998) and CIFAR-10 Krizhevsky (2009).
Setup. For MNIST, we considered model widths m {32, 128, 512, 2048). and for CIFAR-10 used m E {256, 512, 1024, 2048}. All experiments were run for 20 epochs using the cross-entropy loss. For each setting, we performed three independent runs to account for randomness and reported the resulting errors.
Results. Figure 6 presents the training error and parameter deviation on MNIST, while Figure shows the corresponding results for CIFAR-10. In both datasets, we observe the same empirical trends identified earlier: increasing the network width leads faster convergence and smaller devi- ations from the initialization, demonstrating that these behaviors persist even on substantially more complex real-world tasks.
These results, utilizing the cross-entropy loss, also suggest a future direction: examining the be- havior of network parameters under cross-entropy to show that they remain close to initialization, consistent with the observations made using the MSE loss in this paper.
D.3 ADDITIONAL PROJECTIONS OF LABEL VECTORS
In Section we analyzed how the structure of the label vector y influences optimization by exam- ining its projection onto the eigenspectrum of the KAN-TK. Here, extend this analysis several additional structured label functions to further illustrate the relationship between label-kernel align- ment and convergence behavior.
We construct one-dimensional datasets with n 50 samples drawn uniformly from [-1, 1]. We consider four structured label functions:
Setup.
y exp(x), y In |æl + + +1,
y x
y = sin (00.4
26

---

Published as a conference paper at ICLR 2026
700
Random
Lnbel
750
250
Wmn of Prseedoonn
239
720
120
(a) Labels generated by y = (x)
(b) Labels generated by y
Bandom 100
ibbl Randon Label
Nom hraeciinss
200
Nore Progetharr
2.0
200
1.5-
330
1.0-
0.0
10
(c) Labels generated by xx1
(d) Labels generated by sin ((04ssnnzz)
Figure 8: Projections of structured label vectors onto the eigenspectrum of the KAN-TK matrix. Each plot shows how the label signal distributes across kernel eigenvectors: concentration top cigenvalues indicates more favorable alignment and thus faster convergence.
For each function, we compute the indinite-width KAN-TK, H and project the corresponding label vector onto its eigenbasis.
Results. Figure & shows the projection profiles across all four functions. In every case, the struc- tured label vectors place a substantial portion of their energy on the leading eigenvectors of the kernel, those associated with the largest eigenvalues. Such concentration indicates strong align- ment with the dominant kernel directions, which in turn predicts rapid convergence under gradient descent, consistent with theoretical characterization in Theorem 4.6) By contrast, shown the random-label experiments in the main text, unstructured labels distribute their energy more uniformly across the spectrum, resulting in slower and more erratic convergence.
An additional observation is that highly nonlinear mappings (e.g., y = exp(=)) yield especially concentrated projections on the top eigendirections. While this may seem counterintuitive, it flects the fact that smooth monotonic functions align well with the principal components of many kernel operators. Nevertheless, as the underlying label function becomes latory, the energy distribution spreads deeper into the spectrum, indicating reduced alignment and correspondingly slower convergence.
intricate oscil-
D.4 EXAMINING THE TIGHTNESS OF THE CONVERGENCE BOUND
To empirically assess the tightness of the linear convergence bound derived in Theorem compare the theoretical rate with the observed training loss. Recall that the theorem guarantees per-iteration contraction of the loss by at least factor of m2)o/2). We therefore plot this
27

---

Published as a conference paper at ICLR 2026
Ph Cee.r Conpact Retes O.0
Theontolis, GEOAA hon Loweingfo
30
(a) 1) = 0.001
(b) 1) = 0.01
Theonstor
Thapsdol conaeraeeeee Teanse Farc
Liandin SuB m1
-25
0.2
132
124
-32
178
(c) 17 0.1
(d) rj
Figure 9: Comparison of the theoretical convergence bound (red dashed line) from Theorem 4.2 with the empirical training loss (blue solid line) for different learning rates.
theoretical upper bound (red dashed line) together with the empirical training loss (blue solid line) on logarithmic scale.
Setup. We use a two-layer Kolmogorov-Amold Network with hidden layer width m 5000. The training set consists of = 10 samples {(x;, ) where the inputs 5 are drawn uniformly from [-1, 1, 1]. and the labels are generated according to
= exp( (xp + 2
Training is performed with full- gradient descent for 1000 epochs. Importantly, only the first-layer coefficients Oijk updated during training. while the second-layer coefficients Ba are kept fixed, in line with the setting analyzed in Theorem 4.2 We vary the learning rate 17 E (0.001, 0.1,
For visualization, we report
[()
log
C(0))
that is, the base-10 10 logarithm of the ratio of the loss at iteration / the initial loss, and plot it against the theoretical bound / log /2))/2).
Results. As shown in Figure the empirical loss decreases consistently faster than the theoretical prediction, confirming that our analysis provides valid upper bound. The discrepancy between the empirical and theoretical curves reflects the conservatism of the bound, which is derived un- der worst-case assumptions. For small learning rates (1) = 0.001 and r = 0.01), the loss exhibits 0.1. the initial convergence is significantly faster than pre- dicted before flattening out. For 7 = 1, training becomes unstable and the loss fails to decrease, in accordance with the constraints on ") imposed by the theory.
smooth, nearly linear decay. For
28

---

Published as a conference paper at ICLR 2026
Rates m=1000)
0.10
002
legttainnng teron)
-0110
-0155
ReUU n-500
KAN n-7
PeUU m:730
3.30
n-0000
200
600
1000
Figure 10: Convergence of FastKAN and ReLU network with width m 1000 across different
sample sizes.
D.5 KAN VS. RELU NETWORKS ACROSS DIFFERENT SAMPLE SIZES
conducted an additional experiment to demonstrate that a one-hidden-layer KAN can outperform one-hidden-layer ReLU network of the same width when the number of training samples is large.
Setup. We trained standard neural network and a FastKAN model, each with a fixed hidden-layer width of 1000, using varying numbers of training samples (500, 750, 1000}. The experiment was performed on the synthetic dataset introduced in Section 5.1 and all models were trained for 1000 epochs.
with input dimension d = 100,
Results.
The results, shown in Figure I0] indicate that FastKAN exhibits substantially faster con- vergence compared a ReLU network of the same width across all sample sizes.
E RATIONALE FOR FIRST-LAYER TRAINING
In this section, we explain why we focused on first-layer training for KANs. As demonstrated in Sections E.I and E.2 training the first layer alone outperforms training only the second layer and achieves performance comparable to full-network training. This observation supports our assump- tion that training only the first layer is reasonable and efficient approach.
E. COMPARISON OF FIRST AND SECOND LAYER TRAINING
Here, we compare the effects of training only the first layer versus training only the second layer.
Setup. We follow the same experimental setup as in Section 5. The dataset consists of 100 samples drawn from 100-dimensional unit sphere, with random labels assigned the data points. We consider network widths E (500, 1000, 2000, 4000, 8000} and train for 2000 epochs. In the first experiment, the second-layer coefficients are fixed while the first layer is trainable; in the second experiment, the first-layer coefficients are fixed while the second layer is trainable.
training only the first layer yields significantly faster convergence than training only the second layer, supporting the decision focus on first-layer training.
Results.
As shown in Figure 11
E.2 COMPARISON FIRST AND FULL LAYER TRAINING
Next, we compare training only the first layer to full-network training.
Setup. The experimental setup is the same as in Section 5.I with 100 samples from a 100- dimensional unit sphere and network widths (500, 1000, 2000, 4000, 8000). Training is per-
29

---

Published as a conference paper at ICLR 2026
Tre ning Loss: First Layer Second Layer Tha
Final Loss Network Width
0.0
-0.0
-0.0
-377
-1.00
-1.23
-1.25
-1.9
Layer
m=1000
-175
m=1000
m--000
-1.5
Lager Thating satund Lapar buitiig
m=-000
m-5020
1poshs
Wilth (mr)
(a) Training error over all epochs
(b) Final training
Figure 11: Comparison first-layer and second-layer training: (a) training error over all epochs, and (b) final training error.
Training First
Rnal Network Layor Only
-08
Trining
-08
-0.25
0.50
log
-14
-0.75
Fist Layer Only mm-5001
-14
Fdl listiing (m-5500)
-1.4
-1.35
m-2000
m=2000
-1.8
150
m=0000
m-4000
-175
m-0000
-2.00
250
500
750
1000
1250
2505
1250
2000
Epochs
(a) Training error over all epochs
(b) Final training error
Figure 12: Comparison of first-layer and full-network training: (a) training all epochs, and (b) final training error.
formed for 2000 epochs. In the first experiment, only the first-layer coefficients are trained, while in the second, both layers are trained.
Results. Figure 12 shows that first-layer training achieves performance comparable to full-layer training. Moreover, it is more parameter-efficient and converges faster.
F ROLE OF LLMS IN THIS WORK
We used large language models (LLMs), including OpenAl's GPT and Google's Gemini, to assist with writing tasks and commenting on code during the preparation of this manuscript. The models employed strictly under the direct supervision of the authors. All technical content, experi- ments, results, and claims in this work are entirely the responsibility of the authors, and no output from the language models used without thorough verification.
30


Reviews:
[{"review_id": "sBqQYWRogg", "reviewer": "Reviewer_drqW", "summary": "This paper provides a theoretical analysis of two-layer Kolmogorov-Arnold Networks (KANs) in the overparameterized regime, focusing on the simplified case where only the first-layer coefficients are trained via gradient descent while the second layer remains fixed. The authors prove that under mild assumptions and sufficient network width, gradient descent converges to a global minimum with zero training error. They further derive a label-dependent convergence rate, linking optimization dynamics to the eigenspectrum of the proposed KAN Tangent Kernel (KAN-TK). Theoretical results are supported by synthetic experiments validating convergence behavior and illustrating the impact of label alignment on optimization speed.", "questions": "See the weakness.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper provides the first convergence analysis for two-layer KANs under partial parameter training (first layer only), which meaningfully extends existing theoretical work on neural tangent kernel dynamics to a new and interpretable architecture.\n2. Experiments, though simple, effectively confirm theoretical predictions, particularly the “lazy training” regime and label-structure-dependent convergence.", "weaknesses": "1. Limited experimental setting. The experimental validation is confined to synthetic datasets and simple settings. It remains unclear whether the theoretical findings hold in realistic, high-dimensional KAN applications (e.g., vision).\n2. Incomplete comparison to full-layer training. Although Table 1 provides asymptotic comparisons, the paper lacks experimental or theoretical discussion on how fixing the second layer affects expressivity or generalization, which could impact interpretability and optimization flexibility.\n3. Potential overemphasis on width reduction. While the polynomial improvement from $O(n^6)$ to  $O(n^2)$ is mathematically striking, the constants and dependence on other factors (e.g., $d^2g^6/\\lambda_0^2$) may diminish practical advantage, which is not empirically examined.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F9NvRTNcjZ", "reviewer": "Reviewer_WtFP", "summary": "This paper analyses the convergence behaviour of Kolmogorov-Arnold Networks (KAN) with the NTK framework, proving that they require less overparameterisation compared to MLP.\nThe proof is based on standard NTK convergence proof based on stability of parameter, which leads to the lazy training, concluding with the constancy of NTK throughout training and its equivalence to kernel gradient descent.\nThe supporting experiments show that wider KAN shows faster convergence behaviour, and also validates the lazy training phenomenon.", "questions": "1. Is slower convergence rate by (Gao & Tan, 2025) result of training both layers, or loose analysis? Can the proof strategy by authors be used to improve (Gao & Tan, 2025)’s result?\n2. On the other hand, was it impossible to prove the same dependency of overparameterisation on the sample size while training both layers? If so, does it actually require larger width empirically?  \n3. Can you give a table of the assumptions that each basis functions mentioned in the last paragraph of Section 2.1 satisfy? \n\n(Gao & Tan, 2025) On the convergence of (stochastic) gradient descent for kolmogorov–arnold networks. IEEE Transactions on Information Theory, 2025. https://ieeexplore.ieee.org/document/11079726", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "The organisation of the paper is clear, the main claim and strength of the paper, proof sketch, comparisons to the prior research are easy to follow.\nTheoretical results are stated without ambiguity, and the detailed requirement of the network hyperparameters are exactly described.\nThe proof in the supplementary material is also easy to read, each of the derivations describes which computations were performed, and each step is well separated to follow.\nTwo experiments in the main text support the main claims of the theoretical results, like global convergence and lazy training in Figure 2. \nFigure 3 and Figure 6 in the supplementary material shows interesting behaviours of KAN-TK, which allows one to understand inductive bias of overparameterised KAN.\nFinally, the theoretical results obtained is well analysed, highlighting the gains (less overparameterisation w.r.t. sample size, less dependency on condition number) and losses (stronger requirement on learning rate, and slower convergence).", "weaknesses": "Kolmogorov-Arnold network has gained interest due to its interpretability, particularly for its applications in scientific discovery.\nTherefore, it is questionable whether the overparameterised KAN remains interpretable, which seems unlikely according to the proof techniques. \nSpecifically, lazy training implies the parameters remain close to their initialisation, making them nearly random, and overparameterisation prevents direct interpretation..\nFurthermore, one of the core details for interpretability in (Liu et al., 2025) is the use of sparsification and pruning, which incorporates L1 or entropic regularisation.\nThis makes this paper's setting far from the practical use and the global convergence proof less interesting.\n\nWhile the paper claims to prove the superior convergence behaviour of KANs compared to MLPs, the NTK-based convergence proof (from Du et al., 2019) is not a state-of-the-art technique for proving the global convergence of neural networks. \nFor instance, Table 1 in (Polyaczyk and Cyranka, 2024) presents multiple sharper convergence rates. (Poyaczyk and Cyranka, 2024) proves a stronger bound of $n^{1.25}$, and under a similar data distribution assumption, (Liu et al., 2022) shows better convergence behaviour.\n\nIn the assumptions of Section 4, the positive-definite kernel property applies only to MLPs; therefore, the proof needs to be modified for KANs.\nTo the best of the reviewer's knowledge, this property holds if (1) the activation function is analytic and non-polynomial (Prop F.1 of Du et al., 2019) or (2) the Hermite polynomial expansion of the activation function has infinitely many non-zero coefficients (a modification of Thm 3.2 of Nguyen et al., 2021). \nConversely, it can be shown that if the activation function is polynomial, the infinite-width NTK has a finite rank, causing the minimal eigenvalue to become zero when the sample size is sufficiently large.\nThis may rule out some of the basis functions described in the paper, such as Chebyshev polynomials.\nTherefore, it will be helpful if the authors discuss what basis functions will satisfy the assumptions.\n\nWhile the experiments support the claim of theoretical results, a gap exists between the theoretical result and experimental result.\nIn Figure 2(b), although wider networks show a smaller weight distance from initialisation, it is unclear whether these distances are upper-bounded throughout the training as claimed in Lemma 4.2.\nIt would be better if the training time were extended to show a plateau in the distance plot.\nFurthermore, one of the paper's main claims is a reduced dependency on sample size; hence, it would be beneficial to show how convergence or lazy training is affected as the number of samples increases.\n\nThe target functions used in the experiments are nearly trivial, employing either random labels or $y=x$.\nConsidering that Figures 6 and 7 show more challenging target functions, it would be better to use such functions in these experiments, or, if possible, to use the target functions or tasks from (Liu et al., 2025).\n\nI am willing to increase my score to 4 if the third, fourth, and fifth issues in this section are addressed. \nFurthermore, I am willing to raise my score to 6 or 8 if the authors provide promising theoretical results on interpretable global convergence (addressing the first weakness) or demonstrate a faster convergence rate that substantiates their claim (addressing the second weakness).\n\n(Liu et al., 2025) KAN: Kolmogorov-arnold networks, ICLR 2025. https://openreview.net/forum?id=Ozo7qJ5vZi\n\n(Du et al., 2019) Gradient descent finds global minima of deep neural networks, ICLR 2019. https://openreview.net/forum?id=S1eK3i09YQ\n\n(Polyaczyk and Cyranka, 2024) Improved Overparametrization Bounds for Global Convergence of SGD for Shallow Neural Networks, TMLR. https://openreview.net/forum?id=RjZq6W6FoE\n\n(Liu et al., 2022)  Loss landscapes and optimization in over-parameterized non-linear systems and neural networks. Applied and Computational Harmonic Analysis, 2022. https://www.sciencedirect.com/science/article/pii/S106352032100110X\n\n(Nguyen et al., 2021) Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networks, ICML 2021. https://proceedings.mlr.press/v139/nguyen21g.html", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YFHH1DNIcI", "reviewer": "Reviewer_VZoT", "summary": "This paper presents a theoretical study of the optimization dynamics of two-layer Kolmogorov–Arnold Networks (KANs) in the overparameterized regime, under the setup where only the first-layer coefficients are trained and the second-layer coefficients remain fixed. The authors derive global convergence guarantees for gradient descent and provide a label-dependent convergence rate that depends on the eigenspectrum of the KAN Tangent Kernel (KAN-TK).\n\nTheir main results show:\n\nGlobal convergence with zero training error for sufficiently wide KANs.\n\nPolynomial width scaling of n^2, is much smaller than MLPs\n\nA fine-grained convergence bound connecting label alignment with KAN-TK eigenvectors to convergence speed.\n\nEmpirical experiments verifying width-dependent and label-structure-dependent convergence.", "questions": "please provide motivations \n1. why KAN is O(n^2) much smaller than MLP O(n^6); this is not clear to me\n2. training only first layer is a bit weird, can you train only the second layer instead (more like random feature KAN, which would be interesting)", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Timely theoretical contribution.\nKANs are a rapidly growing architecture class (with numerous 2024–2025 references), yet theoretical understanding of their training dynamics is scarce. This work fills an important gap by analyzing convergence properties in a simplified yet meaningful setup.\n\nMathematical rigor.\nThe paper provides clear theorems (Theorem 4.1 and 4.5) with appropriate assumptions and proof sketches. The use of lazy training analysis and the explicit derivation of the KAN Tangent Kernel are technically sound and well-aligned with the NTK literature.\n\nImproved scaling over MLPs.\n\nLabel-dependent convergence analysis.\nTheorem 4.5 and the supporting experiments (Figures 3a–b) provide nice intuition about how label structure interacts with the kernel spectrum — a useful insight beyond the standard NTK treatment.\n\nExperimental verification.\nAlthough synthetic, the experiments are cleanly executed and directly corroborate the theoretical claims (width scaling and label alignment).", "weaknesses": "Limited novelty beyond applying NTK analysis.\nWhile the paper is well-executed, its main theorems closely parallel classical NTK results (Du et al., 2019; Arora et al., 2019), with the main novelty being the explicit KAN-TK form. The analysis may be perceived as incremental rather than fundamentally new.\n\nSuggestion: Clarify what features of KAN-TK qualitatively differ from NTK (e.g., structure induced by spline or RBF bases) and how they affect learning beyond improved scaling.\n\nNo discussion of second-layer training dynamics.\nThe paper emphasizes “first-layer only” training for tractability, but it remains unclear how this extends to full training, or whether the key conclusions persist when both layers are updated. For KANs the 2 layer setting is more realistic than MLPs though\n\nMissing discussion on practical implications.\nAlthough the theory is strong, the practical takeaway (e.g., how this informs KAN training in real models like FastKAN or WavKAN) is underdeveloped. Readers would appreciate even brief remarks connecting this to design choices (e.g., number of basis functions, initialization scales). It would be good to cite the KAN 2.0 paper on scientific applications, and also the study of initialization schemes for KANs https://arxiv.org/pdf/2509.03417", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)