KAN: Kolmogorov–Arnold Networks
Authors: Ziming Liu
Yixuan Wang
Sachin Vaidya
Fabian Ruehle
James Halverson
Marin Soljacic
Thomas Y. Hou
Max Tegmark
Venue: ICLR 2025 Oral (2025)
Keywords: Kolmogorov-Arnold networks
Kolmogorov-Arnold representation theorem
learnable activation functions
interpretability
AI + Science
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: Inspired by the Kolmogorov-Arnold representation theorem, we propose Kolmogorov-Arnold Networks (KANs) as promising alternatives to Multi-Layer Perceptrons (MLPs).

Abstract: Inspired by the Kolmogorov-Arnold representation theorem, we propose Kolmogorov-Arnold Networks (KANs) as promising alternatives to Multi-Layer Perceptrons (MLPs). While MLPs have fixed activation functions on nodes ("neurons''), KANs have learnable activation functions on edges ("weights''). KANs have no linear weights at all -- every weight parameter is replaced by a univariate function parametrized as a spline. We show that this seemingly simple change makes KANs outperform MLPs in terms of accuracy and interpretability, on small-scale AI + Science tasks. For accuracy, smaller KANs can achieve comparable or better accuracy than larger MLPs in function fitting tasks. Theoretically and empirically, KANs possess faster neural scaling laws than MLPs. For interpretability, KANs can be intuitively visualized and can easily interact with human users. Through two examples in mathematics and physics, KANs are shown to be useful ``collaborators'' helping scientists (re)discover mathematical and physical laws. In summary, KANs are promising alternatives for MLPs. Despite the slow training of KANs, their improved accuracy and interpretability show the potential to improve today's deep learning models which rely heavily on MLPs. More research is necessary to make KANs' training more efficient.

Published as a conference paper at ICLR 2025
KAN: KOLMOGOROV-ARNOUJ NETWORKS
Ziming Liu Yixuan Wang Sachin Vaidya'
Fabian Ruehle
James Halverson Marin Soljačic
Thomas
Hou?
Max Tegmark" 1:4
1 Massachusetts Institute of Technology
2 California Institute of Technology
Northeastern University
4 The NSF Institute for Artificial Intelligence and Fundamental Interactions
ABSTRACT
Although machine learning a powerful tool for science, its black-box nature hin- ders the extraction of interpretable knowledge. In particular, although Multi-Layer Perceptrons (MLPs) are universal approximators, it is challenging to interpret what MI.ss are doing under the hood. This paper, inspired by the Kolmogorov- Arnold representation theorem, proposes Kolmogorov-Amoll Networks (KANs) promising alternatives MLPs, especially when interpretability is desired. While MLPs have fixed activation functions on nodes ("neurons")) KANs have learnable activation functions on edges ("weights"), KANs learn interpretable ID functions their edges whose connection graph is also simple enough to be ex- plained. Through two examples in mathematics and physics, KANs are shown be useful "collaborators" helping scientists (re)discover mathematical and physi- cal laws. Moreover, KANs are shown be more accurate and have faster scaling laws than MLPs in function fitting and PDE solving, both theoretically and empir- ically. However, we admit that training KANs could slower than MLPs, which should be addressed in the future scale them up.
Model
Multi-Layer Perceptron (MLP)
Kolmogorov-Arnold Network (KAN)
Theorem
Universal Approximation Theorem
Kolmogorov-Araole Representation Theorem
My aolw; w x b))
Jeel Σ m26bx,)
(Shallow)
flx)
f(x)
flood activation functions
(b)
leamatiee activation functions edges
Model (Shallow)
operation modes
Jearmable un edler
MLP(X) (W 0. W.
W.o a W2X00
KANO) (0, X2)
(Desp)
MLP(X)
KAN(x)
wondraoor food
(Degp)
W.
nonltseur leumable
Inncur dearmable
W
Figure 1: Multi- Perceptrons (MLPs) vs. Kolmogorov-Amol Networks (KANs)
INTRODUCTION
Multi-layer perceptrons (MLPs) Haykin (1994); Cybenko [1989]; Hornik et al. (198). also known as fully-connected feedforward neural networks, are foundational building blocks of today's deep
zmliu @ mit.edu

---

Published as a conference paper at ICLR 2025
learning models. The importance of MLPs can never be overstated, since they are the default mod- els in machine learning for approximating nonlinear functions, due to their expressive power guar- anteed by the universal approximation theorem Hornik et al. (1989). However, MLPs often lack interpretability, which makes them less useful for tasks when interpretability is key, e.g., when we want to extract symbolie formulas from datasets. In science, symbolic functions are prevalent, e.g.. E mc² (energy-man relation), = 1+ecose (ellipse). p = . /Z (Boltzman distribution). Although MIPs can numerically approximate these functions to reasonable accuracy, they cannot reveal symbolic structures of these equations.
Therefore, need representation theorem that is more aligned with symbolic representations than the universal approximation theorem. In our search, the good old Kolmogorov-Amol repre- sentation theorem (KA theorem) came our attention. Although the KA theorem has long been considered irrelevant for learning Girosi & Poggio (1989) because the theorem does not guarantee smoothness, we are more optimistic about the smoothness of deeper representations. For example, as we will show, f 22, 23, T4) = exp(sin(z + = + sin(2z + xz)) can be smoothly repre- sented by three-layer network, but two-layer network that attempts to fit this function leads to pathological representations.
Unsurprisingly, the possibility of using Kolmogorov-Amold representation theorem to build neural networks has been studied Sprecherr& & Draghici| (2002); Köppen (2002); Lin & Unbehauen (199);; Lai & Shen (2021); y Leni et al. [2khoouy et al. (2022); Montanelli & Yang| (2020). However, most work has stuck with the original 2epth-2 (2n +2n+1 1) representation, and many did not have the chance leverage more modern techniques (e-g., back propagation) to train the networks. Our contribution lies in generalizing the original Kolmogorov-Amoll representation to arbitrary widths and depths, revitalizing and contextualizing in today's ss deep learning world, as well as using empir- ical experiments to highlight its potential for AI + Science due to its accuracy and interoperability.
Named after the two great Mathematicians, Andrey Kolmogorov and Vladimir Amold, this new type of network is called the Kolmogorov-Aroold Network (KAN). Like MLPs, KANs have fully- connected structures. However, while MLPs place fixed activation functions nodes ("neurons"), KANs place learnable activation functions on edges ("weights"), as illustrated in Figure [ Each learnable weight parameter in an MLP is replaced by learnable ID function (parametrized spline) in a KAN. KANs' nodes simply sum incoming signals without applying any non- ooneeinarities.
Although interpretability is our initial motivation to develop KANs, KANS demonstrate impressive accuracy and fast scaling laws as well, both theoretically and empirically. Despite their elegant mathematical interpretation, KANs are nothing more than combinations of splines and MLPs, lever- aging their respective strengths and avoiding their respective weaknesses. Splines are accurate for low-dimensional functions but suffer from curse of dimensionality (COD) problem. MLPs, On the other hand, suffer less from COD thanks to their ability to learn features and compositional structure, but are less accurate than splines in low dimensions. KANs have MLPs on the outside and splines on the inside, combining the best of two things into one.
The paper is organized as follows: In Section 2 introduce the KAN architecture, analyze the network's approximation ability, and propose two training techniques to make KANs interpretable and accurate. In Section we show that KANs interpretable and can be used for scientific discoveries. We use two examples from mathematics (knot theory) and physics (Anderson local- ization) demonstrate that KANs can be helpful "collaborators" for scientists (re)discover math and physical laws. In Section [ we show that KANs are more accurate than MLPs for data fitting and PDE solving with better scaling laws. We conclude in Section Due to limited space, we defer related works to Appendix ynd discussion to Appendix Z
2 HELMOGOROV-ARNOR NETWORKS (KAN)
Multi-Layer Perceptrons (MLPs) are inspired by the universal approximation theorem. We instead focus on the Kolmogorov-Amold representation theorem, which can be realized by new type of neural network called Kolmogorov-Aron Arnold networks (KAN). We review the Kolmogoroy- theorem in Section 2.1 to inspire the design of Kolmogorov-Arolle Networks in Section 2.2 Sec- prooovies mathematical description of KANs' expressive power. Section 2.5 and Section 2.4 propose techniques make KANs accurate and interpretable.
2

---

Published as a conference paper at ICLR 2025
2.1 KOLMOGOROV-ARYOLI REPRESENTATION THEOREM
Vladimir Arnold and Andrey Kolmogorov established that if f is a multivariate continuous function on bounded domain, then can be written as a finite composition of continuous functions of single variable and the binary operation of addition. More specifically, for a smooth f: [0, [0,1]" - R,
2n++1
f(x) = f(01, ... ,
Σ A
(1)
q
where 6.pp= : |0,1] - R and &y R + R. In sense, they showed that the only true multivariate function is addition, since every other function can be written using univariate functions and sum. One might naively consider this great news for machine learning: learning a high-dimensional func- tion boils down to learning a polynomial number ID functions. However, these functions can be non-smooth and even fractal, they may not be learnable in practice Poggio et al. (2020). Be- cause of this pathological behavior, the Kolmogorov Amold representation theorem was regarded as theoretically sound but practically useless Poggio al. 2020).
However, we are more optimistic about the usefulness of the Kolmogorov- theorem for ma- chine learning. First of all, need not stick to the original Eq. (I) which has only two-layer non-linearities and a small number of terms (2n + 1) in the hidden layer: we will generalize the network arbitrary widths and depths. Deeper and wider networks potentially have stronger ex- pressive power with smooth functions. Moreover, most functions in science and daily life are often smooth and have sparse compositional structures Lin al. (2017). potentially facilitating smooth Kolmogorov-Amoll representations.
2.2 KAN ARCHITECTURE
Suppose we have a supervised learning task consisting of input- pairs {x, yi where we want to find f such that y/ ~ fxx) for all data points. Eq. ( implies that we are done if we can find appropriate univariate functions Φg.p and Φ.- This inspires us to design a neural network which explicitly parametrizes Eq. ① Since all functions be learned univariate functions, we can parametrize each ID function as B-spline curve, with Iearnable coefficients of local B-spline basis functions Now we have a prototype of KAN, whose computation graph is exactly specified by Eq. and illustrated in Figure (b) (with the input dimension n = 2), appearing as two-layer neural network with activation functions placed on edges instead of nodes (simple summation is performed on nodes), and with width 2n + 1 in the middle layer.
As mentioned, such network is known to be too simple to approximate any function arbitrarily well in practice with smooth splines! We therefore generalize our KAN to be wider and deeper. The key insight comes from the analogy between MI.Ps and KANs. In MIPss, once we define a layer (which is composed of a linear transformation and nonlinearties), we can stack more layers to make the network deeper. To build deep KANs, we should first answer: "what a KAN layer?" It turns out that KAN layer with nin dimensioila inputs and nout -dmensionnl outputs can be defined as matrix of ID functions
+ = (0
p 1,2, ... 7lns
9 1,2 Rout.
(2)
where the functions 6g.0 have trainable parameters (parameterized as B B-splines, see Appendix []. as detaild below. In the Kolmogov-Arood theorem, the inner functions form KAN layer with 7'n and nout = 2n + 1, and the outer functions form a KAN layer with nin = 2n + 1 and nout = 1. So the Kolmogorov-Amot
representations in Eq. (1) simply compositions of two KAN layers. Now it becomes clear what it means to have deeper Kolmogorov-Arnol representations: simply stack more KAN layers! The shape of a general KAN is represented by an integer array
[no, n1, "..))
where n, is the number of nodes in the ith layer of the computational graph. We denote the zth neuron in the 1th layer by (1, and the activation value of the (1, ineurrn by sy,s. Between layer / and layer + 1, there are nynl+1 activation functions: the activation function that connects (1, i) and (1 (1+ 1, is denoted by
0iijdt 1 0, --1, i i= 1, m4.
j =1,,.. 11111
(4)
Details in Appendix and illustrated in Figure 18 right.
3

---

Published as a conference paper at ICLR 2025
The pre-activation of ØL is simply 21.2 the post-activation of Φt,j. is denoted by øiJ. (11,,(221..). The activation value of the (1 + 1,j) neuron is simply the sum of all incoming post- activations:
Bi.s
j j= 1,,, M-+1-
(5)
In matrix form, this reads
(1)
$11.1(+)
21,1, (.)
01.2.1())
(1,2,()))
01,2,
X1-1
X1,
(6)
o ml (.)
where D, is the function matrix corresponding to the hth KAN layer. A general KAN network is composition of L layers: given input vector X I2" the output of KAN
KAN(x) ( PL-2
D Φo)x..
We can also rewrite the above equation to make it more analogous to Eq. assuming output dimension nz 1, and define f(x) = KAN(X):
n 2
n2
N1 Σ
no
f(x) Σ 11-1-1 $1-1=1
Σ
Σ
22,88,52
01.12.11
Σ
Do io (Kio)
(8) which is quite cumbersome. In contrast, our abstraction of KAN layers and their visualizations are cleaner and intuitive. The original Kolmogorov-Arool representation Eq. (1) corresponds to a 2- Layer KAN with shape 'n, 2n + 1, 1]. Notice that all the operations are differentiable, so we can train KANs with back propagation. For comparison, MLP can be written interleaving of affine transformations and non-linearities o:
(9) It is clear that MLPs treat linear transformations and nonlinearities separately W and while KANs treat them all together in o. In Figure (c) and (d), we visualize a three-layer MLP and a three-layer KAN, to clarify their differences. Implementation details of KANs are left in Appendix[[]
MLP(x) (W 1-1 o W1-2o0o.... W c a Wo)x.
Remark: Complexities, Assuming a KAN with depth L, width N, grid size G, spline order k. The model has ONN parameters. Suppose training batch has size B, memory usage is ( BN2GT.) the number of operations is O(22 BN2GL) both for forward and backward runs. The 2 factor is due to the recursive computation of order le splines.
2.3 KAN'S APPROXIMATION ABILITIES
SCALING LAWS
Recall that in Eq. ( the 2-Leyr width-(2n + 1) representation may be non-smooth. However, deeper representations may bring the advantages of smoother activations. To facilitate an approx- imation analysis, we still assume smoothness of activations, but allow the representations to be arbitrarily wide and deep, as in Eq. 7 To emphasize the dependence our KAN on the finite set of grid points, use & and &C below to replace the notation o (6).
and $1, used in Eq. ( and
Theorem 2.1 (Approximation theory, KAN), Let (I1, ".,,))) ,x)). Suppose that a function
fx) admits a representation
f ( L
$ Fo)x,
(10)
in Eq. 7 where each of the LLij are (k 1)-times continuously differentiable. Then there exists a constant C depending on f and its representation, such that we have the following approximation bound in terms of the grid size G: there exist k-th order B-spline functions &G such that for any 0 < S k. we have the bound
|| ( o 06-20.0.. ....o $ + )x|lem S CG-K-11mm
(11)
Here adopt the notation of T -norm measuring the magnitude of derivatives up to order mi
||و|| cm
max sup
[D*9(z)).
19 [9
4

---

Published as a conference paper at ICLR 2025
We leave the proof and an in-depth discussion on the implication of the theorem in Appendix D Asymptotically, provided that the assumption in Theorem 2.1 holds, KANs with finite grid size can approximate the function well with a residue rate independent of the dimension. This comes naturally since we only use splines to approximate 1D functions. In particular, for m = O, we recover the accuracy in 1 norm, which in turn provides bound of RMSE on the finite domain, which gives a scaling exponent k ++1 1. Of course, the constant C' is dependent on the representation; hence it will depend on the dimension. Notice that if the assumption in the theorem holds for shallow KAN, it automatically holds for deeper KAN by setting the remaining layers to identity. A more general version of approximation theory in larger function class can be found in Wang et al. (2024b). More discussion on how our results are related to neural scaling laws is included in Appendix K We also remark that: since the assumption in the theorem a strong one, the neural scaling law should not be expected to be universally applicable to all machine learning applications. Now the basic architecture of KANs is in place, we propose a few techniques to make KANs accurate and interpretable.
2.4 TRICKS FOR INTERPRETABILIE PRUNING AND SYMBOLIPYING KANS
How do we choose the KAN shape? If we know that the dataset is generated via the symbolic formula f2, y) exp(sin(TTz) + y²), then we know that [2, 1, KAN is able to express this function. However, in practice we do not know the shape a priori, so it would be nice to have approaches to determine this shape automatically. The idea is to start from large enough KAN and train it with sparsity regularization followed by pruning. One may even symbolify activation functions into symbolic functions like exp. sine, etc, to make KANs a useful tool for symbolic regression. The idea is to match learned spline functions with candidates in a symbolic function library specified by human users and replace the spline functions with the best-fitting ones. Details of these simplification tricks are included in Appendix M
2.5 A TRICK FOR ACCURACY: GRID UPDATE AND GRID EXTENSION
Grid update Since imput data and (especially) hidden activations can have eerriig ranges in training. update grids on the fly based on the statistics of input/activation ranges. The grid is initialized to be in [-1,1] (e.g., when G = 5, the grid points are [-1, -0.6, -0.2, 0.2, 0.6, 1.0). but once it receives input/activations, say, in the range [-3,3] (the maximum and minimum values are 3 and -3, respectively), the grid will be updated to [-3,3] (correspondingly, grid points become 13.-1.8.-0660066,.1 1.8,00)) to accommodate the whole range.
Grid extension A spline can be made arbitrarily accurate to target function as the grid can be made arbitrarily fine- tne-grained. This good feature can be inherited by KANs. By contrast, MLPs do not have the notion of fine-graining". For KANs, one can first train a KAN with fewer parameters and then extend it to KAN with more parameters by simply making its spline grids finer, without the need to retrain the larger model from scratch. The main idea of grid extension is: for each 1D function defined on a coarse grid, we determine the coefficient of a finer grid using least squares that minimize the difference between the two curves evaluated data samples. Details of how to perform grid extension are included in Appendix L and in Figure
2.6 BENEFITS OF DEEP KANs
It is one of our major contributions to generalize the 2-layer KA representations to multiple layers. Although it is challenging to prove the benefits of deeper KANs theoretically, we want to present a concrete example where 3-layer KANs admit smooth representations while 2-layer KANs do not.
We consider fitting a function f 22, x44 = exp( { (sin( T (z} + x²)) + sin(T(=3 + x3)))) where we draw samples (3000 training, 1000 training) uniformly from 1|4 We train a 3L KAN 1,4) and 2L KAN ([4,9,1)) with the LBFGS optimizer for 250 steps, with increasing -= 3, 5, 10, 20, 50 (50 steps for each G). As shown in Figure 2 we see that the 3-layer KAN has smooth representations (as expected, since the parse tree of the symbolic formula has depth 3), while the 2-layer KAN learns highly oscillatory functions on some edges. The 3-layer KAN also achieves lower losses than the 2-layer KAN. While the 3-layer KAN has a small train-test gap, the 2-layer KAN starts to overfit at large grid sizes.
3 KANS ARE INTERPRETABLE
In this section, we show that KANs can be interpretable on synthetic toy tasks and realistic research questions in math and physics.
5

---

Published as a conference paper at ICLR 2025
(a) 3- 3-Layer KAN
(b) 2-Layer KAN
(c) loss
M
Figure 2: Fitting the function f( x2, 2z, 24) = exp(> ( (sin(1((a) + x²)) + sin(T(23 + +2))))-
(a) 3-Layer KAN admits smooth representations. (b) The 2-Layer KAN learns highly oscillatory representations. (c) The 3-layer KAN achieves lower losses and has a smaller train-test gap than the 2-layer KAN.
(4) muitiplication
divisien
deeper compoillion
explsints? x5) sinle +20
30
X2
Figure 3: KANs are interpretable for simple symbolic tasks.
Synthetic toy datasets We first examine KANs' ability to reveal the compositional structures in
symbolic formulas. Three examples are presented in Figure KANs are able reveal the com- positional structures present in these formulas, as well as learn the correct univariate functions. (1)
Multiplication f(z, v) ry. KAN computes via the equation 2ry (x: + y)² (x² y²).
(2) Division of positive numbers fzz y) z/y. KAN computes it via cxp(logr logy). (3)
Deeper compositions f 22, 23 x:4) exp( exp(sin(z)) ( x²) + sin + x²)). Discussion about the implications of these examples is left in Appendix R We also discussed an unsupervised learning
paradigm and how we can convert unsupervised learning to supervised learning by borrowing ideas
from contrastive learning, detailed in Appendix
Application to Mathematics: Knot Theory Knot theory is subject in low -ww-dimensional topology that sheds light on topological aspects of three-manifolde and four-manifolds and has a variety of applications, including in biology and topological quantum computing. In Davies et al. (2021), supervised learning and human domain experts were utilized to arrive at a new theorem relating algebraic and geometric knot invariants. They network attribution methods signature a is mostly dependent on meridinal distance (real imag μ4)) and longitudinal distance X. We show that KANs can not only identify these important variables with much smaller networks and much more automation, but also present some interesting new results and insights.
find that the
We treat 17 knot invariants as inputs and signature as outputs. Similar to the setup in Davies al. (202T), signatures (which are even numbers) are encoded as one-hot vectors and networks are trained with cross-entropy loss. We find that extremely small [17, 1, 14] KAN is able to achieve 81.6% test accuracy (while DeepMind's 4-layer width-300 MLP achieves 78% test accuracy). The [17, 1, 14] KAN (G 3, A 3) has &e 200 parameters, while the MLP has R 3 10 parameters. It remarkable that KANs can be both more accurate and much more parameter efficient than MLPs at the same time. In terms of interpretability.
scale the transparency of each activation according to its magnitude, so it becomes immediately clear which input variables are important without the need for feature attribution (see Figure 4 left top): signature is mostly dependent on py, and slightly

---

Published as a conference paper at ICLR 2025
Supervised learning
Unsupervised learning signatur depandercce
bew 2itecd
variablee Oeed scc: 78.29)
Figure 4: Knot dataset. Supervised mode (left): we rediscover DeepMind's three important vari- ables. Unsupervised mode (right): discover three "new" relations without supervision.
dependent on 14 and X, while dependence on other variables is small. We then train [3, 14] KAN on the three important variables, obtaining test accuracy 78.2% (Figure 4 left bottom). More ablation results and symbolic formula results are included in Appendix
We attempt to make discoveries beyond DeepMind's in the unsupervised learning mode, where we treat all 18 variables (including signature) as inputs. We train 200 networks with different random seeds. They can be grouped into three clusters, with representative KANs displayed in Figure 4 These three groups of dependent variables are (1) rediscovering DeepMind's relation in unsuper- vised learning. (2) cusp volume is by definition of the multiplication of two translations. (3) short geodesic 9r is upper bounded by two times of injecitivy radius Peteessnn (2006).
interesting rediscover several known mathematical relations. The good news is that the results discovered by KANs are probably reliable; the bad news that we have not discovered anything new yet. It is worth noting that we have chosen a shallow KAN for simple visualization, but deeper KANs can probably find more relations if they exist. We would like to investigate how discover more complicated relations with deeper KANs in future work.
that KANs unsupervised mode
Remark: symbolic regression benchmarks We have presented KANs interpretability as an in- teractive tool with human users. However, as a network-based method, its strong capability (in fitting even non-symbolic functions) makes it unfavorable for standard symbolic regression bench- marks. For example, KAN ranks second-to-last GEOBENCH (Anonymous) 2024). whereas the last-ranked one EQL is also network-based model, which has been shown to be useful at least for certain problems (Martius & Lampert 2016 Dugan et al.) 2020) despite its inability to do well on benchmarks. On the one hand, we would like explore ways restrict KANs' hypothesis space so that KANs can achieve good performance on symbolic regression benchmarks. On the other hand, we want to point out that KANs have good features that are not reflected by existing benchmarks: (1) interactivity. It is relatively easier to visualize the training dynamics of KANs, which gives human users intuition on what could go wrong hence facilitating debugging. (2) The ability "discover" new functions. If the ground truth formula contains special function but is not given in the sym- bolic library, SR methods will fail. However, KANs can discover the need for a new function whose numerical behavior suggests maybe it is Bessel function; see Figure 23 (d) for example.
4 KANS ARE ACCURATE
In this section, we demonstrate that KANs are more accurate at representing functions than MLPs in various tasks (regression and PDE solving). When comparing two families of models, it is fair to compare both their accuracy (loss) and their complexity (number of parameters). Moreover, in Appendix Q show that KANs can naturally work in continual learning without catastrophic
7

---

Published as a conference paper at ICLR 2025
forgetting. All experiments reported in the paper are reproducible CPUs usually within minutes, at most in a day. Codes are built based on pytorch Paszke et al. (2019).
Aid 20X2
(X. expisinind +y²)
Rx.y)
No.
Saripio
For i
17
Number of parometars
Number of surandkes
Numberr of paramictors
Number of paromoters
Number of parameters
Figure 5: Compare KANs to MLPs on five toy examples. KANs can almost saturate the fastest scaling law predicted by our theory (a = 4), while MLPs scales slowly and plateau quickly.
Toy datasets In Section 2.3 our theory suggested that test RMSE loss scales as l x N --+++)) N N = 3) with model parameters N. However, this relies on the existence of a smooth Kolmogorov-Amoll representation. As sanity check, we construct five examples we know have smooth KA representations: (1) f(x) = (20x), which is the Bessel function. Since it is a univari- ate function, it can be represented by spline, which is a [1, 1] KAN. (2) f(z y) exp(sin(Tz)) + )²) We know that it can be exactly represented by a [2, 1. 1] KAN. (3) [ = xy. We know from Figure ple f(1 F100) = exp( 100 Σi= sin" (*2:)) which can be represented by a [100, 1. KAN. (5) A four-dimensional example f(21, 22, Z3, T3. = exp(} (sin(x(2²) + +z)) + sin(x(=3 + zi)))) which can be represented by a [4, 4, 1] KAN. The empirical scaling for KANs is quite aligned with theory and outperforms MLPs. Details of training are included in Appendix N
3 that it can be exactly represented by a 2, KAN. (4) A high-dimensional exam-
Special functions In practice, we may not know the existence of KA representations. Special func- tions of more than one variables are such cases, e.g., a Bessel function f(v = (z).. We collect 15 special functions common in math and physics, summarized in Table 9 in Appendix O We find that: (1) KANS are more efficient and accurate in representing special functions than MLPs, as shown in Figure 6 In all cases, KANs have better pareto frontiers than MLPs. (2) Finding (approx- imate) compact KAA representations of special functions is possible, revealing novel mathematical properties of special functions from the perspective of Kolmogorov-Arnot representations. Details are included in Appendix [
Fitting Images We task KANs with three images: (1) The Cameraman picture the standard picture for the image fitting task. (2) The turbulence profile is taken from PDEBench Takamoto et al. 2022), demonstrating high-frequency and fractal behavior typical in scientific computing. (3) Van Gogh's s The Starry Night is quite challenging because it contains fine-grained details as well. In addition to MLPs, We compare KANs with these stronger baselines: (A) MLP with random Fourier features (MLP RFF). Before feeding input coordinates x (z, to the MLP, we first augment them into a higher-dimensional feature space (x) = (x, + (x), ... &N, (x)), where (x) = (cos(s; xinsin x)), i = 1,... ..... Ny. and s; ~ [0, s² (s controls the frequency bias). We choose N, 50 and 3, 30. (B) SIREN (Sitzmann et al. 2020) uses sines as activation functions in MLPs and uses large initialization for the first layer (effectively creating high- high-frequency features). To compare KANs and baselines as fairly as possible, we try two control strategies (same shape or as,c number of parameters) and report both performance (measured by PSNR) and efficiency (wall time). All methods are listed below in T For all baseline models, means their width the same as KAN 1, while 2 means their number of parameters is (approximately) the same as KAN (vG times wider, where G = 10 is the grid size used in KAN 1). We also explore KAN 2, which uses finer grid (G = 100 instead of = 10) for the first layer only (inspired by the idea of random Fourier features in the input layer). The whole image is treated as the training set and there is no test set. All models are trained with the Adam Optimizer for 15000 steps with learning rate decay (5000 V100 GPU.
steps for learning rate 10-3 14 and 10 -5), with batch size 1024,
We list PSNR and training wall time in Table [ and fitted images in Figure We have a few observations from the results: (1) KANs are comparable to or even outperform baseline methods (including SIREN) in terms of PSNR, however with more training time. (2) having random features in the inputs is useful for MI.Ps, especially high-frequency random features (s = 30 outperforms
8

---

Published as a conference paper at ICLR 2025
10-1
ellipj
ellipkinc
ellipeinc
yv
RM5E
30-3
-8-
KAN test
MLP trath
10
MLP
10-1
kv
Iv
Ipmv m 0
Ipmv_
Ipmv m 2
RM15E
10
10.0
10.4
sph harm m n
sph harm m 1 1_
sph harm m O_n_2
sph harm m 1
sph harm m n
RMS
10
1044
10
10
10
10a
10t
10
10a
107
10r
1.0
10t
10
1.0
104
16
10
102
10"
number parameters
number of parameters
number of parameters
number of parameters
number of paramnteess
Figure 6: Fitting special functions. We show the Pareto Frontier of KANs and MLPs in the plane spanned by the number of model parameters and RMSE loss. Consistently accross all special func- tions, KANs have better Pareto Frontiers than MLPs. The definitions of these special functions are Table g
MEPI
MLP J2T
MLP JPP 16
S
MUS
Figure 7: Image fitting task PDE solution from PDEBench Takamoto et al. (022) KAN outper- forms baseline methods in terms of PSNR.
== 3). We may also understand KANs superior performance as being good at generating random features in early layers. By changing the grid size in the first layer from G 10 to G 100 (KAN 2), PSNR significantly increases with little additional overhead in training time. We show the turbulence profile in Figure Results of the other two images can be found in Appendix A
Solving partial differential equations (PDEs We consider Poisson equation with zero Dirichlet boundary data. For Ω = [-1, 1]2. consider the PDE Uez + dgy = f with zero boundary condition. We consider the data ʃ = 21 + 4 sia() sin(my²) + 2n sin (wx) cos(my²) for which u = sin (TTar) sin(my²) ) is the true solution. We use the framework of physics-ifformed neural networks Karniadkkss et al. (202I) to solve this PDE, with the loss function given by losspa: aloss; + loss; == n. >x
(PINNs) Raissi et al.
> Max (4) (2) f(z))2 + 1 Ene u² where use loss; to denote the interior loss, discretized and evaluated by a uniform sampling of n, points 2 = (z,, 3/) inside the domain, and similarly we use loss, to denote the boundary loss, discretized and evaluated by uniform sampling of n& points the boundary.
no
0.01 is the hyperparameter balancing the effect of the two terms. KANs are shown to have Pareto Frontiers than MLPs for this simple example.

---

Published as a conference paper at ICLR 2025
Table 1: Image Fitring: Comparing various methods
PSNR f
Training Wall Time (8) 4
Method
width
Cam
Turb
Star
Cam
Turb
Star
[2.128, 128.88888..8
KAN
32.06
62.45
50.55
1800
1721
1715
G ==[10010,10010.10
12122.128,128,112
KAN
45.76
82.68
71.82
1809
1734
1727
G -=[100,10.10.10.100
MLP
[2,128, 12,,8,,,,,,,,,2,,
20.76
27.18
18.28
162
92
91
MLP 2
4,404,404,404,404,
22.09
33.14
18.96
182
110
110
SIREN
12,128,12,111 12,,1888118,,,,
27.34
49.51
29.88
254
226
232
SIREN 2
2,404,404, 404,44444
30.79
51.94
53.05
407
400
404
MLP_ BP__RFFTT 1 (8
2.128, 128,8,28,,888,, 128,
22.17
33.58
19.19
176
96
96
MLP RFF (s = 3)
2,404,404,404,4444
24.73
42.57
22.00
192
117
118
MLP RFF (s 30)
12.128,128 128, 128,
23.92
43.07
22.68
174
96
97
MLP RFF 2 (s 30)
2,404,404,40444000
26.26
46.43
28.62
195
117
121
crorr squareo
La squaree
H FIRT
stop
stop
Number parameters
Number of parameters
Figure 8: The PDE example. We plot L2 squared and HI squared losses between the predicted solution and ground truth solution. First and second: training dynamics losses. Third and fourth: scaling laws of losses against the number of parameters. KANs converge faster, achieve lower losses, and have steeper scaling laws than MLPs.
More complicated PDEs. We test more PDE examples in Appendix B showing that KANs can achieve reasonable performance more complicated PDEs. However, we want to note that KANs are slightly slower than MIPs to train in terms of wall time (reported in Appendix BB. despite their smaller number of parameters. The point we want to make with Figure Bis that KANs can achieve the theoretical scaling law in this PDE example (beyond function fitting). but this result should not be interpreted as an immediate real-world improvement. Also, both the LBFGS optimizer and the grid extension technique required to achieve the theoretical scaling law, but in practice, people use the Adam optimizer and do not need grid extension for MLPs, which we explore in Appendix
5 CONCLUSIONS
Inspired by the Kolmogorov-Arool representation theorem, we propose the Kolmogorov-Amold Networks (KANs) promising alternatives MLPsOur contributions three-fold: (1) put the KA theorem in the perspective of modern machine learning, relating to MLPs, and generalize the representation from two-layer multiple layers via the KAN layers introduced, greatly en- hancing expressive power. (2) we show that KANS are interpretable, serving as a useful tool for scientific discoveries. (3) we show that KANs are accurate and have nice scaling laws via theory and experiments. The major limitation of this work, however, is that our numerical examples fo- cus on various aspects of science and are relatively scall-scale. The scalability and extensibility of KANs for large-scale machine-learing tasks are left as future work. We also acknowledg, that the similarities and differences between MLPs and KANs require more study, both theoretically and empirically. For example, reasonable criticism of KANs is that they can be rewritten as MLPs or the other way around since the notion of "edge" vs "node" is somewhat dual. Future work should aim to better clarify similarities and differences from the perspective of optimization, generalization, etc. For example, a recent preprint Wang et al. (2024b) shows that although KANs and MLPs are both universal approximators, KANs have fewer spectral biases than MLPs.
10

---

Published as a conference paper at ICLR 2025
Acknowledgement We would like to thank Mikail Khona, Tomaso Poggio. Pingchuan Ma, Rui Wang. Di Luo, Sara Beery, Catherine Liang, Yiping Lu, Nicholas H. Nelsen, Nikola Kovachki, Jonathan W. Siegel. Hongkai Zhao, Juncai He. Shi Lab (Humphrey Shi, Steven Walton, Chuanhao Yan) and Matthieu Darcy for fruitful discussion and constructive suggestions. 7.I., F.R., J.H., M.S. and M.T. are supported by IAIFI through NSF grant PHY-2019786. The work of FR is in addition supported by the NSF grant PHY-2210333 and by startup funding from Northeastern University. Y.W and T.H are supported by the NSF Grant DMS-2205590 and the Choi Family Gift Fund. V. and M. S. acknowledge support from the U.S. Office of Naval Research (ONR) Multidisciplinary University Research Initiative (MURI) under Grant No. N00014-20-1-2322 on Robust Photonic Materials with Higher-Order Topological Protection.
REFERENCES
Accessed: 2024-04-19.
Elihu Abrahams, PW Anderson, DC Licciardello, and TV Ramakrishnan. Scaling theory of local- ization: Absence of quantum diffusion in two dimensions. Physical Review Letters, 42(10):673,
1979.
Rishabh Agarwal, Levi Melnick, Nicholas Frosst, Xuezhou Zhang, Ben Lengerich, Rich Caruana, and Geoffrey Hinton. Neural additive models: Interpretable machine learning with neural nets. Advances in neural information processing systems, 34:4699-4711 2021.
Fangzhao Alex An, Karmela Padavić, Eric Meier, Suraj Hegde, Sriram Ganeshan, JH Pixley, Smitha Vishveshwara, and Bryce Gadway. Interactions and mobility edges: Observing the gen- eralized audré model. Physical review letters, 126(4):040603, 2021.
Philip W Anderson. Absence of diffusion in certain random lattices. Physical review, 109(5):1492, 1958.
Anonymous. Geobench: new benchmark on symbolic regression with geometric expressions. In Submitted to The Thirteenth International Conference on Learning Representations, 2024. URL under review.
/ttpps://ppenreview net forum Fo
Shayan Aziznejad and Michael Unser. Deep spline networks with control of lipschitz regularity. In ICASSP 2019-2019 IEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1242-3246. IEFE, 2019.
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. arXiv preprint arXiv:2/02.06701, 2021.
Peter L Bartlett, Nick Harvey, Christopher Liaw, and Abbas Mehrabian. Nearly-tight ve-dimension and pseudodimension bounds for piecewise linear neural networks. Journal of Machine Learning Research, 20(63):1-17, 2019.
J Biddle and S Das Sarma. Predicted mobility edges in one-dimensional incommensurate optical lattices: An exactly solvable model anderson localization. Physical review letters, 104(7); 070601, 2010.
Garrett Bingham and Risto Miikkulainen. Discovering parametric activation functions. Nenworks, 148:48-65 2022.
Neural
Pakshal Bohra, Joaquim Campos, Harshit Gupta, Shayan Aziznejad, and Michael Unser. Learning activation functions in deep (spline) neural networks. IEEE Open Journal of Signal Processing, 1:295-309, 2020.
Yifan Chen, Thomas Y Hou, and Yixuan Wang. Exponentially convergent multiscale finite element method. Communications Applied Mathematies and Computation, pp. 1-17, 2023.
Miles Cranmer. Interpretable machine learning for science with pysr and symbolicregression. jl. arXiv preprint arXiv:2305.0/580 2023.
Jessica Craven, Vishnu Jejjala, and Arjun Kar. Disentangling deep learned volume formula. JHEP, 06:040, 2021. doi: 10. (007//HFP0//22211//

---

Published as a conference paper at ICLR 2025
Jessica Craven, Mark Hughes, Vishnu Jejjala, and Arjun Kar. Illuminating new and known relations between knot invariants. 11 2022.
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models. arXiv preprins arXiv:2309.086m 080000 2023.
George Cybenko. Approximation by superpositions a sigmoidal function. Mathematics of control, signals and systems, 2(4):303 -14, 1989.
Alex Davies, Petar Veličković, Lars Buesing, Sam Blackwell, Daniel Zheng, Nenad Tomašev, Richard Tanbur, Peter Battaglia, Charles Blundell, András Juhász, et al. Advancing mathematics by guiding human intuition with ai. Nature, 600(7887):70-74, 2021.
Carl De Boor. A practical guide to splines, volume 27. springer-verlag New York, 1978.
Wojeiech De Roeck, Francois Huveneers, Markus Müller, and Mauro Schiulaz. Absence of many- body mobility edges. Physical Review B, 93(1):014203, 2016.
Ronald Verere Ralph Howard, and Charles Micchelli. Optimal nonlinear approximation.
Manuscripta mathematica, 63:469 478, 1989.
Ronald A Vore, George Kyriazis, Dany Leviatan, and Vladimir M Tikhomirov. Wavelet com- pression and nonlinear n-widths. Adv. Comput. Math., 1(2):197214, 1993.
Renáta Dubcáková. Eureqa: software review. Genetic Programming and Evolvable Machines, 12: 173-178, 2011. URL
Owen Dugan, Rumen Dangovski, Allan Costa, Samuel Kim, Pawan Goyal, Joseph Jacobson, and Marin Soljačić. Occamnet: fast neural model for symbolic regression at scale, arXiv preprint arXiv:2007.10784. 2020.
Alexander Duthie, Sthitadhi Roy, and David E Logan. Self-consistent theory of mobility edges in quasiperiodic chains. Physical Review B, 103(6):L000201, 2021.
Nelson Elhage, Tristan Hume, Catherine Olsson, Neel Nanda, Tom Henighan, Scott Johnston, Sheer EIShowk, Nicholas Joseph, Nova DasSarma, Ben Mann, Danny Hermandez, Amanda Askell, Ka- mal Ndousse, Andy Jones, Dawn Drain, Anna Chen, Yuntao Bai, Deep Ganguli, Liane Lovitt, Zac Hatfield-Dodss, Jackson Kernion, Tom Conerly, Shauna Kravec, Stanislav Fort, Saurav Ka- davath, Josh Jacobson, Eli Tran-Johnson, Jared Kaplan, Jack Clark, Tom Brown, Sam McCan- dlish, Dario Amodei, and Christopher Olah. Softmax linear units. Transformer Circuits Thread, 2022a. http://rsanmaameeeccccuuss/////p
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodss Robert Lasenby, Dawn Drain, Carol Chen, al. Toy models of superposi- tion. arXiv preprini urXiv:2209. 0022099.0060 10652, 2022b.
Daniele Fakhoury, Emanuele Fakhoury, and Hendrik Speleers. Exsplinet: An interpretable and expressive spline-based neural network. Neural Networks, 152:332-346, 2022.
Sriram Ganeshan, JH Pixley, and S Das Sarma. Nearest neighbor tight binding models with exact mobility edge in one dimension. Physical review letters, 1 114(14):14660) 2015.
Federico Girosi and Tomaso Poggio. Representation properties of networks: Kolmogorov's 's theorem is irrelevant. Neural Computation, (14):465-66) 1989.
Mitchell A Gordon, Kevin Duh, and Jared Kaplan. Data and parameter scaling laws for neural machine translation. AnA Rolling Review May 2021, 2021.
/ IKA 7MLxsLSu
Mohit Goyal, Rajan Goyal, and Brejesh Lall. Learning activation functions: A new paradigm for understanding neural networks. arXiv preprint (0X.v:/90600952 2019.
Sergei Gukov, James Halverson, Fabian Ruchle, and Piotr Sułkowski. Learning to Unknot. Mach. Learn. Sci. Tech. 2(2):025035, 2021. doi: 10. 11100/22222111/3/ae911
12

---

Published as a conference paper at ICLR 2025
Sergei Gukov, James Halverson, Ciprian Manolescu, and Fabian Ruchle. Searching for ribbons with machine learning, 2023.
Sergei Gukov, James Halverson, and Fabian Ruehle. Rigor with machine learning from field theory to the poincaréconjeectur. Nature Reviews Physics, 2024. doi: 10. 1038///2354-024-007700-44
1tt.://....r..1.111/.//1.//////0 222214102132222244--222 024 00709-0
Simon Haykin. Neural networks: a comprehensive foundation. Prentice Hall PTR, 1994.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Y.H. He. Machine Learning in Pure Mathematics and Theoretical Physics. G Ref-
crence.Information and Interdisciplinary Subjects Series. World Scientific, 2023. ISBN 9781800613699
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010. 2020.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, arXiv preprint arXiv: 1tt:::::7220000000 2017.
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are uni- versal approximators. Neural networks, 2(5):359-366 -66, 1989.
Joel Horowitz and Enno Mammen. Rate-optimal estimation for general class of nonparametric regression models with unknown link functions. 2007.
Mark C Hughes. A neural network approach to predicting and computing knot invariants. Journal of Knot Theory and Its Ranifications, 29(030)0050005, 2020.
Aysu Ismayilova and Vugar E Ismailov. On the kolmogorov neural networks. Neural Networks, pp. 106333, 2024.
Sajeev John. Strong localization of photons in certain disordered dielectric superlattices. Physical review letters, 58(23):2486, 1987.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
George Em Karniadakis, Ioannis G Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang- Physics-informed machine learning. Nature Reviews Physics, 3(6):422-440, 2021.
L. H. Kauffmaan, N. E. Russkikh, and I. A. Taimanov. Rectangular knot diagrams classification with deep learning, 2020.
Michael Kohler and Sophie Langer. On the rate of convergence of fully connected deep neural network regression estimates. The Annals of Statistics, 49(4):2231-2249, 2021.
Mario Köppen. On the training of a kolmogorov network. In Artificial Neural Networks -CANN 2002: International Conference Madrid, Spain, August 28-30, 2002 Proceedings 12, pp. 474-
479. Springer, 2002.
Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Arizzadenesheli, Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces with applications to pdes. Journal of Machine Learning Research, 24(89):1-97, 2023.
Ad Lagendijk. Bart van Tiggelen, and Diederik S Wiersma. Fifty years of anderson localization. Physics today, 62(8):24-22) 2009.
13

---

Published as a conference paper at ICLR 2025
Yoav Lahini, Rami Pugatch, Francesca Pozzi, Mare Sorel, Roberto Morandotti, Nir Davidson, and Yaron Silberberg. Observation of localization transition in quasiperiodie photonic lattices. Phys- ical review letters, 103(1):013901, 2009.
Ming-Jun Lai and Zhaiming Shen. The kolmogorov superposition theorem can break the curse of di- mensionality when approximating high dimensional functions. arXiv preprint 12X1v:2112.09663, 2021.
Pverre-Emmanee Leni, Yohan D) Fougerolle, and Frédéric Truchetet. The kolmogorov spline net- work for image processing. Image Processing: Concepts, Methodologies, Tools, and Applica- fions, pp. 54-78. IGI Global, 2013.
Xiaopeng Li, Sriram Ganeshan, JH Pixley, and S Das Sarma. Many-body localization and quantum nonergodicity model with single-particle mobility edge. Physical review letters, 115(18): 186601, 2015.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.0888 2020.
Zongyi Li, Hongkai Zheng, Nikola Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Kamyar Arizzadenesheli, and Anima Anandkumar. prysics-informed neural operator for learning partial differential equations. ACM/JMS Journal of Dala Science, 2021.
Henry W Lin, Max Tegmark, and David Rolnick. Why does deep and cheap learning work so well?
Journal of Statistical Physics, 168:1223-12477 2017.
Ji-Nan Lin and Rolf Unbehauen. On the realization of a kolmogorov network. Neural Computation, 5(1):18-20, 1993.
Ziming Liu, Eric Gan, and Max Tegmark. Seeing is believing: Brain-inspired modular training for mechanistic interpretability. Entropy, 26(1):41, 2023.
Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature machine intelligence, 3(3):218 229, 2021.
Georg Martius and Christoph H Lampert. Extrapolation and learning equations. arXiv preprint arXiv:1610.02995. 2016.
Haydn Maust, Zongyi Li, Yixuan Wang, Daniel Leibovici, Oscar Bruno, Thomas Hou, and Anima Anandkumar. Fourier continuation for exact derivative computation in physics-informed neural operators. arXiv preprint arXiv:2211.15960. 2022.
Levi McClenny and Ulisses Braga-Neto. Self-adaptive physics-informed neural networks using a soft attention mechanism. arXiv preprint arXiv:2009.044 04544, 2020.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 11-17359-173722 2022.
Eric Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling. In Thirty-seventh Conference on Neural Information Processing Systems, 2023a. URL https:// http::::///penrviee...oo net wttp://wwwww.eeee/euudidid----ee wow.mriddrrdaaoo 2
Eric Michaud, Ziming Liu, and Max Tegmark. Precision machine learning. Entropy, 25(1):175, 2023b.
Hadrien Montanelli and Haizhao Yang. Error bounds for deep relu networks using the kolmogorov- aroold superposition theorem. Neural Networks, 129:1-6, 2020.
Terrell N. Mundhenk, Mikel Landajuela, Ruben Glatt, Claudio P. Santiago, Daniel faissol, and Brenden K. Petersen. Symbolic regression via deep reinforcement learning enhanced genetic programming seeding. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https: ttttp:://operzeviee. net/forurm d=t jw
14

---

Published as a conference paper at ICLR 2025
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretabilityy. In The Eleventh International Conference on Learn- ing Representations, 2023. URLktttss://oeeneeeee
net 2xtp
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint S2NV::2209.18992 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Petersen. Riemannian Geometry. 2006. ISBN 9780387294032. URL https: o watoggee.com/oooooo
Graduate Texts in Mathematics. Springer New York,
9cekXd552hFC
Tomaso Poggio. How deep sparse networks avoid the curse of dimensionality: Efficiently com-
putable functions are compositionally sparse. CBMM Memo, 10:2022, 2022.
Tomaso Poggio, Andrzej Banburski, and Qianli Liao. Theoretical issues in deep networks. Proceed- ings of the National Academy of Sciences, 117(48):30039330M 30045, 2020.
Michael Poluektov and Andrew Polar. A new iterative method for construction of the kolmogorov- arnold representation. arXiv preprint arXiv:2305.0811 2023.
Maziar Raissi, Paris Perdikaris, and George Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-707, 2019.
Prajit Ramachandran, Barret Zoph. and Quoc V Le. Searching for activation functions. arXiv preprint arXiv:1710.055 2017.
Fabian Ruehle. Data science applications to string theory. Phys. Rept., 839:1-117, 2020. doi: 10.10166j ptw.pep22019..090000.
Johannes Sehmidt-HHibber Nonparametric regression using deep neural networks with relu activation function. 2020.
Johannes Schmidt-Hieber, The kolmogorov- representation theorem revisited. Neural net- works, 197:119-126, 2021.
Mordechai Segev, Yaron Silberberg, and Demetrios Christodoulides. Anderson localization of light. Nature Photonics, 7(3):197-204, 2013.
Utkarsh Sharma and Jared Kaplan. A neural scaling law from the dimension of the data manifold. arXiv preprint arXiv:2004. 10802, 2020.
Jonathan W Siegel. Optimal approximation rates for deep relu neural networks sobolev and besov spaces. Journal of Machine Learning Research, 24(357):1-52, 2023.
Jonathan W Siegel. Sharp lower bounds on the manifold widths of sobolev and besov spaces. arXiv preprint 00V::2402.044077 2024.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in neural information processing systems, 33:7462-7473, 2020.
Huan Song, Jayaraman J Thiagarajan, Prasanna Sattigeri, and Andreas Spanias. Optimizing kernel machines using deep learning. IEEE transactions on neural networks and learning systems, 29 (11):5528-5540. 2018.
Jinyeop Song, Ziming Liu, Max Tegmark, and Jeff Gore. A resource model for neural scaling law. arXiv preprint arXiv:2402.00 20:..0242..111 05/000114444 164, 2024.
15

---

Published as a conference paper at ICLR 2025
David A Sprecher and Sorin Draghici. Space-filling curves and kolmogorov superposition-basee neural networks. Neural Networks, 15(1):57-67, 2002.
Makoto Takamoto, Timothy Praditia, Raphael Leiteritz, Daniel MacKinlay, Francesco Alesiani, Dirk Pflüger, and Mathias Niepert. Pdebench: An extensive benchmark for scientific machine learning. Advances in Neural Information Processing Systems, 15:1596-161,, 2022.
David Thouless. A relation between the density of states and range of localization for one dimen- sional random systems. Journal of Physics C: Solid State Physics, 5(1):77, 1972.
Silviu-Marian Udrescu and Max Tegmark. Ai feynman: A physics-inspired method for symbolic regression. Science Advances, 6(16):eaay6611 2020.
Silviu-Marian Udrescu, Andrew Tan, Jiahai Feng, Orisvaldo Neto, Tailin Wu, and Max Tegmark. Ai feynman 2.0: Pareto-optimal symbolic regression exploiting graph modularity. Advances in Neural Information Processing Systems, 33:4860-4871, 2020.
Sachin Vaidya, Christina Jörg, Kyle Linn, Megan Goh, and Mikael Rechtsman. Reentrant delo- calization transition in one-dimensional photonic quasicrystals. Physical Review Research, 5(3): 033170, 2023.
Z Valy Vardeny, Ajay Nahata, and Amit Agrawal. Optics of photonic quasicrystals. Nature photon- ics, 773):177-187, 2013.
Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations, 2023. URL https:
/oppnreview wwt.://oormmm/daaaaaaaaaaaa
Sifan Wang, Shyam Sankaran, and Paris Perdikaris. Respecting causality is all you need for training physics-informed neural networks. arXiv preprint arXiv:2203.07404, 2022.
Sifan Wang, Bowen Li, Yuhan Chen, and Paris Perdikaris. Piratenets: Physics-infommn informed deep learn- ing with residual adaptive networks. arXiv preprint arXiv:2402.00326, 2024a.
Yixuan Wang, Jonathan Siegel, Ziming Liu, and Thomas Y Hou. On the expressiveness and spectral bias of kans. arXiv preprint arXiv:2410.01803, 2024b.
Yongji Wang and Ching- Yao Lai. Multi-stage neural networks: Function approximator of machine precision. Journal of Computational Physics, pp. 1 112865, 2024.
Yucheng Wang, Xu Xia, Long Zhang, Hepeng Yao, Shu Chen, Jiangong You, Qi Zhou, and Xiong- Jun Liu. One-dimensional quasiperiodic mosaic lattice with exact mobility edges. Physical Re- view Letters, 125(19):196604 2020.
Yucheng Wang, Xu Xia, Yongjian Wang, Zuohuan Zheng, and Xiong-Jun Liu. Duality between two generalized aubry-andré models with exact mobility edges. Physical Review 103(17): 2021.
Hongyi Xu, Funshing Sin, Yufeng Zhu, and Jemej Barbič. Nonlinear material design using principal stretches. ACM Transactions on Graphics (TOG), 34(4):1-11, 2015.
Jinchao Xu and Ludmil Zikatanov. Algebraic multigrid methods. Acta Numerica, 26:591-721, 2017.
Dmitry Yarotsky. Error bounds for approximations with deep relu networks. Neural Networks, 94: 103-114, 2017.
Bing Yu et al. The deep ritz method: deep Baaming-basee numerical algorithm for solving varia- tional problems. Communications in Mathematics and Statistics, 6(1):1-12, 2018.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander Smola. Deep sets. Advances in neural information processing systems, 30, 2017.
16

---

Published as a conference paper at ICLR 2025
Shijun Zhang, Zuowei Shen, and Haizhao Yang. Neural network architecture beyond width and depth. Advances in Neural Information Processing Systems, 35:5669-566 2022.
Shumao Zhang, Pengchuan Zhang, and Thomas Y Hou. Multiscale invertible generative networks for high-dimensional bayesian inference. In International Conference Machine Learning, pp- 12632-12641. PMLR, 2021.
Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas. The clock and the pizza: Two stories mechanistic explanation of neural networks. In Thirty-seventh Conference on Neu- ral Information Processing Systems, 2023, S5wmbQclwe
Xin-Chi Zhou, Yongjian Wang, Ting-Fung Jeffrey Poon, Qi Zhou, and Xiong-Jun Liu. Exact new mobility edges between critical and localized states. Physical Review Letters, 131(17):17601, 2023.
17

---

Published as a conference paper at ICLR 2025
Orignal
STIENT
KAM
MLPI
887 (e
(e00))
Cameraman
SIRENG PA
KANI 20-41/1
MLFI
HLP RFP
MLP KFP (o-30) oraa
2E122
KANI
MLPI
M.P.RITI
MLP RIP (pe32) oee-taa
Turbulence
KANS
RFF (100)
Crighnal
(M0))
MLP RLP (e-d))
Starry Night
KANG
MUPS C1
MLP 2
RFF (6-38)
Figure 9: Comparing various methods for fitting the picture of Cameraman (top), turbulent flow
(middle,from PDEBench Takamoto et al. (2022)) and Van Gogh's the starry night (bottom).
A ADDITIONAL IMAGE FITTING EXAMPLES
B ADDITIONAL PDE EXAMPLES
In Section we showed KANs superior performance over MLPs for solving a 2D Poisson equa- tion with smooth solution. To really understand the capabilities and limitations of KANs, we test KANs by taking three (relatively more) challenging PDEs, as suggested by reviewers: (1) Poisson equations with high-frequency solutions, to test KANs ability to model high-frequency modes. (2) Allen-Cahn equation, to test KANs' ability to model temporal phenomenon and capture sharp tran- sitions. (3) Darcy flow, to test KANs ability to model random structures (e.g., Darcy flow can be used to model porous media). The goal of this section is show that KANs (as they are) can achieve reasonable performance for these challenging PDEs, rather than attempting to establish SOTA per- formance. Indeed, PDE modeling with neural networks is huge field and many techniques (e.g..
18

---

Published as a conference paper at ICLR 2025
adaptive weights (McClenny & Braga- Neto 2020), causality training (Wang et al. 2022 the gat- ing mechanism PirateNet (Wang et al. 2024a)) have been developed that can be very useful to improve KANs in the future.
Remark on the problem setup: In Figure 8 we used the LBFGS optimizer and the grid extension technique to achieve the theoretical scaling law. However, it is more common to use the Adam optimizer and the grid extension technique too specific to KANs. To make fair comparisons in typical user cases, below we use the Adam optimizer and do not grid extension (the grid size is fixed once chosen), to avoid the possibility of any of our algorithmic choices favoring KANs. Said that this does not mean one should avoid using these tricks in practice when the goal is to optimize results rather than make fair comparisons. We note that KAN's training wall time is much reduced from the last version because time but is unnecessary for PDE cases).
disable the symbolic front (which takes up most of the training
B.1 POISSON EQUATION WITH HIGH-FREQUENCY SOLUTIONS
To test KANs' ability approximate high-frequency PDE solutions, revisit the Poisson equation in the main text but impose high-frequency solutions. To be specific, we consider the Poisson equation
Uax + Myy f. E [-1,1]
(12)
with zero boundary condition (u(2--1) = u(x, 1) = u0-1,y) y) u(1.v) = 0) and f which has the solution u(z, y) (in(nTz))in((xz)). We train our models (listed in Table 2 using Adam optimizers with learning rate 10 for 1000 steps except for 10000 steps for MI.P (10x training). The training loss the PINN loss losspte = aloss;+ =
-2n 22n? (issin(nnz)ss(((((
a n. Ex (2,) + (2,) f(=))1 + ns 2hh u² where we use loss; to denote the interior loss. discretized and evaluated by a uniform sampling of n. = 512 points 2; = (x,, V1) inside the domain, and similarly we use loss, to denote the boundary loss, discretized and evaluated by uni- 0.01 is the hyperparameter balancing the
form sampling of = 51 points on the boundary. effect of the two terms. It is clear that larger means the solution is more high-frequency and hence more challenging. We visualize predictions by models in Figure I0 and list their relative Eg error and training wall time in Table 2 We have few observations: MLPs, MLP RFFs (MLP with random Fourier features) achieve qualitatively good predictions given proper hyperparameters. (2)
All the models, i.e., KANs,
Fourier features in MLP RFFs can be harmful to training. Similarly, KANs become unstable with large grid sizes and/or depths. This is in contrast to the results in image fitting in Appendix A Our explanation is that PINN losses have quite sharp loss landscapes, and adding high-frequency features will only make things worse. (3) KANs have slightly higher training time than MLPs, which is due to the recursive evaluations of splines, and derivatives in the PINN objective makes this problem even more severe. As we will see below, the observations drawn in this example apply to the two equations below as well.
ground
training)
(Sm3)
MLP. RFF
KAN [2.110,1]
KAN [2,10,1] (G-20)
KAN [2,100,11]
KAN [2,100.1] (G-20) KAN IM.1100.001101110-110
KAN 12.10,10..0,111 (G-20)
Figure 10: Comparing various methods solving 2D Poisson equation with high-frequency solution (n = 4).
19

---

Published as a conference paper at ICLR 2025
Method
71
11
(2 error
time (s) 4
12 error
time (8) 4
l2 error
time (s)
MLP
0.003
0.027
19
0.553
19
MLP (10x training)
0.001
184
0.328
184
0.022
202
MLP P_P___FFF_F (s 3)
0.001
254
0.001
233
0.084
232
MLP_RFF RFF (s 30)
1.000
250
0.999
242
0.934
249
KAN [2.10.1] G 10
0.006
28
0.135
29
0.729
24
KAN [2,10,1] G 20
0.221
33
0.082
33
0.295
34
KAN 12,100. 10
0.001
51
0.006
52
0.099
51
KAN [2,100.1]
0.326
0.135
71
0.090
KAN 12.10.10.10.] G 10
0.012
89
0.117
95
0.576
92
KAN [2,10,10,10,1] =
0.995
127
0.993
130
0.982
125
Table 2:
Comparing various method on solving 2D Poisson equations.
All MLPs (including
MLP RFFs) have shapes [2, 128, 128,1.
M
Figure 11: Solving the ID Allen-Cahn equation.
B.2 ALLEN-CAHN EQUATION
To test KANs ability to solve temporal PDEs and model phase transitions, we consider the one- dimensional Allen-Cahi equation with the periodic boundary conditions and the quartic double-well potential energy, formulated as below
Ue 0.0001 Mae
5(w² u) 0, E [],, E [0,
u(x, 0) ==cosTTx) = uz(x),
(13)
u( 11,t) =u(1, f),
M1 (1,,) u2 (1,
Since do not have exact solution, reference solution is obtained via the direct Euler forward method (1000 mesh points in space and in time). Although this temporal equation seems innocuous, using standard PINN training (using MLPs) lead to problem the solution would collapse to a zero solution very quickly in time! To solve this issue, Wang et al. (2022) proposes causal training where the temporal domain is divided into several blocks. Each block corresponds to a separate PINN and these PINNs are trained sequentially in time, where the previous block is used to initialize the next block. Each block covers At 0.1, so there are blocks in total. The training loss the
PINN loss losspde = loss, + ayloss,, + a,loss, = a. n, Σ |4, 0.0001 + 5(2 -2) + as n. Th (u(1, t4) u( t,)) (1, u2 (-1,t,)) + at Ei (u(x,, 0) un (z
where we use loss; denote the interior loss, discretized and evaluated by a uniform sampling of n, = 512 points = (z,, y:) inside the domain, and similarly we use loss; to denote the boundary loss, discretized and evaluated by uniform sampling of n4
points on the boundary. loss, to 51 points on the boundary. We choose a; = 1, Ob 1, a, = 100. We train each temporal block with Adam optimizer with a learning rate 10-3 for 1000 steps. We show in Figure II their prediction profiles. With 1000 training steps, KANs have already learned good qualitative evolution (although with some imperfections). Training KANs for 10000 steps probably helps, but that will take about 10h to train so we did not try this given the limited time during rebuttal. With 1000 training steps. MLPs do not learn the correct qualitative evolution, but adding training steps 10000 makes MI.Ps learn
denote the initial profile, discretized and evaluated by uniform sampling of
20

---

Published as a conference paper at ICLR 2025
Model
2 error
Training wall time (s)
KAN [2.5.5.1] G
3.4 10"
2801
KAN [2,5,5,1]
10
3.9 10-3 3
2831
MLP
1.5 x 10
478
MLP (10x training)
3.9 x 10
4766
MLP RFF (s 30)
8.0 x
599
Table Comparing various models on the ID Allen-Cahn equation. MLP
All MLPs including
MFF_RFFs RFFs have the shape 12.228122,128.111
the evolution quite accurately. For MLP with high-frequency random features (s 30), the training curve fails to decrease, which is similar to the observation in the Poisson case in Appendix B.]
B.3
DARCY FLOW
(2
[2,100,1]
12.10.10.10.1
12,10,11
[2.100.1]
KAN 12..0.00000
MLP 12.128.128, 128.1
MUP RFF 12,128,128, 128,188,,, (S=3)
MUP RFF (2.128,128 12.22,222,18.11 (5-30)
Figure 12: Prediction of various models Darcy flow
We use Darcy flow to test KANs ability model random media (modeled Gaussian mixtures). The equation is similar to the Poisson equation, but the permeability axx) can be spatially dependent (x (x, y)):
21

---

Published as a conference paper at ICLR 2025
Model
12 error 4
Training wall time (s) I
KAN [2,10,1] G
3.9 10
71
KAN [2,10,1] 10
1.3 x 10
66
KAN [2,10,1] 20
3.9 x 10
66
KAN [2,100.1] =
1.7 x 10
81
KAN 2,100,1] 10
4.3 x 10
107
KAN [2,100,]] 20
1.9 10-2
136
KAN [2.10.10.10.10.]]C:
8.5 10-5
123
KAN 12.10.10,1011001 10
1.2
123
KAN 12.10,10,101101]] G = 20
1.3
125
ML.P
3.0 10
30
MLP (10x training)
4.5 10-6
277
MLP_ RFF (s 3)
5.9 x 10-6
31
MLP RFF (s = 30)
4.0 x 10
31
Table 4: Comparing various models on darey flow. All MLPs including MI.P_RFI have the shape
1,.128.28,128.1111
V (a(x) Vu(x))
(xx) (x)) X 82,
N.
2 (y wo.t)
a(x) = 1 +
exp
Fa Da, ~ Ul-,,,,,, ~ U(0.1.0.3)
20 a,
(=1
N
[
2 + (y)
(x
exp
2 20
Fu,i Vuss ~ U[-1,1), U10.1, 0.3.
(14) Our setup is exactly the same as the Poisson equation in B.1 except that the ground truth solution is different, and the left differential operator is slightly different from simple Laplacian. We visualize prediction solutions in Figure 12 and report errors and training time Table 12
B.4 DISCUSSION
From the three examples above, we conclude that KANs can produce reasonable performance for PDE solving, but face few challenges that should be addressed to make them competitive with SOTA PDE methods: (1) Slow training. In the image fitting task, we find that KANs typically have 2* more wall time than MLPs of same sizes, due to the recursive computation of splines. However, the slowdown factor is even worse for PDE solving. Potential solutions include more efficient computations of splines (e.g., pre-computing spline coefficients), or using other activation functions (e.g., Fourier bases or radial basis functions) to avoid recursive evaluations. (2) Stability at large depths and large grid sizes. In image fitting, we find that larger depths and larger grids lead to better performance, However for PDEs, shallow but wide KANs typically perform better than deep KANs. When grid size is small, increasing it can gain more accuracy; however, when grid size reaches, say, 20, training can be totally messed up. Potential solutions include leveraging gating mechanisms in PirateNet (Wang al. 2024a), adding residual connections (He et al. (0166), and trying other regimes (e.g., deep Ritz method) instead the PINN loss.
C MNIST
To test KANs scalability for high-dimensional datasets, we train KANs on MNIST. We normalize the pixel values into [0.1], and flatten the 28x28 image into a 784-dimensional vector. We train models (listed in Table 5) with the Adam optimizer (10-2 learning rate) for 2000 steps on the cross- entropy loss, with batch size 1024. The whole training dataset (60000) and test dataset (10000) are used to evaluate train/test loss/acc. We report these metrics in Table 5
There are a few observations: (1) the shape 1784,100,100 10] (1 hidden layer of size 100) is optimal both for MLP and for KAN, which is interesting observation. This seems imply there is something
22

---

Published as a conference paper at ICLR 2025
universal across different architectures. (2) the effect of grid size: increasing grid size decreases training loss (since enhances fitting capability), however, the test metrics may get worse (c.g., for [784,10]) may get better (e.g., for 1784,100,10J), Combined with (1), this seems to imply that increasing grid size beneficial when the shape the network is correct but might be harmful otherwise. (3) KANS and MLPs have comparable performance in terms of loss and accuracy. This is probably because the MNIST dataset is too simple. KANs consume much more training time than MLPs Besides the 2 factor slowdown due to the recursive computation of order-k splines, grid updates are also quite expensive due to the high-dimensional inputs. We expect these slowdown factors to have straightforward solutions, and combining KANs with Convolutional neural networks is promising direction to incorporate symmetry inductive biases into architectures.
Model
Train loss
Test loss
Train Acc f
Test Acc f
Time (s)
+
KAN [784,10]
1.5 x 10
2.8 x 10
95.7%
93.0%
83.8
KAN [784,10)
9.9 10-
3.3 x 10
97.0%
92.4%
96.8
KAN [784,10] G
3.4 10
4.4 x 10
99.1%
91.7%
155.4
MLP [784,10]
2.3 x 10
2.8 x 10
93.7%
92.5%
5.8
KAN [784,10,10] G 3
8.3 x 10
2.2 x 10
97.5%
94.5%
106.8
KAN [784,10,10] 5
4.0 x 10
3.1 10
98.7%
94.2%
121.8
KAN 10,101 G
1.8 10
3.7 10
99.4%
94.2%
168.0
MLP [784,1010]]
1.6 10
2.3 10
95.1%
93.7%
6.3
KAN [784, G
4.0 x 10
2.0 x 10
99.0%
97.4%
419.3
KAN 170,,100,10] G
5.6 x 10
9.6 x 10
100.0 %
98.2%
435.9
KAN 100,100 G =
3.8 x 10
9.2 x 10
100.0 %
98.2%
531.8
MLP [784,100,10]
3.5 x 10
9.7 x 10
100.0 %
97.9%
8.3
KAN 108..100,000.11 =
1.3 10-2
1.8 10
99.6%
97.6%
498.6
KAN 1784,100.000,10 G
1.6 x 10
1.9 X 10
99.5%
97.6%
551.1
KAN 1784,,00,100,1000 G -==10 10
1.1 10
2.0 x 10-1
99.7%
97.5%
655.3
MLP [784,100,100,10)
1.0 X 10
1.5 X 10
99.7%
97.7%
9.6
KAN [784, 10,,,000,,00,1100000 G
1.3 10-2
1.8 x 10
99.6%
97.8%
631.5
KAN [784.100,100010001000 G
1.9 10
1.6 10
99.4%
97.3%
643.6
KAN [784,1000,10011000110 G
1.5 10
1.6 x 10-1
99.6%
97.4%
813.3
MLP 1784,100110001100.0000
1.8 x 10
1.4 X 10-1
99.6%
97.7%
11.0
Table 5: Comparing KANs and MLPs on MNIST
D HYPERPARAMETER SEARCH OF MLPs AND KANS
To help understand how hyper-parameters affect the comparison of KAN vs MLP performance, we conduct a hyperparameter sweeping for the function fitting task f(x, y) = exp(sin(xx)) + y²)
(randomly generated 1000 training and test samples from U[-1,1]3 1)9). We sweep few things below:
Optimizer: Adam or LBFGS.
Learning rate: For Adam, we choose learning rates from {10- 4.3 10 1,13333
10-3 10-2). For LBFGS, we choose learning rates from (10-2,3 2,3 x 10 10-2 10-1;1).
10-2,10-1,3 1010-,,3
Network width: 10 or 100.
Network Depth: 2.
When we use Adam, we train MLPs for 60000 steps and train KANs with 10000*6 steps (10000 steps for each grid size
In Figure [3 we show the train/test losses for MLPs (red) and KANs (green) under different conditions. We have few observations: (1) learning rate does not seem to play a big effect (except that large learning rates for Adam lead to more oscilla- tions). (2) In terms of test performance, KANs outperform MLPs shallow models (2 or 3 layers), but are comparable for 4-yyeer models. (3) In terms of training performance, KANs can fit to much lower losses than MLPs, both with LBFGS and Adam (despite wild oscillations for Adam, which can probably be mitigated by learning rate decay). KANs are prone to "overfit". which might be good or bad depending the context, i.e., expect KANs improve test performance with more
23

---

Published as a conference paper at ICLR 2025
training data, but also expect KANs to have more performance degradation when data contains noises.
LBFGS
1r-3e-2
1r-ee-1
1r-3e-1
w=10
depth-2
8 16
w=100
w=10
depth
w=100
w=10
depth-4
w= 100
15"*
1300
3003
300
1000
500
3000
Adam
Ir-10-4
3r-4
Ir-e--3
1r-3-3
1rle-2
w=10
depih-2
w=10 w=100
467
8
depth-3
w=100 w=10 w=100
depth-4
20008
40000
000000
20000
43000
000000
20000
50000
136000
20003
20000
500000
20008
40300
00000
Figure 13: Hyperparameter search for KANs and MLPs. Hyperparameters include depths, widths, optimization methods, and learning rates.
E WHAT IF THE NETWORK HAS MORE LAYERS THAN NEEDED?
In Figure 2 we have demonstrated that when the KAN network has a depth smaller than needed, the learned activation functions can be highly oscillatory, appearing to some non-smooth functions. We are also curious about what happens if the network is deeper than needed. We consider fitting a 2D function f(x, y) exp(sin(Txx) + y²). We know that a 2L KAN can smoothly represent the function (Figure left). When we attempt to fit the function with more layers (with sparsity penalty), some edges would become (nearly) identities, shown for 3L (middle) and 4L (right). The identity shorteuts easy to form with linear residuals, while SiLU residuals can lead to complex network structure even under sparsity penalty. We suspect this is a pathology when depth becomes larger, which want to investigate in the future.
24

---

Published as a conference paper at ICLR 2025
le-2
le-1
Ir=
Method
train loss
test loss
train loss
test loss
train loss
test loss
MLP D=2, W=10
7.2 x 10
7.7 x 10-2
6.6 X 10
7.1 x 10
1.0 x 10-1
1.0 x 10
MLP D=2, W=100
5.1 x 10 2
5.4 x 10
4.9 x 10 10- 2
5.3 x 10
3.1 x 10
3.3 x 10
MLP D=3, W=10
1.4 x 10-2
1.4 x 10-2
4.2 10-3
4.5 10
3.5 10
3.6 10 10-3
MLP D=3, W=100
8.1 x 10-3
8.8 10-3
5.5 10-3
5.7 x 10-3
2.5 10
2.5 x 10
MLP D=4, W=10
1.5 x 10
1.7 10
3,6 10-3
3.8 10
5.0 x 10
5.2 x 10
MLP D=4, W=100
2.4 10
2.5 x 10-3
1,4 10
1.5 x 10
1.6 x 10 10
1.7 x 10
KAN D=2, W=10
1.2 x 10-6
2.0 x 10
x 10
2.5 x 10
4.2 x 10
2.7 x 10
KAN D=2, W=100
7.3 x 10-6
2.9 x 10-4
4.2
1.9 10
1.3 x 10
2.2 x 10
KAN D=3, W=10
7.1 10
8.0 10
4.2 10
4.3 10
2.5 10
3.6 10
KAN D=3, W=100
2.0 x 10
1.5 x 10-3
5.6 x 10
7.7 x 10
8,8 X 10
9.1 X 10
KAN D=4, W=10
2.2 x 10-6
5.4 x 10-6
3.9 x 10
2.4 x 10
3.9 X 10
2.4 x 10
MLP D=4, W=100
8.2 x 10-6
1.6 x 10-3
8.4 x 10-6
1.6 x 10-3
2.7 x 10
8.5 x 10
Table 6: Results for the example: faz y) exp(sin(TT x²)) and the LBFGS optimizer. Grid
search width, depth, and learning rate.
Ir le-4
Ir= le-3
Ir le-2
Method
train loss
Test loss..
train loss.
test loss.
train loss.
test loss.
MLP D=2, W=10
1.6 x 10
1.7 x 10
4.6 10
4.9 10
4.7 10
5.2 10
MLP D=2, W=100
2.3 x 10
2.4 x 10
2.4 x 10-4
2.6 x 10 10-4
1.6 x 10
1.7 x 10
MLP D=3, W=10
4.1 x 10-3
4.2 x 10-3
3.4 x 10
3.4 x 10-3
4.1 x 10
4.3 x 10
MLP D=3, W=100
3.5 x 10
4.1 x 10
3.0 X 10-3
3.9 x 10 10-3
2.7 x 10
4.0 x 10
MLP D=4, W=10
3.4 10
3.7 x 10 3
2.1
2.1 10
1.5 x 10
1.5 x 10
MLP D=4, W=100
7.3 10
8.3 10
7.6 10
9.2 10
7.9 10
9.4 x 10
KAN D=2, W=10
2.4 10-7
3.4 x 10-1
2.4 10-7
1.0 X 10
2.2 10-7
1.3 x 10
KAN D=2, W=100
1.4 10
2.0 x 10
1.6 x 10
3.1 x 10
1.6 x 10
4.8 x 10
KAN D=3, W=
1.5 X 10-6
7.1 10-3
2.1 x 10-7
4.4 x 10
3.2 X 10
2.5 X 10
KAN D=3, W=100
1.2 x 10
8.2 x 10
1.8 x 10
1.2 x 10
1.8 X 10
1.7 x 10
KAN D=4, W=10
2.2 x 10-5
2.5 x 10-2
2.8 10-6
1.0 10-2
5.6 10
1.0 10-2
KAN D=4, W=100
1.9 x 10-7
3.2 10
1.9 10-7
2.5 10
3.4 10
2.7 10
Table 7: Results for the example: f(x, v) = exp(sin(TT: + y²)) and the Adam optimizer. Grid search width, depth, and learning rate.
F EXISTENCE OF REDUNDANT NEURONS/EDGES WITH SMALL OR NO SPARSITY PENALTY
In Figure 15 show that sparsity penalty strength À controls the number of redundant neurons. When A= = O, all five neurons appear to be active, with a few neurons/edges appearing to be highly similar. When À 0.001, only two neurons active and they appear almost identical (except that they differ by a minus sign). When À = 0.1, there is only one active neuron in the hidden layer hence there is redundant neuron.
G THE NECESSITY OF SKIP CONNECTIONS
In Figure 16 we show the necessity of using skip connections, i.e., the learnable function f(x)
b(x) +splim(s)) with non-zero b(zz). By default we choose b(x)) SiL.U To test the necessity of
such a b(z), we use a simple 2D function regression task f fæz y) exp(sin (T.) +y?). [2,1.1] KANs
(G 10) trained with the LBFGS optimizer with samples drawn from UI-1, 1J2 (1000 training
and 1000 test samples). We visualize KANs at step 10: KANs using SiLU and linear residual
connections have already learned the correct representation, while KANs without skip connections still struggle to learn the correct representations.
The intuition is: since B-splines are piecewise polynomials, they behave like order-k polynomials locally. When KANs become deeper (layer I.) without skip connections, the function would become
25

---

Published as a conference paper at ICLR 2025
2-Layer KAN
3-Layer
4-Layer KAN
Linear residual
SiLU residual
Figure 14: Fitting 2D function fax y) exp(sin(Mx) + y²). We know that a 2L KAN can smoothly represent the function (left). When attempt fit the function with more layers (with sparsity penalty), some edges would become (nearly) identities, shown for 3L (middle) and 4L (right). The identity shortcuts are easy form with linear residuals, while SiLU residuals can lead to complex network structure even under sparsity penalty.
2. = 0
== 0.001
2 =
Existence of redundant neurons/edges with small or sparsity penalty, À is the sparsity penalty strength.
Figure 15:
order-k-L polynomials which is quite pathological (it is known that high-order polynomials have bad numerical properties). By including the skip connections, the function can have low-order polynomial components by leveraging the skip connections.
SiLU
Linear
Zero (no skip)
Tan RAESE
300ga
Figure 16: Skip connections (either SiLU or linear) make training landscapes smoother, leading to faster training. The visualizations are for KANs at step 10.
26

---

Published as a conference paper at ICLR 2025
H THE NECESSITY OF GRID EXTENSIONS
One may ask: why don't it we just use the large grid size from scratch, instead of using a small grid size first and then do grid extension? In Figure 17 show that KAN with large grid size can easily get stuck at local minima (probably due to bad loss landscape). By contrast, a well- initialized KAN with large grid size (obtained by grid extension from smaller grid sizes) does have such problem. With grid extension: we train the model starting from G = 3 for 50 steps with LBFGS, and then we do grid extension increase G be 10, (each grid trained for another 50 steps). Without grid extension: The KAN is initialized to have G 20 and is trained for 200 steps.
With grid extension
Without grid extension
grd estantion rntermen Prart
steps
Figure 17: Grid extension is needed to avoid bad loss landscapes when initial grid sizes are large.
I IMPLEMENTATION DETAILS OF KAN
X2,1
40x Σ
800
G
1_1
1,2
1_3
11.4
1,5
12g
4
4
4 4 4 4
460
c/0/00
M M
G,
X0,1
X0,2
Figure 18: Left: Notations of activations that flow through the network. Right: an activation function is parameterized B-spline, which allows switching between coarse-grained and fine-grained grids.
Implementation details. Although KAN layer Eq. (5) looks extremely simple, it is non-trivial to make it well optimizable. The key tricks are:
(1) Residual activation functions. We include basis function b(x) (similar residual connec- tions) such that the activation function ((x) is the sum of the basis function b(x) and the spline function:
((x) wxb(x) w.spline(s).
(15)
We set
b(x) =ihn())==(11+++++)))=)
x
(16)
27

---

Published as a conference paper at ICLR 2025
in most cases. spline(2) is parametrized as linear combination of B-splines such that
spline(x) Σ (x(()))
(17)
where o, are trainable (see Figure for an illustration). In principle wy and w, are redundant since it can be absorbed into ((x) and spline(z)) However, we still include these factors (which are by default trainable) to better control the overall magnitude of the activation function.
(2) Initialization scales. Each activation function is initialized to have w, = 1 and spline(x) > 0 wy initialized according the Xavier initialization, which has been used initialize linear layers in MLPs.
(3) Update of spline grids. We update each grid the fly according its input activations, to address the issue that splines are defined on bounded regions but activation values can evolve out of the fixed region during training Grid updates (grid size Gi - G1) use the same least square method as grid extensions (grid size G + G2 > G1), as detailed in
Parameter count. For simplicity, let assume network
(1) of depth L.
(2) with layers of equal width ni = m1 -.... 117. = N,
(3) with each spline of order & (usually k 3) G' intervals (for G + 1 grid points).
Then there are in total O(N2 + k)) (NN2 parameters. In contrast, an MLP with depth L and width N only needs O(N2L) parameters, which appears to be more efficient than KAN. Fortunately, KANs usually require much smaller than MLPs, which not only saves parameters, but also achieves better generalization (see e.g.. Figure 5 and X and facilitates interpretability. We remark that for problems, we can take N = L = and the KAN network in our implementation is nothing but spline approximation. For higher dimensions, we characterize the generalization behavior of KANs with a theorem below.
J PROOF OF THEOREM 2.
Proof. By the classical ID B-spline theory De Boor (1978) and the fact that Dr,i,j continuous functions can be uniformly bounded on a bounded domain, we know that there exist finite-grid B-spline functions b such that for any 0 < < k
[ o - o Po x x-
( 1 - f
Po Po)xelem S CoG
with constant Co independent of G. We fix those B-spline approximations. Therefore we have that the residue R₁ defined via
p 11- . pn)x x - ( o.... &G $6 c t1 ....o Po)x R3 = ( G o &
satisfies
| Cc
with another constant independent of G. Finally notice that
$ x= RI-1 + RI-2 .... Rı + Ro
we know that holds for another constant C' independent of G.
Remark:
We can be more precise about the dependence the constant in the theorem. Define the compositionally smooth function class C,,W. as the class of functions in the form of (IO) such that the input dimension equals n, the width or maxosisı n; in the definition B equals W 2 n, depth equals I., smoothness equals k. Then C only depends on W, L,, k and max ( illo.....
28

---

Published as a conference paper at ICLR 2025
Paper
Idea Intrinsic dimensionality
Scaling exponent a
Sharma & Kaplan Sharma & Kaplan (2020)
(k: + 1)/d
Michaud et al.
Michaud et al. (2023b)
maximum arity
(k: + 1)/2
Poggio et al.
Poggio et al. 2020)
compositional sparsity
m/22
Ours
K-A representation
k +
Scaling exponents from different theories x N e: test RMSE loss, N: number of model parameters, d: input intrinsic dimension, k: order of piecewise polynomial, m: derivative order as in function class Wm..
Table 8:
K NEURAL SCALING LAWS
We remark that although the Kolmogorov- theorem Eq. IT) corresponds to a KAN represen- tation with shape (d, 2d + 1, 1], its functions are not necessarily smooth. On the other hand, if we able to identify a smooth representation (maybe at the cost of extra layers or making the KAN wider than the theory prescribes), then Theorem 2.] indicates that can beat the curse of dimensionality (COD). This should not come as a surprise since we can inherently learn the structure of the function and make our finite-sample KAN approximation interpretable.
Neural scaling laws: comparison to other theories. Neural scaling laws are the phenomenon
where test loss decreases with more model parameters, i.e., x N- where l is test RMSE, N is the number of parameters, and is the scaling exponent. A larger a promises more improvement by simply scaling up the model. Different theories have been proposed to predict a. Sharma & input manifold of
Kaplan [Sharma & Kaplan (2020) suggest that a comes from data fitting intrinsic dimensionality d. If the model function class is piecewise polynomials of order l: (I: = for ReLU), then the standard approximation theory implies (k + 1) d from the approximation theory. This bound suffers from the curse of dimensionality, so people have sought other bounds independent of d by leveraging compositional structures. In particular, Michaud et al. Michaud et al. (2023b) considered computational graphs that only involve unary (e.g.. squared, sine, exp) and binary (+ and x) operations, finding a = (k: + 1)/d" = (k + 1)/2, where d' = 2 is the maximum arity. Poggio et al. |Poggio et al.] (2020) leveraged the idea of compositional sparsity and proved that given function class Wm. (function whose derivatives are continuous up to m-th order), one needs N 0 ) number of parameters to achieve error c, which is equivalent to a = 2 approach, which assumes the existence of smooth Kolmogorov-Amold representations, decomposes the high-dimensiionl function into several ID functions, giving a A + 1 (where & is the piecewise polynomial order of the splines). We choose k = 3 cubic splines a 4 which is the largest and best scaling exponent compared to other works. We will show in Section 4 toy datasets that this bound 4 can fact be achieved empirically with KANs, while previous work Michaud et al. (0236) reported that MLPs have problems even saturating slower bounds (e.g.. a = 1) and plateau quickly. Of course, we can increase i to match the smoothness of functions, but too high & might be too oscillatory, leading to optimization issues.
m Our
Comparison between KAT and UAT. The power of fully-connected neural networks is justified by the universal approximation theorem (UAT), which states that given function and error tolerance 6 >> 0, a two-layer network with >> (e)) neurons can approximate the function within error e. However, the UAT guarantees no bound for how (e) scales with e. Indeed, it suffers from the COD, and N has been shown to grow exponentially with d in some cases Lin et al. (0117). The difference between KAT and UAT is a consequence that KANs take advantage of the intrinsically low-dimensional representation of the function while MLPs do not. In KAT, we highlight quantify- ing the approximation error in the compositional space. In the literature, generalization error bounds, taking into account finite samples of training data, for similar space have been studied for regres- sion problems; see Horowitz & Mammen (2007); Kohler & Langer (202T), and also specifically for MLPs with ReLU activations Schmiitt Hieber 2020). On the other hand, for general function spaces like Soboley or Besov spaces, the nonlinear n--widths theory Devere et al. (9899 1993);
Thii done by drawing B-spline coefficients c, (N(00, 0²) with small a. typically we set a 0.1.
'Other possibilities are: (a) the grid is learnable with gradient descent, e.g., Xu et al. (0155): (b) use nor- malization such that the input range is fixed. We tried (b) at first but its performance is inferior our current approach.
29

---

Published as a conference paper at ICLR 2025
Siegel (2024) indicates that never beat the curse of dimensionality. while MLPs with ReLU activations can achieve the tight rate Yarotsky (2017); Bartlett et al. (2019): Siegel (002)). This fact again motivates to consider functions of compositional structure. the much nicer functions that we encounter in practice and in science, to overcome the COD. Compared with MIPs, we may smaller architecture in practice, since learn general nonlinear activation functions; Schmidt-Hicber (2020) where the depth of the ReLU MLPs needs to reach at least log to have the desired rate, where the number of samples, Indeed, we will show that KANs nicely aligned with symbolic functions while MLPs are not.
also
L DETAILS OF GRID EXTENSION
Fitting f(x,)) exp(sin(rx)) + y²)
KAN [2.5,1]
KAN [2,1.1]
train test
train test
194
184
RWSE
RMSSE
12
09163
gridade
3000
100
2000
86
669
1890
540
1806
3800
13a
400
68
1900
14K8
3900
30k
stop
stop
33
training time (pecondusttep)
6A112..111
1022 loz5
6MM12 5.1 nomt of (aggn
KSN12 1.11 10m1 near ofaaaaed)
SAM 12 band meliaa of co))
10
10
10
grld size
grld 922
Figure 19: We can make KANs more accurate by grid extension (fine-graining spline grids). Top left (right): training dynamics of [2. 5, 1] (,2 KAN. Both models display staircases in their loss curves, i.e. loss suddently drops then plateaus after grid extension. Bottom left: test RMSE follows scaling laws against grid size G. Bottom right: training time scales favorably with grid size G.
We next describe how to perform grid extension (illustrated in Figure 18 right), which is basi- cally fitting new fine-grained spline to an old coarse-grained spline. Suppose we want to ap- proximate ID function in bounded region [a, b] with B-splines of order k. A coarse-grained grid with G₁ intervals has grid points at {lo a, 11,42 /2, ... (c., b}. which is augmented to (t-k,... ,t-,, to.... tc, +tG++)). There are G1 B-spline basis functions, with the 2th B-spline (x(x) being non-zero only on f+t..ft11l (i = 0, ..... G1 + k 1). Then f on the coarse grid expressed in terms of linear combination of these B-splines basis functions Scoarse (z) = Σ G respondingly (n)e(x) E5=0 D (z). The parameters c/s can be initialized from the pa- rameters e; by minimizing the distance between ffno(x) fcoarse (x)) (over some distribution of x):
a, cz). Given finer grid with G2 intervals, f on the fine grid is cor-
2
Ga+k-1 Σ
Gi+k-1 Σ
(4) argmin
(BB()))
(B(()))
(18)
(ef
1=0
which can be implemented by the least squares algorithm. We perform grid extension for all splines in KAN independently.
Complexity of grid extension Suppose we have batch size B, the number of evaluations to create the "supervised" dataset is O B). The least-square problem requires the number of operations
30

---

Published as a conference paper at ICLR 2025
O(G3B) since it is a singular- value decomposition and typically B > G2. This analysis also applies to grid refinements, only by setting G G1.
Toy example: staircase-like loss curves. We use a toy example f(z,)) y) = exp(sin(TT*) + 3²) to demonstrate the effect of grid extension. In Figure 19 (top left), we show the train and test RMSE for a [2,5,1 1] KAN. The number of grid points starts as 3, increases to a higher value every 200 LBFGS steps, ending up with 1000 grid points. It is clear that every time fine graining happens, the training loss drops faster than before (except for the finest grid with 1000 points, where optimization ceases work probably due to bad loss landscapes). However, the test losses first go down then go up, displaying U-shape, due to the bias-variance tradeoff (underfitting vs. overfitting). We conjecture that the optimal test loss is achieved at the interpolation threshold when the number of parameters match the number of data points. Since our training samples are 1000 and the total parameters of [2, 5, 1] KAN 15G (G is the number of grid intervals), we expect the interpolation threshold to be G = 1000/115 R 67, which roughly agrees with our experimentally observed value G ~ 50.
Small KANs generalize better. Is this the best test performance we can achieve? Notice that the synthetic task can be represented exactly by a [2, 1, 1] KAN, so we train a [2, 1, 1] KAN and present the training dynamics in Figure 19 top right. Interestingly, it can achieve even lower test losses than the 2, 5, 1] KAN, with clearer staircase structures and the interpolation threshold is delayed result of fewer parameters. This highlights a subtlety of choosing KAN
to larger grid size architectures. we do not know the problem structure, how can we determine the minimal KAN shape? In Section 2.4 will propose a method auto-discover such minimal KAN architecture via regularization and pruning.
Scaling laws:
comparison with theory. We are also interested in how the test loss decreases as the number of grid parameters increases. In Figure 19 (bottom left), a [2,1,1] KAN scales roughly as test RMSE x G G-3, However, according to the Theorem 2.1 we would expect test RMSE xx G-4. We found that the errors across samples are not uniform. This is probably attributed to boundary effects Michaud et al. (023b). In fact, there are few samples that have significantly larger errors than others, making the overall scaling slow down. If we plot the square root of the median (not mean) of the squared losses, we get a scaling closer to G-1. Despite this suboptimality (probably due to optimization), KANs still have much better scaling laws than MLPs, for data fitting (Figure 5) and PDE solving (Figure 8) In addition, the training time scales favorably with the number of grid points G, shown in Figure bottom right
External vs Internal degrees of freedom. A new concept that KANs highlights is a distinction between external versus internal degrees of freedom (parameters). The computational graph of how nodes are connected represents external degrees of freedom ("dofs"), while the grid points inside an activation function are internal degrees of freedom. KANs benefit from the fact that they have both external dofs and internal dofs. External dofs (that MIPs also have but splines do not) are responsible for learning compositional structures of multiple variables. Internal dofs (that splines also have but MLPs do not) are responsible for learning univariate functions.
M TECHNIQUES FOR INCREASING INTERPRETABILITY
M.1 SIMPLIFICATION TECHNIQUES
1. Sparsification. For MLPs, LI regularization of linear weights is used to favor sparsity. KANs can adapt this high-level idea, but need two modifications:
(1) There is no linear "weight" in KANs. Linear weights are replaced by learnable activation func- tions, so we should define the LI norm of these activation functions.
(2) We find LI to be insufficient for sparsification of KANs; instead additional entropy regular- ization is necessary (see Appendix W for more details).
'When G = 1000, training becomes significantly slower, which is specific to the use of the LBFGS opti- mizer with line search. We conjecture that the loss landscape hecomes bad for G = 1000, so line search with trying find optimal step size within maximal iterations without early stopping.
31

---

Published as a conference paper at ICLR 2025
We define the LI norm of an activation function be its average magnitude over its Np inputs, i..,
No 161 = No Σ ((z(o))
(19)
Then for KAN layer & with Vin inputs and nout outputs, we define the 1.1 norm of & to be the sum of LI norms of all activation functions, i.e.,
Hows
+
(20)
1=1 j=1
In addition, we define the entropy of to
Bin Mout
و log
-
(21)
The total training objective lotal is the prediction loss bpood plus LI and entropy regularization of all KAN layers:
L-1
Lrotal pprod +
P1
Σ
1
+ 12 Σ s(
(22)
1=0
where Pl, Ag are relative magnitudes usually set to #l = μ1 = 1, and À controls overall regulariza- tion magnitude.
2. Visualization. When we visualize a KAN, to get a sense of magnitudes, we set the transparency of activation function or,i.j proportional (anh(3Ar..) where = 3. Hence, functions with small magnitude appear faded out to allow us to focus on important ones.
Pruning, After training with sparsification penalty, we may also want to prune the network to smaller subnetwork. We sparsify KANs on the node level (rather than on the edge level). For each node (say the tth neuron in the 1th layer), we define its incoming and outgoing score as
I = max((o) x
O = max
(23)
and consider node be important if both incoming and outgoing scores are greater than threshold hyperparameter = 10 by default. All unimportant neurons are pruned.
4. Symholification. In cases where suspect that bolic (c.g., cos or log), we provide an interface to set them to be a specified symbolic form, fix__siiil symbolic (1, i, j, f) can set the (1, i,) activation to be f. However, we cannot sim- ply set the activation function to be the exact symbolic formula, since its inputs and outputs may have shifts and scalings. So, we obtain preactivations and postactivations y from samples, and fit affine parameters (a, b, d) such that x cf(az + b) + d. The fitting is done by iterative grid search of a, b and linear regression.
activation functions are in fact sym-
Besides these techniques, provide additional tools that allow users to apply more fine- fine-grained control to KANs, listed Appendix V
M.2 A TOY EXAMPLE: HOW HUMANS CAN INTERACT WITH KANS
Above we have proposed a number of simplification techniques for KANs. We can view these simplification choices buttons one can click on. user interacting with these buttons can decide which button is most promising to click next to make KANs more interpretable. We use an example below showcase how user could interact with a KAN obtain maximally interpretable results,
Let us again consider the regression task
f(x, y) exp (TT) + +2²).
(24)
Given data points (z,, Vs. fi), 2, ... Np. hypothetical user Alice is interested in figuring out the symbolic formula. The steps Alice's interaction with the KANs are described below (illustrated in Figure
32

---

Published as a conference paper at ICLR 2025
exp(sin(aa) +y")
Step 1: train
Step 2 prune
with sparsification
Step 3a: set sine
Step output symbolic formula
Step train affine parameters
Step 3c: set exponential
Step 3b: set squared
1.0e J.0y² (3.14x)
reach machine
Step 6:
number Snap
sin(sx)
Figure 20: An example of how to do symbolic regression with KAN.
Step 1: Training with sparsification. Starting from a fully-connected [2, 5, 1] KAN, training with sparsification regularization can make it quite sparse. 4 out of 5 neurons in the hidden layer appear useless, hence want prune them away.
Step 2: Pruning. Automatic pruning is seen to discard all hidden neurons except the last one, leaving [2, 1] KAN. The activation functions appear be known symbolic functions.
Step 3: Setting symbolic functions. Assuming that the user can correctly guess these symbolic formulas from staring the KAN plot, they can set
fix fix__ 1ex__symolice (0, 0, 0, 'sin')
fix wtw.olsmmllccocom (0, 0, 'x"2")
(25)
fix symbosiciiii.c.c (1, 0, 0, 'exp").
In case the user has no domain knowledge or no idea which symbolic functions these activation functions might be, we provide a function suggest_symboll ic to suggest symbolic candidates.
Step 4: Further training. After symbolifying all the activation functions the network, the only remaining parameters are the affine parameters. We continue training these affine parameters, and when we see the loss dropping to machine precision, we know that we have found the correct sym- bolic expression.
Step 5: Output the symbolic formula. Sympy is used to compute the symbolic formula of the output node. The user obtains 1g.1009"+00sii which is the true answer (we only displayed two decimals for TT).
N ACCURACY: TOY SYMBOLIC DATASETS
We train these KANs by increasing grid points every 200 steps, in total covering G (3. 5, 10, 20, 50, 100, 200, 500, 1000]. We train MLPs with different depths and widths as base- lines. Both MLPs and KANs are trained with LBFGS for 1800 steps total. We plot test RMSE as a function of the number of parameters for KANs and MLPs in Figure 5 showing that KANs have better scaling curves than MIPss, especially for the high-dimensional example. For comparison, we plot the lines predicted from our KAN theory as red dashed (a k+ = 4). and the lines predicted from Sharma Kaplan Sharma & Kaplan 2020) as black-dashed (a = (k + 1)/d d = 4/d). KANs can almost saturate the steeper red lines, while MLPs struggle to converge even as fast as the slower black lines and plateau quickly. We also note that for the last example, the 2-Layer KAN [4, 9, 1] be- haves much worse than the 3-Layer KAN (shape [4, 2, 2, 1)). This highlights the greater expressive power of deeper KANs, which is the same for MLPs: deeper MLPs have more expressive power than shallower ones. Note that we have adopted the vanilla setup where both KANs and MI. I.Ps
33

---

Published as a conference paper at ICLR 2025
Mixire aage
Jacchia cliptic laadkons
clipsie.3)
(221))
1:21 1017
[2:22444]
1.33 20
6,48 D4
Incomptex clipte anegua load
[2244]
[221441
bastoplle clipic inopra of soral kini
dlpccaeii vi
(22//1)
6.6
122.01
8,80
Bord
Besod finction secad
1M3
10"1
Modited Uossa fxectice sord
3071
1824)
30
Nedtree lecal fanzion of the tina tind
Ins.al
PA
1:38 1071
PA,33.11
9:28 30
147% 10
Avastaed Logesake luacion in
(221)
(2:44)
5,30 x x0
1.7% D0
Avactired Lagendre traction
(241)
[240]
1.30
Avosiiel Logesale (uacio i
(jun/2.n.
(224)
6.8
[132.1]
2.80 10
9.89
bennt,
12211
opersal humeeeio (o
Jernin J.
(214)]
18140)
30"7
200 14
qpercal hnmeerir (+t- 1a-
qt Jeenl. al
(2211)
4.70 101
123111
1.50 10
1:81 10"
hameeia (s) La
okJaeni Laval
(221)
[2222]]
9.40
4.21 x x.10
Table 9: Special functions
trained with LBFGS without advanced techniques, e.g., switching between Adam and LBFGS, or boosting Wang & Lai (2024). We leave the comparison of KANs and MLPs in advanced setups for future work.
O ACCURACY: SPECIAL FUNCTIONS
We choose MLPs with fixed width 5 or 100 and depths swept in {2, 4, 63. We run KANs both with and without pruning. KANs without pruning: We fix the shape of KAN, whose width are set and depths are swept in (2.3,4,5.6). KAN with pruning. We use the sparsification (A 10-2 or 10-3) and pruning technique in Section Mc. obtain smaller KAN pruned from fixed- shape KAN. Each KAN initialized have 3. trained with LBFGS, with increasing number of grid points every 200 steps cover G {3, 5, 10, 20, 50, 100, 2003. For each hyperparameter combination, we run random seeds.
For each dataset and each model family (KANs or MLPs), we plot the Pareto frontier 5 in the (number of parameters, RMSE) plane, shown in Figure KANs performance is shown sistently better than MLPs, i.c., KANs can achieve lower training/test losses than MLPs, given the same number parameters. Moreover, we report the (surprisingly compact) shapes of our auto- discovered KANs for special functions in Table 9 On one hand, it is interesting to interpret what these compact representations mean mathematically. On the other hand, these compact representa- tions imply the possibility of breaking down high high-dimensional lookup table into several ID lookup tables, which can potentially save a lot of memory, with the (almost negligible) overhead to perform few additions at inference time.
con-
P ACCURACY: FEYNMAN DATASETS
Feynman datasets Given the structure of the dataset, we may construct KANs by hand, but can KANs find more compact representations? The Feynman dataset Udrescu & Tegmark (2020): Udrescu al. 2020), consisting of symbolic equations physics, is a good testbed. We find that KAN shapes discovered by pruning usually smaller than
KAN shapes,
with comparable accuracy. Here we focus on a sample equation called the relativistic velocity addi- tion formula fu, v) (u (1 + wr). We can manually construct 5-Layer KAN to compute this function, considering the resources required by two multiplications, one inversion and two ad- ditions. However, the auto-discovered KANs are only 2 layers deep! In hindsight, this is actually expected if we recall the rapidity trick in relativity: define the two "rapidities" a = arctanh u and b = arctanh v. The relativistic composition of velocities are simple additions in rapidity space, i.e., uie tanh (arctanh u + arctanh v), which can be realized by two-layer KAN. 1t
We compare four kinds of neural networks:
Pareto frontier is defined as fits that are optimal in the sense of no other fit heing both simpler and more accurate.
34

---

Published as a conference paper at ICLR 2025
Jas stge
Praee Jiead koy
lnmaeeeeee (onor RUSE)
Rladd Unnen los AMESi)
Utowaed Javci RMBE
HLF donoriua RMSS
EAl dace
Batukees
161
16.2
R4244)
B4B
(4244)
4 ux
13/ ay
V:4 weee
18:5
K42A1]
14.10
145
142-10"
10x 33
1:211
(9 uneaft
22211
12211
22.11
133-
113
6.1
1150
1331
1230
13311
481 1611
175 13
1.40 93
LIk4
23.01
130 u0
1.5% 33
1274
02311
23311
104-10
214 39
1930
134
Fa
(B331.1)
[3334()
336 kmm
1003 10
141% 3-
134.
6.
2131
3,09-00
1374
6- 4 2o77
14 Jehool
03321
12301
92.11
<91
singl (.)
5:30
P50032
1444
33.0
23.41
2.80
5.31
Li0s 31
208 uo
scefrctt)
814
2331
BAR
03531
132-10
Hill coud)
1
1542210
RA10
632"
140%
5.92
111127
By
[21211
BMI
22.11
247
443
145
2131
64431
125
13
67 M6Y
BM
BAJA4)
878 00
185 31
01:0.0
(8 R2
Vi-
BM
2.4 am
166 31
BL1137
301 ana0))
61 ooel
6,3
B33210
BAR
22.11
Libx
101-18
432
Table 10: Feynman dataset
(1) Human- cunttn-constructuedd KAN. Given a symbolic formula, we rewrite it in Kolmogorov-Arool representations. For example, to multiply two numbers x and y, we can use the identity zy =
(m+t)) (x-y)² which corresponds to a [2, 2, KAN. The constructued shapes are listed in the "Human constrcce KAN shape" in Table 10
(2) KANs without pruming. We fix the KAN shape to width 5 and depths are swept over (2,3,4,5,6).
(3) KAN with pruning. We use the sparsification (A = 10-2 or 10-3) and the pruning technique from DoetionnM.Jpr to obtain smaller KAN from fixed-shape KAN from (2).
(4) MLPs with fixed width 20, depths swept in {2, 32,3,44 4,5,4,5,6) 6}. and activations chosen from (Tanh, SiLUU.
Each KAN is initialized to have G = 3, trained with LBFGS, with an increasing number of grid points every 200 steps to cover G (3, 5, 10, 20, 50, 100, 200J. For each hyperparameter combina- tion, we try 3 random seeds. For each dataset (equation) and each method, we report the results of the best model (minimal KAN shape, lowest test loss) over random seeds and depths in Table [0] We find that MLPs and KANs behave comparably on average. We conjecture that the Feynman datasets too simple let KANs make further improvements, in the sense that variable depen- dence is usually smooth or monotonic, which is in contrast to the complexity of special functions which often demonstrate oscillatory behavior.
We report the pruned KAN shape in two columns of Table 10 column is for the minimal pruned KAN shape that can achieve reasonable loss (i.c., test RMSE smaller than 10 3); the other column for the pruned KAN that achieves lowest test loss. It is interesting observe that auto-discovered KAN shapes (for both minimal and best) are usually smaller than our human constructions. This means that KA representations can be more efficient than we imagine. At the same time, this may make interpretability subtle because information is being squashed into a smaller space than what we are comfortable with.
Q "CONTINUAL LEARNING" OF ID TOY FUNCTION?
We show that KANs have local plasticity and can avoid catastrophic forgetting by leveraging the locality of splines, for ID functions. The idea is simple: since spline bases are local, a sample will only affect a few nearby spline coefficients, leaving far-away coefficients intact (which is desirable
35

---

Published as a conference paper at ICLR 2025
Phase
Phase 2
Phase 3
Phase
Phase 5
Dota
nmma
num
KAN
MMNA
MMAt
MUMA
Mmm
Aumm
MLP
AMNA
Figure 21: A toy continual learning problem. The dataset is ID regression task with 5 Gaussian peaks (top row). Data around each peak is presented sequentially (instead of all once) to KANs and MLPs. KANs (middle row) can perfectly avoid catastrophic forgetting, while MLPs (bottom row) display severe catastrophic forgetting.
since far-away regions may have already stored information that we want preserve). By con- trast, since MLPs usually use global activations, e.g., ReLU/Tanb/SSLU etc., any local change may propagate uncontrollably to regions far away, destroying the information being stored there.
We use toy example to validate this intuition. The ID regression task composed of 5 Gaussian peaks. Data around each peak is presented sequentially (instead of all at once) to KANs and MLPs, as shown in Figure 21 top row. KAN and MLP predictions after each training phase are shown in the middle and bottom rows. As expected, KAN only remodels regions where data is present on in the current phase, leaving previous regions unchanged. By contrast, MLPs remodels the whole region after seeing new data samples, leading to catastrophic forgetting. We want to mention that this toy example is somewhat trivial and is attributed to local activation functions rather than the KAN architecture. We simply feel this is a cute example to share in case anyone inspired by it. However, this should not be interpreted as solving the continual learning problem. Indeed, when we try deeper KAN [1,5,5,1], the continual learning feature partially lost, depending on grid sizes (see Figure 22
[1.5,5.1]
A
-1
[1.5,5.1] KAN (G-10)
m
Figure 22: When KANs become deep, the continual learning ability is partially lost. Top: grid size
200; bottom: grid size 10.
36

---

Published as a conference paper at ICLR 2025
(a) multiplication
divisien
x/y
numerical categury @argming xi
(n deeper compositions
(x1 X2) (Xg xa)
(4) special function
(c) phase transition
exp( Jo(20x) + y
tanh(5ix] + * 1))
X
X1 X2
X3
X1
X2
X3
X4
Figure 23: KANs are interepretable for simple symbolic tasks.
R INTERPRETABLLITY SUPERVISED TOY DATASETS
We tried KANs for 6 symbolic tasks, shown in Figure 23 (1) Multiplication f(x, y) zy. A |2,5,
KAN is pruned [2, 2, KAN. The learned activation functions are linear and quadratic. From the computation graph, we see that the way it computes zy is leveraging 2xy = (æc y)² (z 12²++y+ y²))
(2) Division positive numbers f(x y) = z/y. [2, 1] KAN is pruned to a [2, 1, 1] KAN. The learned activation functions are logarithmic and exponential functions, and the KAN is computing z/y by leveraging the identity z/y = exp(logx - logy). ) (3) Numerical to categorical. The task is to convert real number in [0, 1] to its first decimal digit (as hots), c.g., 0.0618 + 01 0, 0, 0, -.....,, 0.314 + [0, 0. 0, 1, .... Notice that activation functions are learned to be spikes located around the corresponding decimal digits. (4) Special function fær y) exp( (20xx) + One limitation of symbolic regression is that will never find the correct formula of a special function if the special function is not provided prior knowledge. KANs can learn special functions the highly wiggly Bessel function Jo (20x) is learned (numerically) by KAN. (5) Phase transition ( I2, T3) (5 _²² + 23 1)). Phase transitions are of great interest in physics. so we want KANs to be able to detect phase transitions and to identify the correct order parameters. We use the tanh function simulate the phase transition behavior, and the order parameter the combination of the quartic terms of , 3z. Both the quartic dependence and tanh dependence emerge after KAN training. This a simplified case of localization phase transition discussed in Section (6) Deeper compositions [x1 22, I3, xT4) = v( - x²)²² + (æ3 - .² To compute this, we would need the identity function, squared function, and square root, which requires at least three-layer KAN. Indeed, we find that a [4, 3, 3, 1] KAN can be auto-pruned to [4, 2, 1, 1] KAN, which exactly corresponds to the computation graph would expect.
INTERPRETARILITY: UNSUPERVISHD TOY DATASETS
Given a set of variables (2,,,2,"" Ta), we want to discover a structural relationship between the variables. Specifically, want to find non-zero such that f ,.,)) R O.
Unsupervised toy dataset Given a set of variables (I1, Ø2. z2). we want to discover a struc- tural relationship between the variables. Specifically, want to find non-zero f such that f ( 22, ... Σx) zs O. Via contrastive learning formulation, we are able to turn this unsupervised learning problem into supervised learning (details in Appendix
37

---

Published as a conference paper at ICLR 2025
seed (data) or (random)
seed 2024 (data) (random)
X1 X2
X3
X4
X5
X6
X1
X2
X3
X4
X5
X6
Figure 24: Unsupervised learning of toy task. KANs can identify groups of dependent variables,
i.e. (2, and (rs) in this case.
We demonstrate that the unsupervised paradigm works for a synthetic 6D dataset, where (21 #2; 433) dependent variables such that T3 exp(sin(z11 + zz)): (Iz, Is5 are dependent variables with Is = zq: 2g is independent of the other variables. In Figure 24 we show that for seed = 0, KAN reveals the functional dependence among 21 222 and Iı; for another seed 2024, KAN reveals the functional dependence between I4 and It.. Our preliminary results rely on randomness (different seeds) to discover different relations; in the future we would like to investigate more systematic and more controlled way to discover a complete set of relations. Even so, our tool in its current status can provide insights for scientific tasks. We present our results with the knot dataset in Appendix TT
We tackle the unsupervised learning problem by turning it into a supervised learning problem on all of the d features, without requiring the choice of a splitting. The essential idea to learn a function f F2) 0 such that f is not the O-function. To do this, similar to contrastive learning, we define positive samples and negative samples: positive samples are feature vectors of real data. Negative samples are constructed by feature corruption. To ensure that the overall feature distribution for each topological invariant stays the same, we perform feature corruption by random permutation of each feature across the entire training set. Now we want to train network y such that g ((eaa)) = 1 and (Xtake) = 0 which turns the problem into a supervised problem. However, remember that we originally want (X
0 and (Xtake) O. We can achieve this by having g = a where (xx) = exp( 2 is a Gaussian function with a small width w, which can be conveniently realized by KAN with shape [..,, 1] whose last activation is set to be the Gaussian function a and all previous layers form f. Except for the modifications mentioned above, everything else is the same for supervised training.
2²
T INTERPRETABILITY. KNOT THEORY
Knot theory is a subject in low-dimensione topology that sheds light on topological aspects of three- manifolds and four-manifolds and has a variety of applications, including in biology and topological quantum computing. Mathematically, a knot K is an embedding of 5 into 53, Two knots K and K' are topologically equivalent if one can be deformed into the other via deformation of the ambient space 53, in which case we write [K] [K']. Some knots are topologically trivial, meaning that they can be smoothly deformed to standard circle. Knots have a variety of decformationninvariic features ff called topological invariants, which may be used to show that two knots are topologically inequivalent, [K] & [K'] if f(K) f(("") fKK).. In some cases the topological invariants are geometric in nature. For instance, a hyperbolic knot K has a knot complement $ \ K that admits a canonical hyperbolic metric g such that vol, K) is topological invariant known as the hyperbolic volume. Other topological invariants are algebraic in nature, such as the Jones polynomial.
Given the fundamental nature of knots in mathematics and the importance of its applications. it is interesting to study whether ML can lead to new results. For instance, in Gukov et al. (2023) reinforcement learning utilized to establish ribbonness of certain knots, which ruled out many potential counterexamples to the smooth 4d Poincaré conjecture.
Our results have one subtle difference from results in Davies et al. (2021): they find that signature is mostly dependent on μ4, while we find that signature is mostly dependent on f4,- This difference could be due to subtle algorithmic choices, but has led us carry out the following experiments: (a)
38

---

Published as a conference paper at ICLR 2025
Formola
Discovered by
test
yd with Signature
r with DM
aCc
formula
Human (DM)
83.1%
0.946
B
1.12sinn(49944 (.0 0.552
[3. 1J KAN
62.6%
0.837
0.897
0.04e (188(1-0.554)
C
1.17tan(-1.51-00 1.43(1
(.4p:)
[3. KAN KAN
71.9%
0.871
0.934
1.32c -:110--.11100-.._.."
D
-0.09 + 1.04exp( 9.50
0.62sin(0 + T.26))
[3.2. KAN KAN
84,0%
0.947
0.997
0.32ton(0.03A
6.59) 0.11e-1 alle-1-7700 117--177703--001
1.00e-T.:00.000000 0.00 0.27atan(0.53pp - 0.6) +
0.09 + cxp( 2.58(1
0.36pt 1))
B
4,70Am-
[3.2,1] KAN Pade approx
82.8%
0.946
0.997
4.08g
2-02-22001--
10p
1J KAN(33 1J KAN
77.8%
0.925
0.977
0.32 18 1.. 99. 0.05X
Table 11: Symbolic formulas of signature as a function of meridinal translation μ (real pr. imag p4) and longitudinal translation A. In Davies et al. (202T), formula A was discovered by human scientists inspired by neural network attribution results. Formulas B-F are auto-discovered by KANs. KANs can trade-off between simplicity and accuracy (B, C. D). By adding more inductive biases, KAN is able to discover formula E which is not too dissimilar from formula A. KANs also discovered formula F which only involves two variables (u, and X) instead of all three variables, with little sacrifice in accuracy.
ablation studies. We show that (t, contributes more to accuracy than 14, (see Figure for example, Mr alone achieve 65.0% accuracy, while μi, alone only achieve 43.8% accuracy. (b) We find a symbolic formula (in Table II] which only involves μa, and X, but can achieve 77.8% test accuracy.
To investigate (2), ie., obtain the symbolic form of a,, we formulate the problem as a regression task. Using auto-symbolic regression introduced in Section M. we convert trained KAN into symbolic formulas. We train KANs with shapes [3, 1], [3. 1, TJ. 33, 2, 1]. whose corresponding sym- bolic formulas are displayed in Table B-D. It is clear that by having larger KAN, both accuracy and complexity increase. So KANs provide not just single symbolic formula, but a whole Pareto frontier of formulas, trading off simplicity and accuracy. However, KANs need additional inductive biases to further simplify these equations to rediscover the formula from Davies et al. (2021) (Ta- Al). We have tested two scenarios: (1) in the first scenario, we assume the ground truth formula has multi-variate Pade representation (division of two multi-variate Taylor series). We first train [3, 2, 1] and then fit to a Pade representation. We can obtain Formula in Table 11 which bears similarity with DeepMind's formula. (2) We hypothesize that the division is not very interpretable for KANs, so we train two KANs (one for the numerator and the other for the denominator) and divide them manually. Surprisingly, we end up with the formula F (in Table IT] which only involves [4, and A, although (4; is also provided but ignored by KANs.
unsupervised learning Knot data are positive samples, and we randomly shuffle features to obtain negative samples. An [18, 1, 1] KAN is trained classify whether given feature vector belongs to a positive sample (1) or a negative sample (0). We manually set the second layer activation to be the Gaussian function with a peak one centered zero, so positive samples will have activations at (around) zero, implicitly giving a relation among knot invariants Ele (2;) = where 2; stands for a feature (invariant), and is the corresponding activation function which can be readily read off from KAN diagrams. We train the KANs with = (10-2, 10-33 favor sparse combination of inputs, and seed = 10. ,99).
U INTERPRETABILITY ANDERSON LOCALIZATION
Application to Physies: Anderson localization Anderson localization is the fundamental phe- nomenon in which disorder in a quantum system leads to the localization of electronic wave func- tions, causing all transport to be ceased in Appendix U Here, we apply KANs to numerical data generated from quasiperiodic models to extract their mobility edges (phase transition boundaries), including generalized Aubry-
(1958). More background information is available
39

---

Published as a conference paper at ICLR 2025
System
Origin
Mobility Edge Formula
Accuracy
Theary
aF +
GAAM
KAN auto
1-62E² + 21.000E Q.6E 3-50 + 0.91% 45.13A 54.45
99.0%
Theary
E + Acoehp
98.6%
KAN
13.99ein(..25ssnn00.7A... -118)-814))11874- 51.08xppp (etpacppppp00000000000000 0.930 -00-0007770mm00005
0.S4arctam 58E 0.26) 0
97.1%
0.03arctan(0.440 0.62) -)0
MAAM
KAN (step
4.190.22%5n(00.77 2.17)
0.77umtamn00553F
0.19) + arctun(0.97p + 0.15)
0.35)
28.93 - 250hh 0.16) 0.34arstan (0.655
0.51) + +...0.0ttaa - =
KAN man (step 3) auto
4.63F 10.25/ 0.94.in(0.977
6.81) + tanh(0.8p
0.45) 0.007²
97.7%
11.78ein00.66p 1.41) +2.44
1.32)
KAN man (step 4A)
6.92F 6.23(
0.922 1)a 257 0.055 + 0.95exsh 0.4)
1)*
96.6%
12.90oosh
[0.53p 0.16) + 19.89
KAN man (step 4B)
7.25Æ 8-81(- 1)2 4.0((-7 p 0.04) 12.71 (..71- (0.3p 1)2
95.4%
0.86)"
10.29
Table 12: Symbolic formulas for two systems GAAM and MAAM, ground truth ones and KAN- discovered ones.
(a) GAAM, automatic mode
(b) MAAM,
Stp trakking
Sup astrrat regroskes
Step training
Styp 31 achonade
Acc 98.7%
Axc SUPS
Arc M0.7%
MAAM, manuul mode
Stp maaliingg
falegennttiic mailing
Step AN fasng synthale docks))
Step dih flinng sombele prquairttti
9EA%%
Aue: 97.7%
Anc 91.%
Acc 9e.0%
Anc 65.1%
Human-KAN collaboration discover mobility edges of GAAM and MAAM. The human user can choose to be lazy (using the auto mode) or more involved (using the manual mode).
Figure 25:
André model (GAAM) Ganeshan et al. (2015) and the modified Aubry- André model (MAAM) Bid- Ganeshan dee & Sarmal 2010). leaving results on a simpler tutorial case, the Mosaie model (MM) Wang al. (2020), to Appendix
We highlight how users (scientists) interact with KANs to get more interpretable results (in Figure 25). For the simpler GAAM case where the mobility edge is a quadratic function, the user choose to be lazy and let KANs automatically do everything all the way through. KANs will be able to output the correct formula with some negligible еrгоr terms (shown in Table 12). However, complex MAAM case, the fully automated mode find user can choose to interact with KANs by fixing some activation to be known symbolic formulas and do further training. In the end, the user can obtain a family of symbolic formulas (instead of just one) that trade off between accuracy and simplicity.
for the
complicated formula. A
Anderson localization is the fundamental phenomenon in which disorder in quantum system leads to the localization of electronic wave functions, causing all transport to be ceased Anderson [1958). In and dimensions, scaling arguments show that all electronic eigenstates exponen-
40

---

Published as a conference paper at ICLR 2025
tially localized for infinitesimal amount of random disorder Thouless 1972) Abrahams et al. (1979). In contrast, in three dimensions, a critical energy forms a phase boundary that separates the extended states from the localized states, known mobility edge. The understanding of these mobility edges is crucial for explaining various fundamental phenomena such as the metal-insulator transition in solids Lagendijk et al. (2009), well localization effects of light in photonic de- vices Segey et al. (2013): Vardeny et al. (2013); ; John (1987); Lahini et al. (2009); Vaidya et al.) (023). is therefore necessary develop microscopic models that exhibit mobility edges to en- able detailed investigations. Developing such models is often more practical in lower dimensions, where introducing quasiperiodicity instead of random disorder can also result in mobility edges that separate localized and extended phases. Furthermore, experimental realizations of analytical mobil- ity edges can help resolve the debate on localization in interacting systems De Roeck et al. (2016);
Li et al. (2015). Indeed, several recent studies have focused on identifying such models and deriving exact analytic expressions for their mobility edges An ct al. (2021); Biddle Sarma (0100); Duthic
ct al. (202T); Ganeshan al. (2015): Wang et al. 2020 202 Zhou et al. (2023).
Here, we apply KANs to numerical data generated from quasiperiodic tight-binding models to ex- tract their mobility edges. In particular, examine three classes of models: the Mosaic model (MM)|Wang et al. (2020), the generalized Aubry-Andrt model (aAAMGaannnshnn et al. (2013) and the modified Aubry-Andre model AMA (MAAM)Biddlle [iiddle & Sarma (0010). For the MM, we testify KAN's ability to accurately extract mobility edge as a ID function of energy. For the GAAM, we find that the formula obtained from KAN closely matches the ground truth. For the more complicated MAAM, we demonstrate yet another example of the symbolic interpretability of this framework. A user can simplify the complex expression obtained from KANs (and corresponding symbolic formu- las) by means of "collaboration" where the human generates hypotheses to obtain better match (e.g., making an assumption of the form of certain activation function), after which KANs can carry out quick hypotheses testing.
To quantify the localization of states in these models, the inverse participation ratio (IPR) com- monly used. The IPR for the hthh eigenstate, (x),,, is given by
Σn Iuoe 1 12)
IPR
(26)
(Enl
where the sum runs over the site index. Here, we use the related measure localization - the fractal dimension of the states, given by
log()PR))
DR
(27)
log(N)
where N is the system size. D2 0(1) indicates localized (extended) states.
Mosaic Model (MM) We first consider a class of tight-binding models defined by the Hamilto- nian Wang et al. 2020)
H H=t ( Ch+ren++ + H.c.) + E Vv(A,,))
(28)
where t is the nearest-neighbor coupling, ex (ctt) is the annihilation (creation) operator at site and the potential energy Vn is given by
( (2com22mbb ( + Φ) j j= mx 10.
V (A
(29)
otherwise,
To introduce quasiperiodicity, we set b to be irrational (in particular, we choose to be the golden ratio 1ty5), x is an integer and the quasiperiodic potential occurs with interval k. The energy (E) spectrum for this model generically contains extended and localized regimes separated by a mobility edge. Interestingly, unique feature found here is that the mobility edges are present for an arbitrarily strong quasiperiodic potential (i.e. there are always extended states present in the system that co-exist with localized ones).
The mobility edge can be described by g(A, E) = A - 1fx (E) = o. g(>, E) > 0 and g(,, E) O correspond localized and extended phases, respectively. Learning the mobility edge therefore
41

---

Published as a conference paper at ICLR 2025
hinges learning the "order parameter" (\,, E). Admittedly, this problem be tackled by many other theoretical methods for this class of models Wang et al. (2020). but we will demonstrate below that our KAN framework is ready and convenient to take in assumptions and inductive biases from human users.
Let us assume a hypothetical user Alice, who new PhD student in condensed matter physics, and she is provided with a [2. 1] KAN as assistant for the task. Firstly, she understands that this classification task, so is wise set the activation function in the second layer to be sigmoid by using the f ex__ymmolicc functionality. Secondly, she realizes that learning the whole 2D function g(,, E) is unnecessary because in the end she only cares about X(E) determined by g), 0. In so doing. it is reasonable to assume g(,, E) À h(E) 0. Alice simply sets the activation function of À to be linear by again using the fix _ex__ssmmoliccc functionalityy. Now Alice trains the KAN network and conveniently obtains the mobility edge, shown in Figure 26 Alice can get both intuitive qualitative understanding (bottom) and quantitative results (middle), which well match the ground truth (top).
K = 2
3
= 4
K = 5
Poatdd
Figure 26: Results for the Mosaic Model. Top: phase diagram. Middle and Bottom: KANs can obtain both qualitative intuition (bottom) and extract quantitative results (middle). & 2 golden ratio.
1+ the
Generalized Andre-Aubry Model (GAAM) We next consider a class of tight-binding models de- fined by the Hamiltonian Ganeshan et al. (2015)
H -= Σ (c441cn + H.c.) Σ V
(30)
where / is the nearest-neighhor coupling, Cx (c(,) is the annihilation (creation) operator at site n and the potential energy V, is given by
cos(2mmn) + Φ)
Va 0) 21
(31)
a cosmb + 6)¹
which is smooth for a E (-1, 1). To introduce quasiperiodicityy we again set b to irrational (in particular, we choose to be the golden ratio). As before, we would like to obtain expression for the mobility edge. For these models, the mobility edge is given by the closed form expression Gane-
shan et al. (2015): Wang et al. (202I).
E 2(t A).
(32)
42

---

Published as a conference paper at ICLR 2025
We randomly sample the model parameters: o, a and À (setting the energy scale = 1) and calculate the energy eigenvalues as well as the fractal dimension of the corresponding eigenstates, which forms our training dataset.
Here the "order parameter" to be learned is g(a; E, = aE + 2(A - 1) and mobility edge corresponds g 0. Let us again assume that Alice wants to figure out the mobility edge but only has access to IPR or fractal dimension data, so she decides to use KAN to help her with the task. Alice wants the model to be as small as possible, so she could either start from large model and use auto-pruning to get a small model, or she could guess reasonable small model based on her understanding of the complexity of the given problem. Either way, let us assume she arrives at [4, 2, 1. 1] KAN. First, she sets the last activation to be sigmoid because this is a classification problem. She trains her KAN with some sparsity regularization to accuracy 98.7% and visualizes the trained KAN in Figure 25 (a) step She observes that & is not picked up at all, which makes her realize that the mobility edge independent of á (agreeing with Eq. (32). In addition, she observes that almost all other activation functions are linear or quadratic,
she turns on automatic symbolic snapping, constraining the library to be only linear or quadratic. After that, she immediately gets a network which is already symbolic (shown in Figure 25 better) accuracy 98.9%. By using s symbo liboooici_oi fo Porumul la functionality, Alice conveniently gets the symbolic form of g. shown in Table 12 GAAM-KAN auto (row three). Perhaps she wants cross out some small terms and snap coefficient to small integers, which takes her close to the true answer.
(a) step 2), with comparable (even slightly
This hypothetical story for Alice would be completely different if she is using a symbolic regres- sion method. If she is lucky, SR can return the exact correct formula. However, the vast majority of the time SR does not return useful results and it is impossible for Alice to "debug" or inter- act with the underlying process of symbolic regression. Furthermore, Alice may feel uncomfort- able/inexperienced to provide a library of symbolic terms as prior knowledge to SR before SR is run. By constrast in KANs, Alice does not need to put any prior information to KANs. She can first get some clues by staring at trained KAN and only then it is her job to decide which hypothesis she wants to make (e.g.. "all activations are linear or quadratic") and implement her hypothesis in KANs. Although is not likely for KANs to return the correct answer immediately, KANs will always return something useful, and Alice can collaborate with it to refine the results.
Modified Andre-Aubry Model (MAAM) The last class of models we consider is defined by the Hamiltonian Biddle & Sarma 2010))
Le Pln-nnn' (cf,oo + VV,(,,)cc..
(33)
where t the strength of the exponentially decaying coupling in space, en (ct) is the annihilation (creation) operator at site n and the potential energy Vn is given by
(a Φ) = λ coco(22nd) + +),
(34)
As before, introduce quasiperiodicity, we & be irrational (the golden ratio). For these models, the mobility edge is given by the closed form expression Biddle & Sarmal 2010)).
A cosh(p) = E + E + t1exp(p)
(35)
where we define t = texp(-p) as the nearest neighbor hopping strength, and we set t below.
Let us assume Alice wants to figure out the mobility edge for MAAM. This task is more complicated and requires more human wisdom. As in the last example, Alice starts from a [4, 2, 1, KAN and trains it but gets an accuracy around 75% which is less than acceptable. She then chooses a larger KAN and successfully gets 98.4% which is acceptable (Figure 25] (b) step 1). Alice notices that Φ is not picked up on by KANs, which means that the mobility edge is independent of the phase factor & (agreeing with Eq. (35). If Alice turns on the automatic symbolic regression (using a large library consisting of exp. tanh etc.), she would get a complicated formula in Tabel [2] MAAM-KAN auto, which has 97. 17% accuracy. However, if Alice wants to find a simpler symbolic formula, she will want to use the manual mode where she does the symbolic snapping by herself. Before that she finds that the [4, 3, 1] KAN after training can then be pruned to be [4, 2, 1, 1], while maintaining 97.7% accuracy (Figure 25 (b)). Alice may think that all activation functions except those dependent on p are linear or quadratic and snap them to be either linear or quadratic manually by using f fix_symbol i c. After snapping and retraining, the updated KAN is shown
[4. 3, 1,
43

---

Published as a conference paper at ICLR 2025
Functionality
Descriptions
model. fit (dataset)
training model on dataset
model.plot
plotting
model model.prune()
pruning
fix the activation function 01.i.j to be the symbolic function fun suggest symbolic functions that match the numerical value of ن, use top symbolic suggestions from suggest_symbolle to replace all activation functions return the symbolic formula
model. www_syiiooiiccccacaoccoa (1, fun)
model. stggel..uggest_symmolllolo 11,
model..auto_symmbll
mode -.syyboolc_fooo
Table 13: KAN functionalities
in Figure 25 (c) step 3, maintaining 97.7% accuracy. From now on, Alice may make two different choices based on her prior knowledge. In case, Alice may have guessed that the dependence on is cosh, so she sets the activations of p to be cosh function. She retrains KAN and gets 96.9% accuracy (Figure 25 (c) Step 4A). another case, Alice does not know the cosh p dependence, so she pursues simplicity and again assumes the functions of p to be quadratic. She retrains KAN and gets 95.4% accuracy (Figure 25 (c) Step 4B). If she tried both, she would realize that cosh is better in terms of accuracy, while quadratic is better in terms of simplicity. The formulas corresponding to these steps are listed in Table I2 is clear that the more manual operations are done by Alice, the simpler the symbolic formula (which slight sacrifice in accuracy). KANs have "knob" that a user can tune to trade-off between simplicity and accuracy (sometimes simplicity can even lead to better accuracy, in the GAAM case).
V KAN FUNCTIONALITIES
Table 13 includes common functionalities that users may find useful.
W DEPENDENCE ON HYPERPARAMETERS
We show the effects of hyperparamters on the (xx, u) exp(sin(mær) + y²) case in Figure 27 To get interpretable graph, we want the number of active activation functions to be small (ideally 3) as possible.
(1) We need entropy penalty reduce the number of active activation functions. Without entropy penalty, there are many duplicate functions.
(2) Results can depend on random seeds. With some unlucky seed, the pruned network could be larger than needed.
(3) The overall penalty strength effectively controls the sparsity.
(4) The grid number G also has subtle effect interpretability. When G is too small, because each one activation function is not very expressive, the network tends to use the ensembling strategy, making interpretation harder.
(5) The piecewise polynomial order i only has subtle effect on interpretability. However, it be- haves bit like the random seeds which do not display any visible pattern this toy example.
X REMARK ON GRID SIZE
For both PDE and regression tasks, when we choose the training data on uniform grids, we witness sudden increase in training loss (i.e., sudden drop in performance) when the grid size is updated to a large level, comparable to the different training points in one spatial direction. This could be due implementation of B-spline in higher dimensions and needs further investigation.
44

---

Published as a conference paper at ICLR 2025
Offect regulorzaation
Elfect random
expaantd
plesteon
expeningur
exptenino +
Standard setup
(4) Effect lambda joverall ponalty strengtt)
Hd Eflect (sumber of gri point))
1
0.0001 appentssie?? y
A 0.01
G enplaninay
G 3 optantest ?y?
G-5 ronnteey?
polyoomial
2
k- sagaintoo 1f
=
k = exptemnte/ 4yy)
exphhon--3 -y!
Figure 27: Effects of hyperparameters on interpretability results.
Y RELATED WORKS
Kolmogorov-Arnol theorem and neural networks. The connection between the Kolmogorov-
Arnold theorem (KAT) and neural networks is not new in the literature Poggio (2022); Schmidt- Hieber 2021); Sprecher & Draghici (2002); Köppen (2002): Lin & Unbehauen (1993): Lai & Shen Ismayilova [
Ismailov (2024 Poluektov Polar 2023 but the pathological behavior of inner functions makes KAT appear unpromising in prac- tice Poggio (2022). Most of these prior works stick to the original 2-layer width- (anthh-(nn 1) networks, which were limited in expressive power and many of them even predating back-propagation. Therefore, most studies were built on theories with rather limited or artificial toy experiments. More broadly speaking. KANs also somewhat related generalized additive models (GAMs) Agarwal al. (002)) graph neural networks Zaheer et al. (2017) and kernel machines [ong et al. (2018). The connections are intriguing and fundamental but might be out of the scope of the current paper. Our contribution lies in generalizing the Kolmogorov network to arbitrary widths and depths, revitalizing and contexualizing them in today's deep learning stream, as well highlighting its potential role as a foundation model for AI + Science.
2202 Leni (2013): Fakhourr al. (002);
Neural Scaling Laws (NSLs). NSLs are the phenomena where test losses behave as power laws against model size, data, compute etc Kaplan et al. (2020); Henighan et al. (2020); Gordon et al. (2021); Hestness Hestmess et al. (2017); Sharma & Kaplan 2020); Bahri et al. (202 Michaud et al. (2023a); Song et al. (2024). The origin of NSLs still remains mysterious, but competitive theories include intrinsic dimensionality Kaplan et al. (202)). quantization of tasks Michaud et al. (2023a), resource theory Song et al. 2024), random features Bahri et al. (021), compositional sparsity PPogiio 2002), and maximum anity Michaud al. (2023b). This paper contributes this space by showing that high-dimensional function can surprisingly scale as a ID function (which is the best possible bound one can hope for) if it has smooth Kolmogorov-Arnol representation. Our paper brings fresh optimism to neural scaling laws. We have shown in our experiments that this fast neural scaling law can be achieved on synthetic datasets, but future research is required address the question whether this fast scaling is achievable for more complicated tasks (e.g., language modeling): Do KA representations exist for general tasks? If so, does our training find these representations in practice?
Mechanistic Interpretability (MI). MI is an emerging field that aims mechanistically under- stand the inner workings of neural networks Olsson et al.] (2022); Meng et al. (2022); Wang et al.
2023 Elhage et al. (2022b); Nanda al. (203)) Zhong et al. (2023) Liu al. (2023) Elhage
45

---

Published as a conference paper at ICLR 2025
et al. (2022a) Cunningham al. (2023) MI research can be roughly divided into passive and active MI research. Most MI research is passive in focusing on understanding existing neural networks trained with standard methods. Active MI research attempts to achieve interpretability by designing intrinsically interpretable architectures or developing training methods to explicitly encourage inter- pretability Liu et al. (2023); |Elhage al. (2022a). Our work lies in the second category, where the model and training method are by design interpretable.
Learnable activations. The idea of learnable activations in neural networks is not new in machine learning. Trainable activations functions are learned in differentiable way [Goyal et al. (2019);
Fakhoury et al. (2022); Ramachandran et al. ); Zhang al. (2022) or searched in discrete way Bingham & Miikkulainen (2022). Activation function are parametrized as polynomials Goyal
ct al. (2019), splines| Fakhoury et al. (2022) Bohra et al. (2020); Aziznejad & Unser (219), sigmoid
linear unit Ramachandran et al. (2017) 17).. ). or neural networks Zhang et al. (2022). KANS s use B-splines
to parametrize their activation functions.
Symbolic Regression. There are many off-the-shelf symbolic regression methods based on genetic algorithms (Eureka Dubcáková 2011). GPLeam |PPll PySR Cranmer (2023)), neural-networ based methods LELL|Martius & Lampert (2016). OccamNet Dugan et al. (2020)). physics-inspinee method (AI Feynman Udrescu& & Tegmark (2000)] ; [drreccu et al. 22200). and reinforcement learning-based methods Mundhenk et al. (2021). KANs are most similar to neural network- methods, but differ from previous works in that our activation functions are continuously learned before symbolic snapping rather than manually fixed Dubcáková (201I;; Dugan et al. (2020).
Neural Operators (PINOs). In Section PDE. we demonstrate that KANs replace the paradigm of using MLPs for imposing PDE loss when solving PDEs. We refer to Deep Ritz Method Yu et al. (2018). PINNs Raissi et al. (2019); Karniadakis et al. (202T) for PDE solving, and Fourier Neural operator Li al.
Physics-Iffrrmed Neural Networks (PINNs) and Physics-Informet
2020)) PINOs Li et al. (2021); Kovachki et al. (2023); Maust et al.) (2022). DeepONet Lu et al.
2021 for operator learning methods learning the solution map. There is potential to replace MLPs
with KANs in all the aforementioned networks.
AI for Mathematics. Al has recently been applied to several problems in Knot theory, includ- ing detecting whether a knot is the unknot Gukov et al.] (2021): Kauffman et al.) (2020) or a ribbon knot Gukov et al. (2023), ; and predicting knot invariants and uncovering relations among them Hughes (000)); Craven et al. [202] 2022); Davies et al. (202I). For a summary of data
science applications datasets in mathematics and theoretical physies see e.g. Ruehle (2000) He
(222), and for ideas how to obtain rigorous results from ML techniques in these fields, see Gukov al. (2024).
Z DISCUSSION
In this section, we discuss KANs' limitations and future directions from the perspective of mathe- matical foundation, algorithms and applications.
Mathematical aspects: Although we have presented preliminary mathematical analysis of KANs (Theorem 2. our mathematical understanding of them is still very limited. The Kolmogorov- Arnold representation theorem has been studied thoroughly mathematics, but the theorem corre- sponds to KANs with shape (n, 2n + 1, 1]. which is very restricted subclass of KANs. Does our empirical success with deeper KANs imply something fundamental in mathematics? An appeal- ing generalized Kolmogorov-Amoll theorem could define "deeper" Kolmogorov-Armole represen- tations beyond depth-2 compositions, and potentially relate smoothness of activation functions to depth. Hypothetically, there exist functions which cannot be represented smoothly in the original (depth-2) Kolmogorov-Arma Arnold representations, but might be smoothly represented with depth-3 or beyond. Can we use this notion of "Kolmogorov-Amold depth" characterize function classes?
A Algorithmic aspects: We discuss the following:
(1) Accuracy. Multiple choices in architecture design and training are not fully investigated so alternatives can potentially further improve accuracy. For example, spline activation functions might be replaced by radial basis functions or other local kernels. Adaptive grid strategies can be used.
46

---

Published as a conference paper at ICLR 2025
(2) Efficiency. One major reason why KANs slowly is because different activation functions cannot leverage batch computation (large data through the same function). Actually, one can interpolate between activation functions being all the same (MLPs) and all different (KANs), by grouping activation functions into multiple groups ("multi-head"", where members within group share the same activation function.
(3) Hybrid of KANs and MLPs. KANs have two major differences compared to MLPs:
activation functions
edges instead of on nodes.
(ii) activation functions are learnable instead of fixed.
(4) Adaptivity. Thanks the intrinsic locality of spline basis functions, we can introduce adaptivity in the design and training of KANs to enhance both accuracy and efficiency: see the idea of multi-level training like multigrid methods in Zhang et al.) 2 [uu & Zikatanov (2tpaanoooo001)))) (2017). or domain-dependent basis functions like multiscale methods as in ChChn et al. (2023).
We have presented some preliminary evidences that KANs than MLPs in science-related tasks, e.g., fitting physical equations and PDE solving. We expect that KANs may also be promising for solving Navier-Stokes equations, density functional theory. or any other tasks that can be formulated as regression or PDE solving. We would also like to apply KANs paachne-learning-rlateee
Application aspects:
effective
tasks, which would require integrating KANs into current architectures, e.g., transformers one may propose "kansformers" which replace MLPs by KANs in transformers.
KAN as a "language model" for AI + Science The reason why large language models are so transformative is because they are useful to anyone who can speak natural language. The language of science is functions. KANs are composed of interpretable functions, so when a human user stares at KAN, it is like communicating with it using the language of functions. This paragraph aims to promote the people use different languages to communicate, we expect that in the future KANs will be just one of the languages for Al + Science, although KANs will be one of the very first languages that would enable Al and human to communicate. However, enabled by KANs, the Al-Scieniss-Collaaorttion paradigm has never been this easy and convenient, which leads us to rethink the paradigm of how want to approach AI Science: Do want AI scientists, do want AI that helps scientists? The intrinsic difficulty of (fully automated) AI scientists is that is hard to make human preferences quantitative, which would codify human preferences into AI objectives. In fact, scientists different fields may feel differently about which functions are simple or interpretable. As result, it is more desirable for scientists have an AI that can speak the scientific language (functions) and can conveniently interact with inductive biases of individual scientist(s) to adapt to specific scientific domain.
paradigm rather than our specific tool KANs. Just like
47


Reviews:
[{"review_id": "fTXFX7hbeu", "reviewer": "Reviewer_cGWE", "summary": "The paper proposes a replacement for multilayer perceptrons based on the Arnold-Kolmogorov theorem which gives a two layer network for function approximation as a sum of univariate functions of individual features followed by a function of the sum. The Arnold-Kolmogorov theorem has previously been declared irrelevant for building learning machines since the involved functions can be nonsmooth [cite].\nThe paper reconsiders the Arnold-Kolmogorov theorem for building learning models and proposes that deep versions of networks using the Arnold-Kolmogorov theorem might not be limited by the non-smoothness that appears in shallow networks. The paper proposes Kolmogorov Arnold networks which are stacked layers of the univariate function+sum layers that appear in the 2-layer network.\nThe paper further gives an approximation theorem which bounds the error in the derivatives up to some order for a function approximated by a KAN network with B-splines.\nThe advantages of KANs over MLPs are said to be that KANs can be more interpretable than MLPs which makes them useful for symbolic regression tasks and are useful in data-driven scientific discovery.", "questions": "Are the constants C in the proof in appendix B and the statement of Theorem 2.1 the same? If not, different notation would be better.  \n\nDid you compare the symbolic regression results with other methods? Some baselines would be useful.\n\nAs asked above, could you please clarify how the residual error bound in the proof of Theorem 2.1 is obtained?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The main contribution of the paper is a practical instantiation of the Arnold-Kolmogorov theorem as a deep network which can avoid the difficulties plaguing shallow versions of the theorem. There are a number of experiments validating the method ranging from replicating discovery results from knot theory, automatic and interactive symbolic regression and solution of partial differential equations. The paper is generally well-written and clear.", "weaknesses": "I have a few concerns, listed in the following.\n\n1. In light of previous concerns in the literature with the Kolmogorov-Arnold theorem as a basis for learning machines, I think the paper should focus more on establishing whether the problem of non-smoothness is in fact avoided in deeper versions. Since this would establish whether the proposed model is general enough and can scale to high dimensions.\n\n2. The paper presents Theorem 2.1 which gives a bound on the supremum of the error in the derivative. However, it is not clear whether the bound is actually useful since it depends on f and the network in an unknown way through the constant C.\nIf a general theoretical argument demonstrating smoothness is too difficult, perhaps the paper could investigate the smoothness of the learned functions (or the gradients during training) to provide empirical evidence that the model learns smooth functions. \n\n3. Regarding the proof of Theorem 2.1 in Appendix B. It is not clear to me how the bound on the residual is obtained in line 894.\nIf I understand correctly, the bound on B-spline approximation in line 887 is for a single univariate function, phi_{l,i,j}, whose inputs are exact. However when the inputs from the layer below are also approximated and then summed, one would expect the bound of sup of error in the derivatives to also be summed? Whereas the error in line 887 for exact inputs and that in line 894 for approximate inputs is exactly the same. Could you clarify how the error for the residual is obtained?\n\n4. I don’t quite understand how in the symbolic regression tasks the units are automatically converted into symbolic functions. I also couldn’t find the metric used for accuracy in the SR experiments.\n\n5. I think the motivation described in lines 62-64 should be rephrased. Regardless of the aim of interpretability, smoothness is still quite essential for learning which is the point of Giros & Poggio [1989]. \n\nMinor corrections.\n\nLine 16. Interpretabe -> interpretable\n\n205. KAT -> KAN\n\n473. pareton -> pareto", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xRVzFLuIOw", "reviewer": "Reviewer_5Dsi", "summary": "This paper proposes a novel network architecture named KAN based on the Kolmogorov-Arnold representation theorem. Unlike MLPs, the KAN architecture is composed of univariate but learnable activation functions and the summation operation. It extends the Kolmogorov-Arnold representation of a function to deeper layers, which partially addresses the issue of non-smooth functions in the original two-layer Kolmogorov-Arnold representation. KAN shows good interpretability and performance in tasks that deal with symbolic formulae (e.g., applications in scientific discovery).", "questions": "1.\tAbout the experiment in Figure 5. It is known that standard MLPs (e.g., with ReLU activations) perform poorly in fitting periodic functions such as sine. Is it possible to also try MLPs with other activation functions (e.g., the sine function) to see if it can get better or even comparable performance with KAN?\n\n2.\tAbout Table 1. Does the line labeled by “Theory” refer to the ground-truth formula and the other lines the formulae predicted by KAN? If so, why does the ground-truth formula also have an “accuracy”?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1.\tThe design of KAN is well motivated and rooted in previous literature, i.e., the Kolmogorov-Arnold representation theorem. Why this theorem was not widely applied for machine learning previously is also discussed.\n\n2.\tThe presentation of the paper is clear and easy to follow.\n\n3.\tThe paper covers comprehensive aspects of the proposed architecture, including its theoretical foundation, detailed architectural design, guarantee on approximation ability, implementation details and tricks, etc. \n\n4.\tThe paper clearly states the “comfort zone” and limitations of KAN, and points out potential future directions.\n\n5.\tThe view of external and internal degrees of freedom is interesting.", "weaknesses": "One major concern is that the selection of architectural hyperparameters of KAN (e.g., the number of layers, the number of nodes in each layer) is not discussed in detail, and it seems that we need some prior knowledge of the target function when selecting these hyperparameters. More detailed concerns and questions are listed below.\n\n**1. About the theoretical guarantee of the approximation ability in Theorem 2.1.** Theorem 2.1 states that if the target function admits a structure of the composition of $L$ KAN layers with smooth activation functions, then we can use an $L$-layer KAN (with B-splines) to approximate it well. However, it does not discuss the scenario where the structure (in particular, the number of layers $L$) of the target function is unknown, which is common in real applications. I wonder if we use a shallower KAN (less than $L$ layers) or a deeper KAN (more than $L$ layers) to approximate the target function (which is assumed to have $L$ layers), how will the result in Theorem 2.1 change? It seems that a deeper KAN can still approximate the target function by setting higher layers as identity mappings, but things become more nuanced for a shallower KAN.\n\n**2. About how to select the number of layers in experiments.** Do we start from training a deep and wide KAN, and then use the sparsification trick in Appendix E to prune redundant nodes or even layers? Or do we simply apply a grid search over different number of layers and then select the best performing combination?  It is encouraged to provide a step-by-step description of the process for determining KAN architectures in the experiments.\n\nThere are also some subquestions in this part:\n\n(a)\tIf we are to first train a deep KAN whose number of layers is probably more than the number of layers of the target function, what will this KAN look like after being trained to convergence? Let us assume the target function $f$ is a composition of $L$ symbolic formulae, and the KAN we use has $M>L$ layers. Would the last $M-L$ layers behave as identity mappings, and the first $L$ layers capture exactly $f$’s structure?\n\n(b)\t(As a continuation of question (a)) Although a deeper KAN may have better performance, would the increasing depth degrade its interpretability?\n\n(c)\tIf we use a KAN that is too deep, would there be difficulties in optimization? If so, how many layers would you recommend starting with?\n\n(d)\tAre there advanced and task-dependent strategies (maybe heuristics) to select these hyperparameters, especially for cases where prior knowledge of the target function is limited?\n\n**3. About how to select the number of nodes per layer in experiments.** I find it surprising that in Line 300, a KAN with only one internal node can perform very well in the task of knot theory. That being said, it is unclear how to determine that we only need one node in the intermediate layer. Is it achieved by some prior knowledge of the task or by training a wide KAN and then pruning out redundant nodes?\n\n---\n\nAnother concern is about the necessity of the sparsity loss introduced in Appendix E. Is there any ablation study of this sparsity loss? By ablation study, I refer not only to the analysis of performance changes with and without the sparsity loss but also to how the structure of the learned KAN changes. For example, if there is no sparsity loss, is it possible for a layer to learn multiple identical functions “distributed” over different nodes, such as $ x^{2}/3, x^{2}/3, x^{2}/3$, instead of a single $ x^{2}$?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a1qCGU7KBp", "reviewer": "Reviewer_56QY", "summary": "This paper introduces Kolmogorov-Arnold Networks (KANs) as an alternative to Multi-Layer Perceptrons (MLPs), particularly when interpretability is desired, such as for extracting symbolic formulas from datasets in scientific applications. The authors contextualize the Kolmogorov-Arnold representation theorem within modern machine learning, relating it to MLPs and generalizing the representation from two layers to multiple layers via introduced KAN layers, thereby enhancing expressive power while addressing some pathological behavior of Kolmogorov-Arnold representations. \n\nThe paper aims to demonstrate the interpretability of KANs and their potential to serve as a useful tool for scientific discoveries, presenting examples from mathematics (knot theory) and physics (Anderson localization) where KANs can assist scientists in (re)discovering mathematical and physical laws. Furthermore, the authors show through theory and experiments that KANs are accurate and have favorable scaling laws, while MLPs scale more slowly and plateau quickly.\n\nThe focus of the paper is on relatively small-scale numerical examples from various scientific domains, with the scalability and extensibility of KANs to large-scale machine learning tasks left as future work. Overall, the paper introduces KANs as a promising alternative to MLPs, especially for interpretable and accurate learning in scientific applications, and lays the groundwork for further exploration of their potential.", "questions": "Here are some questions and suggestions for the authors to consider during the rebuttal and discussion phase:\n\n1. **Baselines for symbolic regression:** In the symbolic regression experiments, the authors do not include comparisons to state-of-the-art approaches like PySR (Cranmer, 2023). Such approaches have demonstrated reliable performance on a wide range of symbolic regression tasks and have been successfully applied to real-world scientific discovery problems. To better assess the effectiveness of KANs for symbolic regression, it is important to compare them against the best-performing methods in this domain. Can the authors include experiments comparing KANs to PySR and other leading symbolic regression techniques, and discuss how KANs perform relative to these state-of-the-art baselines in terms of accuracy, efficiency, and interpretability?\n\n2. **Baselines for function regression:** In the function regression experiments, to ensure a fair and meaningful comparison, the authors should consider employing best practices for MLP training, such as using appropriate network sizes, optimization algorithms (e.g., Adam or SGD with learning rate scheduling), and regularization techniques (e.g., dropout or weight decay). Can the authors update their function regression experiments to include MLP baselines trained using best practices and discuss how KANs compare to these more robustly trained MLP models?\n\n3. **Baselines for PDE solving:** In the PDE solving experiments, the authors compare KANs against vanilla MLPs with Glorot initialization, which is known to be a weak baseline in the current literature on Physics-Informed Neural Networks (PINNs). To better demonstrate the effectiveness of KANs in this context, the authors should consider comparing against more advanced and well-established PINN architectures and initialization schemes. Can the authors include experiments comparing KANs to state-of-the-art PINN methods like PirateNets and discuss how KANs perform relative to these stronger baselines?\n\n4. **Continual learning comparison:** The continual learning results presented in the paper are not entirely surprising, given that the spline activations used in KANs are compactly supported, and their local behavior can be modulated by neighboring knots without altering their global structure. For a more fair and informative comparison, the authors should consider evaluating KANs against MLPs with compactly supported activations, such as Radial Basis Functions (RBF), as networks with activations like ReLU or Tanh are inherently prone to catastrophic forgetting. Can the authors provide additional experiments comparing KANs to MLPs with compactly supported activations in the continual learning setting and discuss the implications of these results?\n\n5. **Appendix content and organization:** The Appendix contains a substantial amount of supplementary results that may not significantly contribute to the main messages of the paper and could easily be overlooked by readers. For example, the LAN approach and results presented in Appendix P are interesting but could potentially be the topic of a separate paper (although LANs appear to only introduce a small tweak to the SIREN architecture). This also raises the question of how KANs would perform for an image regression task and why such an evaluation was not provided. In general, to enhance the overall clarity and focus of the manuscript, the authors should carefully evaluate which key results from the Appendix are essential to strengthen the main messages of the paper and consider removing or reorganizing the remaining content. Can the authors provide a rationale for the current organization of the Appendix and discuss how they plan to refine it to better support the core ideas of the paper?\n\n6. **Training details and data normalization:** Some important training details, such as data normalization protocols, are omitted from the text. It appears that KANs require inputs to be in the range [0, 1]. Are the inputs to the MLP networks normalized in the same way? Inconsistencies in data normalization can lead to unfair comparisons and affect the interpretation of the results. Can the authors clarify their data normalization procedures for both KANs and MLPs and discuss any potential impact on the reported findings? \n\n7. **Computational complexity of grid extension and refinement:** The paper introduces the concepts of grid extension and grid refinement for KANs but does not provide a detailed discussion on their computational complexity. Understanding the computational overhead associated with these operations is crucial for assessing the practical applicability of KANs in various settings. Can the authors provide a more in-depth analysis of the computational complexity of grid extension and refinement, including theoretical bounds and empirical measurements, and discuss the implications for the scalability and efficiency of KANs? Moreover, the update of spline grids mentioned in p. 16 seems to be a very important step of the algorithm but it is not adequately explained.\n\n8. **Code availability:**  The authors did not provide any code with this submission and I was unable to assess their implementation. This may have helped with resolving some of the above questions.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper presents a novel neural network architecture inspired by the Kolmogorov-Arnold representation theorem (KST). The authors creatively combine the insights from the KST with modern deep learning techniques, generalizing the original two-layer representation to multiple layers through the introduction of KAN layers. This novel approach to network design demonstrates originality and has the potential to open new avenues for research at the intersection of deep learning and approximation theory.\n\n\nThe authors provide extensive numerical evaluations across a diverse collection of experiments. There they aim to showcase the effectiveness of KANs in various domains, including synthetic toy datasets, special functions, physics equations (Feynman datasets), and partial differential equations. These experiments aim to provide a comprehensive assessment of KANs' performance, interpretability, and scaling laws compared to traditional MLPs. \n\n\nThe paper is well-structured and clearly written, making it accessible to a broad audience. The authors provide sufficient background information on the KST and its relation to neural networks, ensuring that readers can understand the motivation behind KANs. The mathematical formulation of KANs and the approximation theory are presented in a rigorous and understandable manner.\n\n\nThe significance of this work lies in its potential to bridge the gap between deep learning and approximation theory, as well as its implications for interpretable and accurate learning in scientific applications. The authors demonstrate that KANs can be effective in discovering mathematical and physical laws, showcasing some early promise as a tool for scientific discovery. Moreover, the scaling laws exhibited by KANs suggest that they may be a promising alternative to MLPs in certain domains.\n\n\nI would also like to acknowledge that this research has already generated considerable interest in the recent literature, with several studies exploring the possibilities brought by combining deep learning with the KST. The paper has opened new avenues for research, as the community seeks to assess and enhance the capabilities of KANs and related architectures.", "weaknesses": "While I believe that this paper puts forth a creative idea that is worthwhile publishing, the current presentation makes it very difficult to assess whether KANs are indeed a good and competitive method to employ for all the different tasks presented in this work: symbolic regression, function regression, PDE solving, etc. The main reason is that the vast majority of numerical evaluations are poorly designed, either by disadvantaging competing methods, or by comparing against very weak baselines, or by providing no comparison to competing methods (for example, in the symbolic regression benchmarks).\n\n1. **Comparison to state-of-the-art methods:** While the paper presents promising results for KANs in various application settings, such as symbolic regression and solving PDEs, the effectiveness of KANs compared to state-of-the-art methods in each domain remains unclear. The authors primarily compare KANs against basic MLP architectures, which may not represent the best-performing approaches in these specific areas. To provide a more comprehensive assessment of KANs' capabilities, it would be beneficial to compare them against well-established and state-of-the-art methods that are commonly used in practice for each application. This would help to better understand the relative strengths and weaknesses of KANs and their potential to advance the state of the art in these domains.\n\n2. **Overstated claims and conclusions:** Due to the lack of comparisons against state-of-the-art methods in each application area, the current claims and conclusions in the paper may be somewhat overstated. While the results demonstrate the potential of KANs, it is difficult to assess their true effectiveness and superiority without a more comprehensive evaluation against well-established benchmarks and leading approaches. The authors should consider tempering their claims and conclusions to reflect the limitations of their current experimental setup and the need for further validation against state-of-the-art methods.\n\n3. **Interpretability claims and symbolic regression protocol:** The authors emphasize the interpretability of KANs as one of their key strengths. However, the symbolic regression protocol presented in Section E.2 and Figure 10 appears to be largely heuristic and sensitive to various hyperparameter choices. The effectiveness of this protocol in recovering interpretable models from real-world data remains questionable, as the current results focus on rediscovering known scientific formulas. Moreover, the authors do not provide a comparison against state-of-the-art symbolic regression alternatives, making it difficult to assess the relative interpretability of KANs. The arguments against symbolic regression methods in the appendix are not entirely convincing, given the reported success of such methods in challenging settings [1]. To strengthen the interpretability claims, the authors should consider evaluating KANs against leading symbolic regression approaches and demonstrating their effectiveness in recovering interpretable models from real-world data.\n\n4. **Unconventional protocols for training MLPs:** The authors employ unconventional protocols for training MLPs, such as using networks with small widths and full-batch L-BFGS optimization for a limited number of training steps. While these choices may be made to match the parameter complexity and training cost of KANs, they could potentially disadvantage the effectiveness of MLPs and render the comparisons and conclusions questionable. Recent results [3,4] suggest that a fair comparison between KANs and MLPs may lead to different conclusions than those reported in this paper. To address this concern, the authors should consider using more standard and well-established training protocols for MLPs, ensuring a fair and unbiased comparison between the two architectures.\n\n5. **Comparison against plain MLPs in PINNs:** In the context of Physics-Informed Neural Networks (PINNs), the authors compare KANs against plain MLP networks initialized using the Glorot scheme. However, this setup is known to yield a poor baseline in the current PINNs literature, as the derivatives of such networks have limited expressive power to adequately minimize PDE residuals [citation needed]. To provide a more meaningful comparison, the authors should consider evaluating KANs against state-of-the-art PINN frameworks such as PirateNets [5] that have been specifically designed to address the challenges of PDE solving. This would help to better understand the potential advantages of KANs in the context of PINNs and their ability to advance the state of the art in this domain.\n\n[1] Cranmer, M. (2023). Interpretable machine learning for science with PySR and SymbolicRegression. jl. arXiv preprint arXiv:2305.01582.\n\n[2] Cranmer, M., Sanchez Gonzalez, A., Battaglia, P., Xu, R., Cranmer, K., Spergel, D., & Ho, S. (2020). Discovering symbolic models from deep learning with inductive biases. Advances in neural information processing systems, 33, 17429-17442.\n\n[3] Yu, R., Yu, W., & Wang, X. (2024). Kan or mlp: A fairer comparison. arXiv preprint arXiv:2407.16674.\n\n[4] Shukla, K., Toscano, J. D., Wang, Z., Zou, Z., & Karniadakis, G. E. (2024). A comprehensive and FAIR comparison between MLP and KAN representations for differential equations and operator networks. arXiv preprint arXiv:2406.02917.\n\n[5] Wang, S., Li, B., Chen, Y., & Perdikaris, P. (2024). PirateNets: Physics-informed Deep Learning with Residual Adaptive Networks. arXiv preprint arXiv:2402.00326.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IEraUGybw9", "reviewer": "Reviewer_whLk", "summary": "The authors propose a novel neural network architecture based on the Kolmogorov-Arnold Representation Theorem (KAT), called Kolmogorov-Arnold Networks (KANs). Although the KAT has been studied in the context of neural networks, KANs differ from most other works by allowing for wider/deeper representations, in addition to implementing functions implied by the theorem via B-splines. The authors formally describe the main component of their new architecture, the KAN-layer, which draws inspiration from the representation formula implied by the KAT. In essence, a KAN-layer works by approximating the functions implied by KAT via B-splines, as well as adding a 'residual' connection using a silu activation. These KAN-layers can then be composed together to create a deeper KAN architecture. The authors also state a theorem regarding the approximation capabilities of KANs, assuming a specific smooth representation of the target function.\n\nThe authors then go on to detail experiments using their new architecture on two main goals: a) providing interpretable results from scientific data; and b) creating accurate predictions for regression tasks and learning the solution of a PDE. On their first goal, they evaluate KANs on tasks from mathematics (knot theory) and physics (Anderson localization), as well as some toy problems on symbolic regression. For their second goal, they compare KANs against certain MLPs on regression for several smooth functions with closed form solutions, as well as a physics-informed problem where they learn the solution of a 2D Poisson equation. \n\nFinally, the authors summarize their results and argue that KANs offer significant advantages over traditional MLPs for scientific tasks that are \"small-scale\", leaving more thorough experimentation on larger-scale problems for future work.", "questions": "I detail below some questions/suggestions to the authors, in order of how important they would be toward increasing the review score of their submission.\n\n- **[More Challenging Problems For Testing Accuracy]** As mentioned above, I think the paper would greatly benefit from more challenging benchmarks, as the ones reported are too simple and largely not relevant to scientific computing. Seeing KANs outperform realistic MLPs (using higher widths and Fourier features, for example) on more challenging tasks could highly increase my score for the submission, depending on the results. Some potential tasks are detailed above in the previous section.\n\n- **[Comparison Against Symbolic Regression Baselines]** The authors highlight discovery of close-form expressions from data as a key feature of KANs, but experiments lack comparison against other symbolic regression baselines, as the ones detailed in lines 2115-2121. Seeing KANs outperform these baselines could make for a much stronger paper.\n\n- **[Spline Grid Boundary]** Using B-splines require all inputs to fall within a specified interval, which may not be the case for intermediate layers of a deep KAN. On line 852, the authors mention they \"update each grid on the fly according to its input activations,\" but no further information is given as to how this is done, to the best of my knowledge. I feel like this is an important aspect to detail in the paper, maybe even in the main text. Related to this, how would the authors deal with problems where the inputs are defined in an unbounded (or very large) domain, such as when inputs follow a distribution with a 'fat' tail?\n\n- **[Skip Connections]** What is the effect of removing the silu 'basis function' used in the KAN layer? Are there other suitable functions to choose instead of silu? This would be a good ablation to consider reporting.\n\n- **[Grid Extension]** The authors propose progressively extending the grid used for computing B-splines, which improves the predictive accuracy of a given model. Is there an advantage to progressively increasing this grid as opposed to starting with a large one? That would be an interesting ablation to include in the paper.\n\n- **[Continual Learning]** For the continual learning experiment, what happens if a deeper KAN is used? It makes sense that the local nature of B-spline computations yields this nice feature, but it seems to me that once more compositions are at play this property would be lost.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "### Originality/Significance: \nThe paper presents several interesting original thoughts, and helps bring attention to new forms of computation to scientific deep learning. Although the connections between KAT and neural networks are quite old, dating back to the 1960s and 1970s, their work helps renew attention to this field, and applies modern knowledge about neural networks to improve on previous failed attempts of using KAT for applied problems. The authors also present interesting ideas about interpretability for deep neural networks, by using KANs to guide the discovery of closed form expressions for unknown functions.\n\n### Quality:\nThe main idea (exploiting deeper/wider forms of KAT) is well-motivated and appears theoretically promising. The authors provide valuable arguments on section 2 about why such an idea could work. The paper also comes with many experiments being conducted in the main text, as well as in the appendix. Overall, I believe this paper contains very interesting ideas, and helps spark discussion about an old topic using modern lenses.\n\n\n### Clarity:\nThe ideas are presented in a mostly clear manner, with helpful figures to illustrate the main arguments. Their main theorem (Theorem 2.1) is clearly stated in the main text, including all of its assumptions.", "weaknesses": "## Summary of weaknesses:\nIn summary, although the main idea of the paper seems very interesting, my main criticism of this submission is regarding their experiments, which are not conclusive to me as a reviewer. Despite presenting a very large number of different experiments, they are all very simple, and comparisons against MLPs appear to be using very weak baselines. The authors do recognize that they mainly consider small-scale experiments [line 485], but even for scientific tasks there are more realistic and interesting problems to be considered, as will be detailed later. In addition to this, their main theorem regarding approximation power of KANs (theorem 2.1), which implies very strong scaling laws, use an assumption which is known to be wrong for KAT functions (ie: inner functions being $(k+1)$-times continuously differentiable). I further detail these criticisms below, and how the authors might address them.\n\n## Experimental Setup:\nI will mainly focus on the results reported in section 4 here, which is an area I am more familiar with. My main criticisms regarding experiments is that a) all functions considered are very well-behaved and not something for which deep learning would be used in practice, with other regression tools generally being better suited; b) I am not confident that their comparisons against MLPs are done in a consistent and fair way; c) there is not enough information to fully reproduce experiments. I expand on each of these points below.\n\na) For regression tasks (Toy datasets + Special Functions + Feynman Datasets), all the target functions are very simple, beyond what is typical for scientific machine learning. The functions from the \"Special Functions\" dataset are slightly more complicated than from the other two, but still a very simple task given enough training data. A better testbed for examining approximation properties and accuracy would be for example: 1) regression against complicated PDE solutions from publicly available dataset such as PDEBench and/or PDEArena, which are more relevant to scientific computing; 2) regression against images, which was in fact considered in the appendix for their \"LAN\" architecture, but not KANs; 3) weather modeling, using public datasets such as ERA5 (very challenging) or NOAA's SST-V2, for example. Regarding the experiment on PINNs, it would also be more comprehensive to study more than just the Poisson equation, which is again very simple for tasks of this nature. Suggestions for other PDEs to consider, which are not unrealistic for the first iteration of a new architecture include the Allen-Cahn equation and the Darcy problem. In short, although it is true that some problems in scientific computing are simpler or lower dimensional than ones in computer vision and NLP, which are big topics in deep learning, there is still no shortage of more realistic problems in science, which would provide a better testbed for their architecture.\n\nb) Many aspects of the experiments provided are not likely to yield a fair comparison against MLP architectures. In some cases, only MLPs of very small width (such as 5 or 20) are considered, which is not realistic of what someone would use in practice, even if the number of parameters is similar to that of KANs used. For some of the regression tasks, the grid definition of KAN's B-splines can go as high as 1000, which means that a KAN of comparable width/depth still presents far more parameters (and likely flops) than an MLP. To address this issue, I would recommend keeping a fixed number of trainable parameters (or flops/training time), comparing different hyperparameter combinations, both for KANs and MLPs. Ideally, MLPs should have widths of at least 32 or 64 for any real scenario. In addition to this, MLPs are more efficiently trained using the Adam optimizer, whereas all experiments used the L-BFGS algorithm, which is not as common and can often lead to MLPs being stuck on local minima.\n\nc) There are many missing details for reproducing the experiments in the paper, and no code was provided as supplementary material. For example, on many of the regression tasks, the target functions are reported, but not the total number of points in the training/testing set, nor what the domain for these functions is. Other missing information for reproducibility include: 1) the batch size used during training for different benchmarks; 2) the learning rate of the optimizer; 3) the architectures used in the continual learning example. The best way of solving all this missing information issue would be to **provide the code used for each experiment as a supplementary material**, organized neatly for each benchmark.\n\n\n## Theoretical Assumptions:\nA main assumption of theorem 2.1 of the paper is that the functions $\\phi_{l,i,j}$ are $(k+1)$-times continuously differentiable. However, the KAT only guarantees that these functions are continuous, as and the authors themselves mention, it has been shown in the literature that these functions can provably be very chaotic, with infinite set of non-differentiability. The authors argue that by using wider/deeper representations, these irregularities may disappear, but this would require further theoretical evidence. I believe there are more reasonable ways of proving universal approximation guarantees for KANs, albeit without the strong scaling laws implied by the $k$-order spline functions. This main theorem therefore can likely be stated differently, in order to obtain a weaker but more theoretically reasonable statement.\n\n\n\n\n## Other minor considerations that did not affect my score:\n- [line 15 + many other mentions] The analogy of referring to computations on 'edges' vs 'nodes' is not very clear or compelling to me. Depending on how you chose to represent any given architecture as a graph, it could be argued either way whether an operation happens at an 'edge' or at a 'node'.\n- [line 54] Although it is true that \"In science, symbolic functions are prevalent,\" that is not generally the case in scientific computing, where solutions are often defined on complicated domains without reasonable closed-form expressions. In fact, for tasks where numerical solutions are needed (such as solutions to ODEs, PDEs, etc), it is almost always the case that the solutions are not symbolic.\n- [line 260] There are only three examples in Figure 2, not six.\n- [Figure 7] It is more common for the y-axis of the plots to show the error itself, not the error squared, as is done in the figure.\n- [line 875] It would also be nice to see a comparison on the computational complexity of KANs vs MLPs, in addition to parameter count.\n- [line 1819] In the case of PINNs, it is almost always better to sample points randomly and independently at each iteration, as opposed to maintaining a fixed grid. I would be interested to see how KANs perform in this setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YHKUuXmslD", "reviewer": "Reviewer_zHkY", "summary": "This paper proposes Kolmogorov-Arnold networks (KANs) as a more interpretable alternative to multi-layer perceptrons (MLPs), particularly for scientific applications. KANs replace the fixed activation functions on nodes in MLPs with learnable activation functions on edges, represented by B-splines. The paper explores the theoretical approximation capabilities of KANs, proposes training techniques for interpretability and accuracy (pruning, symbolifying, grid extension), and demonstrates their application in mathematics (knot theory), physics (Anderson localization), and other (toy) examples.", "questions": "* How do KANs compare to other interpretable models, like Neural Additive Models or symbolic regression methods, in terms of accuracy, interpretability, and computational cost, especially on larger datasets?\n\n* Can you provide a more detailed analysis of the computational complexity of KANs, both for training and inference? Have you explored any optimizations beyond the \"multi-head\" approach?\n\n* Can you elaborate on the dependence of the constant C in Theorem 2.1 on the dimension? How does this impact the scalability of KANs to higher-dimensional problems?\n\n* Can you provide empirical evidence supporting the claim that KANs can naturally work in continual learning without catastrophic forgetting, beyond the toy example in Appendix J?\n\n* How sensitive are KANs to the choice of hyperparameters, such as the grid size $G$ and the spline order $k$? Have you explored techniques for automatically selecting these hyperparameters?\n\n**Additional Feedback** (this does not affect my score)\n\n* In my opinion, some formulations, e.g. \"the Kolmogorov-Arnold representation theorem was basically sentenced to death\" (l.119), or \"the fastest scaling exponent ever\" (l.1888) appear overly \"dramatic\" for a scientific paper. I suggest the authors scan their paper for cases like these and try to use more neutral/grounded language.\n\n* There are several typos, e.g. \"altnertives\" (abstract), \"On\" (l.084), \"staricase-like\" (l.965), \"theLANs\" (l.1705), \"maximu\" (l.1885). I suggest the authors use an automated spell-checker to find problems like these and correct them (I likely didn't find all typos).\n\n* Some terms are spelled inconsistently in different places of the text, for example \"Deepmind\" vs. \"deepmind\" (I actually would recommend to use \"DeepMind\" instead), or \"ReLU\" vs. \"ReLu\". The authors should decide on one spelling and keep it throughout the paper.\n\n**In summary**\nI believe this is a very interesting paper that should be published, but I feel major revisions are necessary (see above). If the authors implement these revisions (or at least most of them), I will gladly raise my score.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The paper presents an interesting and original neural network architecture inspired by the Kolmogorov-Arnold representation theorem, which has very appealing properties for some applications (e.g. interpetability).\n\n+ The paper is generally well-written and easy to follow. The architecture of KANs is clearly explained, and the figures effectively illustrate the differences between KANs and MLPs.\n\n+ The emphasis on interpretability (and the ability to extract symbolic formulas) is relevant for scientific applications, where understanding the underlying mechanisms learned by the model is crucial.", "weaknesses": "- Technically, the paper could be considered to be over the page limit. While the main text does keep to 9 pages, this is only the case because extremely important sections, e.g., related works and the discussion, are moved to the appendix. I feel important sections like these have to be included in the main text, even if this means other parts of the text need to be shortened to make space for them. I am sure many authors of other submissions have faced similar \"space issues\" and consequently had to cut parts from their main text. This paper is the only submission I am reviewing that \"misuses\" the appendix to circumvent the page limit restrictions (I think the appendix should be reserved for things like detailed proofs or additional, but non-essential, information). Sorry for bringing up a technicality like this, but I feel it is necessary in the interest of fairness.\n\n- The paper contains very strong claims, e.g. that KANs with finite grid size beat the curse of dimensionality. At the same time, the authors admit that the constant $C$ in Eq.11 (Theorem 2.1) implicitly depends on the dimension (but details are left to future work). As the saying goes, extraordinary claims require extraordinary evidence, which unfortunately is missing here. The authors should either weaken their claims or provide strong empirical evidence.\n\n- While reading the paper, I couldn't help but feel that the name \"Kolmogorov-Arnold network\", at least to some degree, overstates the difference to MLPs: KANs and MLPs have more in common than a superficial reading of the paper would suggest. Comparing the Kolmogorov-Arnold Theorem (KAT) with the Universal Approximation Theorem (UAT), a very important aspect is that KAT requires a width of only $2n+1$, whereas the UAT only holds in the limit of infinite width. I understand that relaxing the width-restriction from KAT may be practically useful (and the authors motivate this choice well), but it also blurs the lines between KAT and UAT. To me, KANs seem more like a KAT-inspired re-interpretation of MLPs (rather than something entirely new), which is useful for tasks where a symbolic interpretation of the learned function is desirable. For example, one distinguishing feature of KANs pointed out in the abstract are the learnable activation functions, wheareas MLPs have a fixed activation function. But if I re-write an MLP $W_2\\sigma(W_1\\mathbf{x})$ as $W_2\\phi(\\mathbf{x})$ with $phi(\\mathbf{x}) = W_1\\mathbf{x}$, I can also consider $phi$ to be a \"learnable activation function\". Similarly, the basis functions of the B-splines in KANs could be equivalently considered simply as additional input features, similar to Fourier features (https://arxiv.org/abs/2006.10739), or other tricks that were sometimes used in older works (e.g. feeding the square of an input to a layer, see e.g., \"Neural Networks: Tricks of the Trade\"). I think the authors should try to \"de-mistify\" KANs by pointing out and discussing these similarities between MLPs and KANs more transparently.\n\n- While the examples in mathematics and physics are interesting, the scale of these problems is relatively small. The paper lacks evaluation on larger, more complex datasets. This makes it difficult to assess the true scalability and practical utility of KANs for many real-world applications. For example, a very simple test could be to compare KANs vs. MLPs on something like MNIST. Even if KANs perform worse than MLPs here, this does not mean at all that they are not useful. However, it would certainly help readers to assess whether KANs (in their current form) are likely to be directly relevant for their work or not. I encourage the authors to include at least one example like this.\n\n- The example in Appendix J on continual learning is practically not relevant and too trivial. Of course, the B-splines in KANs are \"local\" and thus the corresponding weights/coefficients are not affected by catastrophic forgetting in this setting, but the toy example is also clearly constructed to take advantage of this fact. The authors should either demonstrate that KANs are less affected by catastrophic forgetting than MLPs in a more realistic/\"less constructed\" setting, or remove this claim from the paper.\n\n- The paper mentions that KANs are slower to train than MLPs by roughly a factor of 10, and that KANs cannot leverage batch computation in the same way as MLPs. However, these discussion are hidden in the appendix. I think this is a highly important and practically relevant aspect, which should be discussed more prominently and in greater detail in the main text. An analysis of the computational cost, including training time and memory usage, is needed to fully understand the trade-off between interpretability and efficiency.\n\n- The appendix is very long with >25 additional pages, not all of which feel strictly necessary to me. I think the authors should consider cutting some parts of the appendix in the interest of brevity. For example, the secion on LANs, while interesting, feels better suited as a separate submission (after doing more experiments/refining the idea).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)