Kolmogorov-Arnold Transformer
Authors: Xingyi Yang
Xinchao Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Kolmogorov-Arnold Network; Transformer
Primary area: foundation or frontier models, including LLMs
TL;DR: In this paper, we introduce the Kolmogorov–Arnold Transformer (KAT), a novel architecture that replaces MLP layers with Kolmogorov-Arnold Network (KAN) layers to enhance the expressiveness and performance.

Abstract: Transformers stand as the cornerstone of mordern deep learning. Traditionally, these models rely on multi-layer
perceptron (MLP) layers to mix the information between channels. In this paper, we introduce the Kolmogorov–Arnold
Transformer (KAT), a novel architecture that replaces MLP layers with Kolmogorov-Arnold Network (KAN) layers to
enhance the expressiveness and performance of the model. Integrating KANs into transformers, however, is no easy
feat, especially when scaled up. Specifically, we identify three key challenges: (C1) Base function. The standard B-spline
function used in KANs is not optimized for parallel computing on modern hardware, resulting in slower inference speeds.
(C2) Parameter and Computation Inefficiency. KAN requires a unique function for each input-output pair, making the
computation extremely large. (C3) Weight initialization. The initialization of weights in KANs is particularly challenging
due to their learnable activation functions, which are critical for achieving convergence in deep neural networks. To
overcome the aforementioned challenges, we propose three key solutions: (S1) Rational basis. We replace B-spline functions
with rational functions to improve compatibility with modern GPUs. By implementing this in CUDA, we achieve faster
computations. (S2) Group KAN. We share the activation weights through a group of neurons, to reduce the computational
load without sacrificing performance. (S3) Variance-preserving initialization. We carefully initialize the activation weights
to make sure that the activation variance is maintained across layers. With these designs, KAT scales effectively and readily
outperforms traditional MLP-based transformers. We demonstrate the advantages of KAT across various tasks, including
image recognition, object detection, and semantic segmentation. It consistently enhances performance over the standard
transformer architectures of different model sizes.

Published as a conference paper at ICLR 2025
KOLMOGOROV-ARNOI TRANSFORMER
Xingyi Yang, Xinchao Wang"
National University of Singapore xyangêu.. nus. edu, xinchao@nus. edu. sg
ABSTRACT
Transformers are the cornerstone of modern deep learning. Traditionally, they use multi-layer perceptron (MLP) layers mix channel information. In this paper, we introduce the Kolmogorov -nnold Transformer (KAT), which replaces MLP layers with Kolmogorov-Arokk Network (KAN) layers to enhance model expressiveness. Integrating KANs into transformers, however, is no easy feat, especially when scaled up. Specifically, identify three key challenges: (C1) Base function. The standard B-spline used in KANs is inefficient for parallel computing, slowing inference. (C2) Parameter and Computation Inefficiency. KAN requires unique function for each input-outpus pair, leading to high computational cost. (C3) Weight initialization. The initialization of KANS is particularly challenging due to their learnable activation functions. To overcome the aforementioned challenges, we propose three key solutions: (S1) Rational basis. We replace B-spline functions with rational functions to improve compatibility with modern GPUs. By imple- menting this in CUDA, we achieve faster computations. (S2) Group KAN. We share activation weights across groups of neurons to reduce computation without initialization. We initialize activation weights to maintain variance across layers. With these designs, KAT scales effectively and readily outperforms traditional MLP-based transformers. We demonstrate the advantages of KAT across various tasks, including image recognition, segmentation, detection, table classification, and graph classification. It consistently enhances performance over the standard transformer architectures of different model sizes.
sacrificing performance. (S3) Suriance-preservvnn
INTRODUCTION
Transformers have become the de facto architecture in deep learning, widely adopted in computer
vision (Dosovitskiy et al. 2021) and natural language processing (Vaswani al. 2017). At their core,
transformers rely on two main components: attention modules and multi-layer perceptrons (MLPs).
Although significant research has explored replacing the attention mechanism (Liu et al. 2021 2022,
Tolstikhin et al., 2021). most variants still rely heavily on MLPs. Surprisingly, there have been few attempts (Shazeer (020) improve MLPs themselves.
Opening up the box, MLPs consist of stacked linear layers with non- linear activations. Theoretically. they can approximate any function given enough neurons (Hornik et al. 1989). However, ML.Ps come with their own problems. They struggle to model complex functions, like fitting periodic patterns with ReLU-like activations. Training these networks often results in slow convergence for high-frequency components (Rahaman et al. 2019 Basri et al. 2020 Ronen et al. 2019). These limitations have driven researchers to explore more expressive alternatives to MLPs.
Recently, Kolmogorov-Arnold Networks (KANs) emerged powerful alternative. KANs are noted for their thcoretical parameter efficiency, potentially requiring fewer parameters to model complex functions (Liu et al. 2024b Yu et al. 2024 Bozorgasl & Chen 2024a; Liu et al. (024a). The key to such success is the learnable base functions for each input-output pair, often parameterized by B-spline curves (Unser et al. 1993 Gordon & Riesenfeld 1974). Given its potential, integrating KANs into transformers (Vaswani el al. 2017) becomes an exciting topic.
"Corresponding Author

---

Published as a conference paper at ICLR 2025
+
+
Accuracy FLOPs Model Size
KAN
GR-KAN
ImageNet Tap-1 Accuracy (%)
KAT
Nors
Norm
Norm
Be11
Attantion
Aztotion
Attandinn
Norm
Norm
Model Sizes
20M
Input Emb.
Input Emb.
Input Emb.
VET KAN
40M
SOM
Transformer
VIT + KAN
KAT[Ours)
MACS (G)
Figure 1: (Left) Architecture of standard transformer (c.g. ViT), ViT+KAN which replaces the MLP with a KAN, and our KAT model. In KAT, the MLP layers are replaced with GR-KAN layers. (Right) Performance on the ImageNet dataset. KAT" indicates that the model was initialized using a pre-trained ViT. Generally, KAT outperforms both the VIT and DeiT models. ViT+KAN performs poorly ImageNet-level training.
Unfortunately, this ambition has been met with limited success. In particular, KANs have been reported to be "10x slower than MLPs, given the same number of parameters". Initial efforts in vision recognition tasks have been disappointing. Even on a small scale, these studies have consistently fallen short of matching, let alone surpassing. the performance of traditional architectures. This lack of improvement often attributed to the limited computational resources and ongoing scalability
problems [Cheon 2024a Bodner al. 2024 Cheon 2024a b).
In preliminary experiment, we replaced the MLP layers in the Vision Transformer (ViT) with KAN layers, creating the ViT+KAN model. As shown in Figure T (Right), this straightforward substitution led to significant challenges during ImageNet-scale training and resulted in poor performance. Scalability, therefore, remains major obstacle for KAN-based models.
Motivation and Challenges. Through dedicated analysis, we have identified several key challenges that hinder the effectiveness of KANs large-scale applications, ultimately limiting their scalability.
(C1) Base function. Standard B-spline functions in KANs are incompatible with modern GPUs because they rely on recursive computations. This recursive nature significantly slows down performance, even in highly optimized implementations.
(C2) Parameter and Computation Inefficiency. KANs require unique parameters and base functions for each input-output pair. leads to exponential growth in parameters and computation overhead the network scales.
(C3) Weight initialization. KANs use the same weight initialization as MLPs. But this approach does not ensure convergence, leading to instability and degraded performance.
Our Approach. In this paper, introduce Kolmogoron-Aroold Transformer (KAT), which suc- cessfully integrates KANs into transformers for large-scale training scenarios such as ImageNet. Beyond simple replacement, We have developed three key innovations (S1-S3) address these challenges (C1-C3) respectively.
(S1) Rational activation. We use rational functions as our base function with full CUDA implementation, improving efficiency and compatibility with modern GPUs.
(82) Group KAN. We share function coefficients and base functions among groups of edges. This strategy reduces computational load significantly without sacrificing performance.
(S3) Variance-preserviee initialization. We carefully initialize the weights to keep the activation variance consistent across layers. This ensures stability during training and improves performance.
By combining solutions S1-S3, we present a new variant of KAN, called Group-Rationas KAN (GR- KAN), to replace the MLP in transformer. GR-KAN is computationally efficient, easy implement, and seamlessly integrates into transformers. Furthermore, our designs allow KAT to load pre-trained weights from VIT models and continue training achieve even better results.
We empirically validate KAT across a range vision tasks, including image recognition, object detection, and semantic segmentation. The results demonstrate that KAT outperforms traditional
2

---

Published as a conference paper at ICLR 2025
MLP-based transformers, with similar computational requirements. As shown in Figure KAT-B achieves 82.3% accuracy on ImageNet- IK, surpassing the ViT-B by 3.1%. When initialized with pre-trained weights from VIT, the performance further improves to 82.7%.
The contributions our paper are threefold. First, we thoroughly analyze the challenges in scaling KAN-based models, particularly focusing inefficiencies in base functions, parameterization, and weight initialization. Based on this analysis, we propose a set solutions: rational activation functions tailored for GPU efficiency, Group KAN reduce computational overhead, and variance- preserving initialization to ensure stable training. Second, leveraging these insights, we introduce the Kolmogorov-ANe Amnol Transformer (KAT) and scale ImageNet-level training. Third, we validate our approach through extensive experiments, showing that KAT not only matches but surpasses the performance of ViT models, all under similar computational requirements.
PRELIMINARY
2.1
KOLMOGOROV-ARNOLJ REPRES THEOREM
The Kolmogorov-Amon representation theorem (Hecht- NiehleN 1987 states that any multivariate continuous function f. defined in a bounded domain, be expressed finite composition of continuous univariate functions and addition. Specifically, for smooth function f [0. 1|" - R, be represented as:
2n++
f(1 Inn
T
Σ
(ap(
Here, each function Q. [0, 1] + and +b (2d+1)(d+1) univariate functions b, and 6y.p are enough for an exact representation d-variate function.
R are continuous. This means that the
This theorem can be written in matrix form as follows:
f(x) = ouu 4 xx
(I)
where d and & out defined
(1,1(-)
L1,(()
Pout = (1)()
22)+1
(2)
(28+111
(20+11((.))
This decomposition illustrates how f can be built from simpler functions, showcasing essential property multivariate continuous functions.
2.2 KOLMOGOROV-ARNOI. NETWORKS
Inspired by the Kolmogorov- Kolmogorov-Amole layer to learn a univariate non-linear function for each edge. By summing these univariate functions, the layer can approximate any multivariate function. Formally, a Kolmogorov- Aroold layer with dm dimenninsoa inputs and dou- dim-diinnsi outputs is illustrated as
representation theorem, (Liu et al. 2024b) define generalized
(1,1())
$1 di (-)
f(x) Too [E (1.)
Edi diddom (I:) where
6dmn. (.)
ddnn (.)
(3)
Note that Eq 3 be KAN network is stacking of L layers: given an input vector Xo E RRdo the output of KAN is K KAN(xx) & --1 2r-2 .... do Xo.
generalized form of Eq 1 such that + Pm Paut- general
In practice, LLu et al. 2024b) parameterizes & use linear combination of SiLU activation (Elfwing al. 2018 and B- B-spline function
(()) = w.s w,silu(a)) lu(x) + w,spline(z)) where silu(x)
--e-x² =
x
spline(I) = (B)(())
(4)
3

---

Published as a conference paper at ICLR 2025
3 WHY ORIGINAL KAN FAILS TO SCALE?
This section examines the scalability of KAN. It is hindered by three factors: the choice of base function, redundant parameters and computations, and initialization problems. These design choices make the vanilla KAN resource-intensive and difficult to apply to large-scale models.
B-spline not GPU Friendly. The use B-spline functions in KAN layers introduces challenges when implemented on GPUs. First, B-splines are not standard functions in CUDA, so using them via PyTorch or NumPy lacks optimized support, resulting slower performance. Second. the localized computations of B-splines complicate parallel processing: each control point affects only small area, leading to sparse or recursive operations that GPUs handle less efficiently. While efficient implementations exist for cubic B-splines (Ruijters & Thévenaz 2012 Ruijters al. 2008 Sigg & Hadwiger 2005), scaling these methods to higher orders is not straightforward.
Parameter and Computation Inefficiency. KAN differs from standard neural networks by using a learnable base function for each input-output channel pair. This design increases the number of parameters and computational demands, especially as the network's s width and depth grow.
In a typical KAN layer with dyn input and dout output channels, a B-spline function order K over G intervals is assigned to each input-output pair. This results in (din, x done) (G K + + dout learnable parameters. In contrast, a standard MLP requires only (din. x dout)) + dout parameters.
For computation Yu et al. 2024) the FLOPs required process one sample using the De Boor-Cox iterative formulation (Boor
1971 are |LOPPs of non-linear function x din + (din X doot) x [9 x
(G + 1.5K) + 2G 2.5K + 3] In contrast, the FLOPs for an equivalent MLP layer is merely
FLOPs of non-linear function dout 2 x (din dout))
Overall, KAN's parameter size and computational effort are O(G + K) and O(GK) times greater than those of conventional MLP, respectively. This makes scaling up KAN challenging.
Weights are not Properly Initialized. Proper weight initialization is essential for training deep neural networks. fundamental principle is variance-preserving meaning that the variance of the signal should remain constant as it propagates through multiple layers, whether forward or backward (LeCun ctt al. 2002 Glorot & Bengio 2010: He et al. 2015). This pariance-preservie principle stabilizes activations and gradients.
However, the KAN paper's initialization strategy violates this principle, Assuming an input æ N/ o3)) and using a zero-order spline, the output variance becomes Var [o[] = 0.01 + 1.064o which differs from Vor[x), Higher-order splines may exacerbate this variance instability. Therefore, the default initialization contradicts the essential variance- preserin- principle.
4 KOLMOGOROV
TRANSFORMER
As discussed earlier, the standard KAN faces three major challenges that limit its use in large, deep neural networks. In this section, we refine its design to better suit modern transformers, allowing us to replace MLP layers with KANs.
4.1 OVERALL ARCHITECTURE
Just as its name imply, Kolmogorov-Amold Transformer (KAT) replaces the MLPs in vision trans- former (Dosovitskiy al. 202T) with KAN layers.
Specifically, for 2D image E #x we first flatten it into a 1D sequence, apply patch embedding and positional encoding, and then pass it through series of KAT layers. At layer the following operations are performed:
MSA(LN( +x--1. E= L=
(5)
full derivation in Appendix D
4

---

Published as a conference paper at ICLR 2025
(0) (LP(LN(xx))) ()) + +x00 [Transformer] (6)
x. KAN(LN(X0 (9)) + xx0 [KT]]
(7)
where X stands for the output feature sequence at the layer. MSA and LN stand for the multi-head self-attention and layer norm. As illustrated, we replace all two-layer MLPs with two-layer KANs while keeping the attention layers unchanged. Although similar efforts have been made in specific domains (Chen al. 2024b a), simple replacement not enough achieve scalability in large models.
Most importantly, here, introduce special kind Group-Rational KAN. We rational functions as the base function for KAN (Section 4.2) and share parameters between a group of edges (Section 4.3). We also specify the weight initialization scheme to ensure stable training (Section 4.4). Together, these enhancements make KAT more scalable and improve performance.
4.2 RATIONAL BASE FUNCTIONS
In our method, we use the rational function (Boullé et al. 2020 Telgarsky 2017 Leung Haykin
1993: Aghaei 2024) as the base function for the KAN layer, instead of the B-spline.
Specifically, we parameterize the function ((z) on each edge as rational over polynomials P(x). ((x) of order n. n.
<(x) = =F()) = w P(x)
((x) = w ao + 01 + am x
(8)
b0 + 61 + ..... b,
an and bm are coefficient of the rational function and is the scaling factor. This function is said to have degree m/n. We hope to learn those azs by and w through end-to-end backpropagation.
To avoid instability caused by poles, where (zz) - 0 and d(x) too, employ Safe Padé Activation Unit (PAU) (Molina et al. 2020) as our basis, which is a modified form of the standard rational function
4g + a1 + + am
F(x)
(9)
1 + bi + bn
Implement Rational Function on GPU. With the rational function, core contribution in this paper is to implement it efficiently on the GPU. Rather than using pytorch with automatic differentiation. we implement fully with CUDA (Nickolls al. 2008).
Similar to Molina et al. 2020), we compute the explicit gradients of 6F SF and
5F
The full
Sa. 8b,
37
expression is shown in Appendix F
To optimize the evaluation of polynomials, employ Horner's method (Horner 1815). which
reformulates a polynomial in a nested form to reduce the computation:
do + a1T + ..... + am ao + x(a: + sa02 + z(...)))
(10)
This allows the evaluation of polynomial of degree with only n multiplications and n additions. By default, we use m = 5 and n 4.
Through this efficient CUDA implementation, of the base function. As shown in Table [ with a scalar input, the rational function with the Horner method is much cheaper than the B-spline used in the KAN paper.
largely reduce the computation for each evaluation
4.3 GROUP KAN
Instead of learning unique base function for each input-output pair, we can share their parameters within a group of edges. reduces the number of parameters and computation. This kind of parameter sharing (LeCun et al.) 1995; 1989) and group-wise computation (Vaswani et al., 2017: Wu & He 2018) have been key techniques in neural network design.
Specifically, we divide the input channels din into groups, sharing parameters among din/ g input channels within each group. Figure 2 Iillustrates the distinctions between the original KAN, our Group KAN, and standard MLP. Unlike MLPs, which employ non-learnable activations, KAN assigns unique function to each onput-output pair. Group KAN reduces the number of parameters by sharing these functions among group of edges.
5

---

Published as a conference paper at ICLR 2025
Ourgut Charnets
Name
FL.Pss
PLLa
2ct
Inpur Chanees
$2.1
$2.3
4x11
B-Spline (G=3, K=3)
204
④
Rational (m=5, n=4)
28
Rational (m=5, n=4) Homer
21
041
194,2
Pa5
(94,
4Pxz
Vondii KAN
(Dur,
Pre-dct
Table 1: One sample FLOPs comparison of different non-linear functions at each edge. Horner's s method with the Rational function reduces FLOPs by approximately 9.3x compared the B-Spline.
Figure Comparing our Group KAN with vanilla KAN and MLPs. While KAN unique function on each input-output pair, Group KAN shares these functions with groups of edges.
Name
No. Params
FLOPs
MEP
din dout doue 3) dout dun X dout + doui + (m - g)
Fune FLOPs dout + (din dous) FLOPs
KAN
den dout
(dan dout) 19 (G 2G 2.5K (2m + 2n + 3) don + 2 x (din x dout)
GR KAN (Ours)
Table Comparison of parameter number and computation among different models. Func FLOPs refers the FLOPs non-linear activation. KAN, K is order and G is grid number. In GR-KAN, m n are the polynomial orders, and is the number of groups. GR-KAN has a parameter size comparable to that of MLP, while the original KAN has O(G - K') times more parameters and O(GK) times more FLOPs.
Group-Rational KAN. We combine the rational function of Section 4.2 with group-wise parameters to implement our Group-Rational KAN (GR-KAN). In practice, share the parameter for the rational function F for each group; however, each edge retains unique scalar w.
Suppose is the index of the input channel. With groups, each group contains dg d channels, where [t/dg] is the group index. The operation of GR-KAN on input vector x can be expressed as
GR-KAN(x) + ox = Ete (1/1a(/
Diz Wi.d (a)),
(11)
With a simple rewrite, this can be expressed in matrix form as the product of a weight matrix W e Rd Xdu and a input-wise wise rational function F
W1,1
Wi,da
GR-KAN(X) WF(x)=
Fiduo/dg (F
Wdwan
2 O'd.a.de dio
(12)
As such, implement this GR-KAN layer as group-wise rational function F followed by linear layer
GR GR-KAN(X) = wineerr ritrear(grou.rratiion
(13)
In this form, sharing parameters across each input channel allows direct application of the rational function to the input vector, equivalently applying it across each grouped edge. In this way, GR-KAN functions as a specialized MIP, with 1) learnable non-linear functions, 2) activation preceding the linear layer, and 3) unique activation functions tailored for each group of edges.
experiments, we notice that for rational function, we share the numerator coefficient am among groups and use different b, for each group. It gets better performance.
Parameter and Computation Savings. The original KAN requires din dout unique activation functions. Through our grouping strategy, only g unique functions are needed, reducing the parameter count to constant overhead compared to standard MLP.
Except the saving on parameter number, this grouping also reduces computational demands. Each input channel computes the activation function & once, shared across all corresponding output channels. In contrast, the original KAN requires that each output channel j to independently compute Φi.j. This results in significant computational savings. The comparison the number of parameters and computation is listed in Table

---

Published as a conference paper at ICLR 2025
Name
a
B[F())
Varla
6.0
Identity
1
ReLU
2
GELU
2.3568
Swish/SiLU
2.8178
GEGLU
0.7112
100L
SwishGLU
0.8434
Figure 3:
Example of fitted functions with rational form. We also show mean square error (MSE) for the fitting.
Table Expected values of F(x)² for various functions.
4.4 VARIANCE- PRISSRVEPRESERVING INITIALIZATION
In this section, we aim to initialize the values for amz bn and w in Group-Rational KAN to ensure variance-press prraance-rreserving behavior across the network. At its core, we prevent the growth or reduction of activation magnitudes throughout the layers, thereby maintaining stability.
We revisit the analysis from (He et al. 2013) and adapt it to KANs. For a GR-KAN layer, the computation for each output yy is given by = =Ee ((x;) = Ein (Wa. F(x))) + +by. We assume that all xj are i.dd (Glorot & Bengio 2010) and uniformly distributed. Here, Wa follows normal distribution N N(0, ) and by initialized to The variance of y/ then be described as:
V Var [y] = di, V VarlwFFFee
(14)
Varlyl = d V or [F
(15)
where æ, y, and w represent the random variables of each element in x4, B/, and way respectively. When layers are stacked, aim for the variance of the input-output activations to remain consistent, expressed as:
Var [r] d,,, (ar(u
(16)
Since F(x:) is the rational function containing coefficients o. and bas the initialization of and these coefficients are interdependent- form F(x) influences the appropriate initialization of w. The crucial step is to calculate E[F(z)*]
Var[æ] and adjust to maintain consistent activation scaling.
For our rational function defined Equation 9 computing (x))) involves evaluating:
B[F(x) = [ (2(2)((2)d==
ao + 01 + ... + am
1+ bb + b ) (x)dx
(17)
where f(x) is the density function of x. Unlike activation functions such as RELU, for which E[F(x)*] 1 &aall Yar[z)), computing [x)) for the rational function is challenging due to the lack of
closed-form solution.
Initialize a, first, then initialize w. To make the process manageable, Instead of sampling w, and b jointly, we proceed sequentially. Initially, we determine and such that F fits established activations like ReLU, GELU, and Swish. Figure 3 illluttraees the fitted functions.
numerically, assuming (0, The calculated gains, a, are documented in Table We use the gain value to initialize w from NV a.
Once a and b are set, we estimate the gain a [[F(=)*)
Varlz
also transfer weights from a pre-trained ViT to our KAT model. This transfer is straightforward for most layers, as KAT can replicate the aicro-architecture of ViT, except for the KAN layer.
Initialize KAT from VIT. In addition random weight initialization,
"This assumption is justified the inputs to the KAN layer are normalized using layer normalization as in Equation The LN layers are initialized to have hias and scaling factor of
7

---

Published as a conference paper at ICLR 2025
MLP
For the GR-KAN, weight transfer is still feasi- ble, as shown in Figure Because the GR-KAN also includes linear layer, the weights of the linear layer from the MLP in the trained VIT.
Idenity
directly load
GR-KAN
Figure 4: One-to-one weight mapping between trained MLP VIT and GR KAN KAT.
For rational layers in GR-KAN, the first one is initialized to behave like an identity function, while the second layer is set to approximate the non-linear function used in the original MLP. This approach allows all the weights of the GR-KAN layer to be cloned from ViT model, ensuring compatibility and efficient initialization.
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
We modify the original VIT (Dosovitskiy et al. (021) architecture by substituting its MLP layers with GR-KAN layers. By default, these KAN layers employ rational function with parameters and 4, and are organized into groups of 8 (g 8). Each transformer block contains KAN layers. The first GR- ER-KAN layer's ' am and bn are initialized to fit the identity function, while the second initialized to mimic the Swish function (Ramachandran et al., 2017). The attention layers are initialized with Mimetic Initialization (Trockman & Kolter (023). ). The remainder of the architecture remains unchanged. We intentionally do use hierarchical architectures (Yu al. 2022) for simplicity. In the main paper, due to the page limit, we focus on vision tasks. But we also extend the architecture for tabular and graph data, detailed in Appendix
5.2 IMAGE RECOGNITION
Experiment Setup. We do experiments on ImageNet-IK [59] image classification benchmark. ImageNet- K is one of the most widely-used datasets in computer vision which contains about 1.3M images of 1K classes on training set, and 50K images on validation set.
We mainly follow the for 300 epochs at 224 resolution. The patch size set to 16. Data augmentation and regularization techniques include RandAugment (Cubuk al. 2020), Mixup (Zhang et al.) (018)), CutMix (Yun) et al. 2019). Random Erasing (Zhong et al. 2020). weight decay, Label Smoothing (Szegedy et al. 2016 and Stochastic Depth (Huang et al. 2016). We adopt AdamW (Loshchilov & Hutter 2019) optimizer with batch size of 1024.
of DeiT (Tourron et al. (021). Specifically, models are trained
We compare with ViT (Dosovitskiy et al. I) and DeiT Touvron et al., 2021), as we share the same archi- tecture, except for the channel mixer. We also report the results of VIT + KAN (Chen al. 2024bb, that simply replacing MLP with standard KAN.
Figure 5: Comparative Analysis of Model Performance and Com- putational Efficiency on ImageNet-IK. measure the FLOPs under 2212 using fvcore package. indicates that the model initialized using a pre-trained ViT model, otherwise trained from scratch.
Model
Channel Mixer
#Parum.
FLOPs
IN-IK Top-1
VIT-TE166 DefFT
MLP
5.7M
1.88
72.7
Results. Our experimental results
MLP
72.2
VIT + KAN
KAN
12.8M
L.7G
64.9
demonstrate that the KAT models con- sistently outperform their counterparts on the IN-Ik 1kk dataset, as shown in Ta-
KAT-T
KAN
5.7M
13G
74.6
VIT:S/16
4.25G
78.8
ble 5 Firstly, the integration of GR- KAN in transformer demonstrates su- perior performance over traditional MLP-based mixers. For instance, the
DeiT-S
VITS + KAN
KAN
MI.P
50.4M
4.2
7.05G
79.8
62.9
KAT'S
KATSS
KAN
2M
4.35G
82.0
V/T
MLP
86.6M
16.87G
79.1
DeiT-B
ML.P
86.6M
16.87G
81.8
KAT-S achieves an accuracy of 81 81.2%, outperforming the DeiT-S model by 2.4%. This improvement underscores
KANBBKKAN KAN
KAN
8M
KAT-B KAT-B"
KAN
86.6M
17.06G
82.3
KAN
86.6M
17.06G
82.8
the potential of KAN mixers enhance model efficacy when properly integrated.
8

---

Published as a conference paper at ICLR 2025
Secondly, the vanilla KAN layer faces scalability issues. VIT-T/S KAN only achieved an accuracy of around 63%, even with much higher computational cost. ViT-B + KAN fails to converge. resulting in NAN error. We addressed these scaling challenges detailed in Section 3 enabling our KAT models to scale successfully. These findings highlight the advantage of KAT models in balancing computational efficiency with improved performance.
5.3 OBJECT DETECTION AND INSTANCE SEGMENTATION
Experimental Setup. We evaluate our approach on the MS-COC02017 (Lin al. 2014) dataset, a standard benchmark for object detection and instance segmentation. In our setup, the KAT is employed the backbone within a ViTDet-based (Li al. 2022 Mask R-CNN (He et al. 2017) model, initialized with weights pre-trained on ImageNet. We followed the standard 3x training schedule, which consists of 36 epochs. The training images were resized to 800 x 1333 pixels. The AdamW W optimizer (Loshchilov & Hutter] 2019) was used with a learning rate of 0.0001 and a total batch size of Our implementation was based on the PyTorch and MMDetection (Chen al., 2019) libraries, and use FP16 precision to reduce training costs. The experiments were carried out on NVIDIA H100 GPUs.
Table compares the performance of different backbones. KAT consistently outperformed other models, particularly in object detection, where it achieved a 3.0 Aph gain on the S-sized model and 1.4 Aphox gain the L-sized model compared to Vit The improvements were most pronounced in smaller models, where computational cost increased by only GFLOPs. This shows that KAT offers better accuracy with minimal overhead.
Results.
Table 4: Performance of Mask-RCNN with different backbones xx schedule.
Backbone
AParam.
FLOPS
Aphot
APho
APhe
Aprrac
AP
AP
PVT-Small
44.1M
43.0
65.3
46.9
39.9
62.5
42.8
Swin-T
48M
267G
46.0
68.1
50.3
41.6
65.
44.9
ConvNeXLT
48M
262G
46.2
67.9
50.8
41.7
65.0
44.9
VITS
43. 8M
423G
44.0
66.9
47.8
39.9
63.4
42.2
ViTDet-S
KATDet-S
44.50 44.5M
424G
47.5
69.0
51.2
41.5
65.7
44.0
VIT-B
113.6M
767G
45.8
68.2
50.1
41.3
65,1
44.4
ViTDet-B
11366M
767G
46.3
68.6
50.5
41.6
65.3
44.5
KATDeE-B
113.7M
770G
47.7
69.1
65.9
44.3
5.4 SEMANTIC SEGMENTATION
Experiment Setup. We evaluated our KAT model the ADE20K dataset (Zhou et al. 2017). This dataset comprises 1,50 semantic categories with 20,000 images in the training set and 2,000 in the validation set. For experiments, we utilized KAT as the backbone for the UperNet framework (Xiao et al. 2018), initializing it with ImageNet pre-trained weights. The training was conducted using the AdamW optimizer (Loshchilov & Hutter, 2019)) with learning rate of 0.0001 and a batch size of 16, across 160,000 iterations. Our implementation was carried out using the PyTorch and mmsegmentation libraries, and the experiments were performed on two NVIDIA H100 GPUs. For comparison, we evaluated UperNet with other backbones, including DeiT, Swin Transformer, and ConvNeXt.
Results. Table summarizes the segmentation results. Overall, KAT demonstrates competitive improvement over plain ViT-based architectures, achieving 2.4% improvement over DeiT-S and 0.2% improvement over DeîT-B. This performance boost comes with slight increase in com- putational cost, reflected in the higher FLOPs. Similar to the detection results, KAT shows more significant gains in smaller models. However, it still falls short compared to models with hierarchical architectures, such as ConvNeXt, which benefit from more efficient structural design.
5.5 ABLATION STUDY AND ANALYSTS
GR-KAN v.s. Activation Function. As GR-KAN can be considered as a special kind of MLP with group rational function, we do ablation study to consider different types of activation for

---

Published as a conference paper at ICLR 2025
Table 5: Performance of Semantic segmentation with Images are
Table Ablation on activation function, with ViT- Ti/16 Variant.
UperNet on ADE20K validation cropped to 512 x 512 for training. The MACs are measured with input size of 512 2048.
Name
Learnable?
IN-Ik Top-1
Backhone
iParam.
FLOPs
mloU
GELU (Default)
No
72.7
Swin-T
ConoNeXI-T
60M
939G
46.7
ReLU
No
72.8
DeïT-S
SILU?
No
69.8
KAT-S
57M
12196
46.1
PReLU
Yes
73.2
Swin-B
121M
1188G
49.5
ConvNcXi-B
122M
1170G
49.6
PAU
Yes
73.6
Defl-B
KAT-T
Yes
74.6
KAT-B
142M
2011G
47.4
MLP and compare with GR-KAN, Superficially, we replace the activation function in MLP in ViT-Ti/16 to different kinds, including GELU (Hendrycks & Gimpel, (016), ReLU (Fukushima 1969). SiLU (Elfwing al., 2018), PReLU (He et 2015) and PAU (Molina et al. 20200. and comparing them with KAT.
Table summarizes the top-1 accuracy on the ImageNet- Ik dataset for each activation function with ViT-17/16. The results indicate that traditional activation functions like ReLU and GELU perform similarly. Learnable activations like PReLU and PAU show an improvement. Notably, Our KAT-T achieves the highest accuracy at 74.6%, outperforming GELU by 1.9%. This suggests that GR-KAN, as used in KAT-T, can significantly enhance the expressiveness MLPs vision transformers.
In addition to accuracy, we analyzed the computational cost of different activations by measuring throughput and peak memory usage on an NVIDIA A5000 GPU (Table 77. All activations had similar peak memory usage. However, KAT-T showed slightly lower throughput compared to baseline activations like ReLU, GELU, and SiLU. This suggests a trade-off between accuracy improvement and computational efficiency due to the increased complexity of rational function computations.
Table 8: Ablation rational function initialization, with KAT-T.
Table 7: Throughput and Peak memory for different activa- tions on A5000 GPU. Input size is fixed to [64, 1000, 512].
Rational Init.
Rational 2 Init.
IN-Ik Top-1
Activation
ReLU
GeL.U
SiLU
PReLU
Ours
Random
Random
53.2
Identity
Identity
69.7
Throughput (butchis)
2643
2668
2313
Swish
Swish
74.4
Peak Memory (M)
1380
1380
1380
1380
1380
Identity
GeLU
74.5
Identity
Swish
74.6
Initialization. We evaluate the KAT-T model using different initializations when training from
scratch. 'Random' refers to initializing all am, bn values using Xavier initialization, without our
pariance-preserving init. As shown in Table 8] the "Identity Swish" initialization achieves the best performance, which we have adopted as our default setting. 'Random' initialization results in significantly lower accuracy.
6 CONCLUSION
In this work, we introduced the Kolmogorov-Arnod Transformer (KAT), novel architecture that successfully integrates Kolmogorov-Amool Networks (KANs) into transformers, addressing key challenges associated with large-scale training scenarios. Our proposed Group-Rational KAN (GR-KAN) variant, with its rational activation functions, group-based parameter sharing, and variance- preserving initialization, demonstrated significant improvements in computational efficiency and scalability. Through extensive experiments vision tasks, including image recognition, object detection, and semantic segmentation, KAT outperformed traditional MLP-based transformers, achieving superior accuracy Net while maintaining comparable computational demands.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
This project is supported by the Ministry of Education, Singapore, under its Academic Research Fund Tier 2 (Award Number: MOE-T2EP20122-0000 work involved in this research work is partially supported by NUS IT's Research Computing group using grant numbers NUSREC-HPC-00001 We thank Weihao Yu, Qiuhong Shen and Runpeng Yu for valuable discussions.
We would like to acknowledge that computational
REFERENCES
Emmanuel Abbe. Community detection and stochastic block models: recent developments. J. Mach. Learn. Res., 18:177:1-177:86 2017. URL WtR.hhtps:///pp/ htt.:pp://ppp.ssaant.ssssss www.steeasssooooe. ar.
CorpusTD: 9134861
Alireza Afzal Aghaci. rkan: Rational kolmogorov- networks. arXiv preprint arXiv:2406.14495, 2024.
K. Bache and M. Lichman. UCI machine learning repository. 2013. URL http://archive ics uci edu/ml
George A Baker and John L Gammel. The padé approximant. Journal of Mathematical Analysis and Applications, 2(1):21-30, 1961.
Ronen Basri, Meirav Galun, Amnon Geifman, David Jacobs, Yoni Kasten, and Shira Kritchman. Frequency bias in neural networks for input of non-uniform density. In International Conference on Machine Learning, pp. 685-694. PMLR, 2020.
Alexander Dylan Bodner, Antonio Santiago Tepsich, Jack Natan Spolski, and Santiago Pourteau. Convolutional kolmogorov-aroold networks. arXiv preprint arXiv:2406. 13155, 2024.
de Boor. Subroutine package for calculating with b-splines, 1971.
Nicolas Boullé, Yuji Nakatsukasa, and Alex Townsend. Rational neural networks. Advances in neural information processing systems, 33:14243-1444 14253, 2020.
Zavarch Bozorgasl and Hao Chen. Wav-kan: Wavelet kolmogorov-armold networks. arXiv preprint arXiv:2405. 12832, 2024a.
Zavarch Bozorgasl and Hao Chen. Wav-kan: Wavelet kolmogorov- networks, 2024b.
Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.0715: 2019.
Yifei Chen, Zhu Zhu, Shenghao Zhu, Linwei Qiu, Binfeng Zou, Fan Jia, Yunpeng Zhu, Chenyan Zhang, Zhaojie Fang. Feiwei Qin, et al. Sckansformer: Fine-grained classification bone marrow cells via kansformer backbone and hierarchical attention mechanisms. arXiv preprint arXiv:2406.09931, 2024a.
Ziwen Chen, Gundavarapu, and DU Vision-kan: Exploring the possibility of kan replacing mlp in vision transformer. https: //tthhh.tttttt//// wttp:///hhhwwwwwwwhcoooioee wwwpsshhhiiiiiiiiiioisoo /Viision KAN git 2024b.
Minjong Cheon. Demonstrating the efficacy of kolmogorov-arood networks in vision tasks, arXiv preprint arXiv:2406. 1XX1:2446..1416, 2024a.
Minjong Cheon. Kolmogorov-arnold network for satellite image classification in remote sensing. arXiv preprint 00XXxx:2406000000 2024b.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition workshops, pp. 702-703, 2020.
11

---

Published as a conference paper at ICLR 2025
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICIR, 2021.
Vijay Prakash Dwivedi and Xavier Bresson. A generalization of transformer networks to graphs. ArXiv, abs/2012.09699, 2020. URL http::t/pppp.....ee http:/ppp..mmmaiiicccoooo....... wtw.:c/oooaaarroor/ooooosusto 229298019
Vijay Prakash Dwivedi, Chaitanya K. Joshi, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Benchmarking graph neural networks. ArXiv, abs/2003. 000982, 2023. URL https://api
wtt. ID: 211677851
Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-weighter linear units for neural network function approximation in reinforcement learning. Neural networks, 107:3-11, 2018.
Kunihiko Fukushima. Visual feature extraction by a multilayered network of analog threshold elements. IEEE Transactions on Systems Science and Cybernetics, 5(4):322-333, 1969.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
William J Gordon and Richard F Riesenfeld. B-spline curves and surfaces. In Computer aided geometric design, pp. 95-126. Elsevier, 1974.
Isabelle M Guyon, Lisheng Sun-Hosoya, Mare Boullé, Hugo Jair Escalante, Sergio Escalera, Zhengy- ing Liu, Damir Jajetic, Bisakha Ray, Mehreen Saeed, Michèle Sebag, Alexander R. Statnikov, Wei-Wei Tu, and Evelyne Viegas. Analysis of the automl challenge series 2015-2018. In Automated Machine Learning, 2019. URL https://aag httras//pp...emmtticccooooa...r org/CorpusID:
semanticscholatt
70088648
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Robert Hecht-Niclsen. Kolmogorov's 's mapping neural network existence theorem. In Proceedings of the international conference Neural Networks, volume 3, pp. 11-14. IEEE press New York, NY, USA, 1987.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv: fttv::/60..8880 08415, 2016.
arXiv preprint
WG Hornrr. A new method of solving numerical equations of all orders, by continuous approximation. In Absiracts of the Papers Printed in the Philosophical Transactions of the Royal Society of London, volume 2, pp. 117-117.
1815.
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks, 2(5): -666 1989.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pp. 646-661. Springer, 2016.
Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar S. Karnin. Tabrransformer: Tabular data modeling using contextual embeddings. CoRR, abs/2012.06678, 2020. URL https
//arxiv. www.o///bbb.... abs// /t/p/012222222 2012.001220 06678
John J. Irwin, T. Sterling, Michael M. Mysinger, Erin S. Bolstad, and Ryan G. Coleman. Zine: A free tool to discover chemistry for biology. Journal of Chemical Information and Modeling, 52:1757 1768, 2012. URL htt..ttts:/// //api. semant nt wor.cccae.......eeooorrooooaeeae 9759396
12

---

Published as a conference paper at ICLR 2025
Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541-551, 1989.
Yann LeCun, Yoshua Bengio, et al. Convolutional networks for images, speech, and time series. The handbook of bruin theory and neural networks, 33))(10):1995, 1995.
Yann LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Robert Müller. Efficient backprop. In Neural networks: Tricks of the trade, pp. 9-50. Springer, 2002.
Henry Leung and Simon Haykin. Rational function neural network. Neural Computation, 5(6): 928-938 188-938,11 1993.
Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In European conference on computer vision, pp. 280-296. Springer, 2022.
Ziyao Li. Kolmogorov-arnold networks are radial basis function networks. ArXiv, abs/2405.06721,, 2024.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pp. 10012-10022, 2021.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 11976-1198, 2022.
Ziming Liu, Pingchuan Ma, Yixuan Wang, Wojeiech Matusik, and Max Tegmark. Kan 2.0: Kolmogorov- networks meet science, arXiv preprint arXiv:2408. 10205, 2024a.
Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruehle, James Halverson, Marin Soljačié, Thomas Y Hou, and Max Tegmark. Kan: Kolmogorov-aroold networks. arXiv preprint arXiv:2404.19756, 2024b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https //axxiv.orggeaaea http://worrg/aa///1........ 11111.111111111111
Alejandro Molina, Patrick Schramowski, and Kristian Kersting. Padé activation units: End-to-end learning of flexible activation functions in deep networks. In International Conference on Learning Representations, 2020. URL https:/t: /ptp://ieeeeeeeeeeeeeeee wwtp://r.t..ffeee.e
John Nickolls, Ian Buck, Michael Garland, and Kevin Skadron. Scalable parallel programming with cuda: Is cuda the parallel programming model that application developers have been waiting for? Queue, 6(2):40-53, 2008
Gist Noesis. Fourierkan, 2024.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International conference on machine learning, pp. 5301-5310. PMIR, 2019.
Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. arXiv preprint arXiv:1710.05941, 2017.
Basri Ronen, David Jacobs, Yoni Kasten, and Shira Kritchman. The convergence rate of neural networks for learned functions of different frequencies. Advances in Neural Information Processing Systems, 32, 2019.
13

---

Published as a conference paper at ICLR 2025
Daniel Ruijters and Philippe Thévenaz. Gpu prefilter for accurate cubic b-spline interpolation. The Computer Journal, 55(1):15-20) 2012.
Daniel Ruijters, Bart M ter Haar Romeny, and Paul Suetens. Efficient gpu-based texture interpolation using uniform b-splines. Journal of Graphics Tools, 13(4);61-69 2008.
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXivv20002.55202. 2020.
Christian Sigg and Markus Hadwiger. Fast third-order texture filtering. GPU gems, 2:313-329, 2005.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 281 18-6282,, 2016.
Matus Telgarsky. Neural networks and rational functions. International Conference on Machine Learning, pp. 3387-3393. PMLR, 2017.
Ily Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mlp architecture for vision. Advances in neural information processing systems, 34:24261- 24272, 2021.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp. 10347-10357. PMLR, 2021.
Asher Trockman and J Zico Kolter. Mimetic initialization of self-attention layers, In International Conference on Machine Learning, pp. 34456-34468. PMLR, 2023.
Michael Unser, Akram Aldroubi, and Murray Eden. B-spline signal processing. i. theory. IEEE fransactions on signal processing, 41 (22)8221--33, 1993.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference Neural Information Processing Systems 20/7, December 4-9, 2017, Long Beach, CA, USA, pp. 2017. URL httpstttt///ooooeetetttttte neureriin..nnnnuneeeeeeee. wwtppppp///001111 2007 hash/ (tteee/223aideeeeeeodoboeod
Abstract html
Joseph Leonard Walsh. Interpolation and approximation by rational functions in the complex domain, volume 20, American Mathematical Soc., 1935.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp. 3-19, 2018
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European conference computer vision (ECCV), pp- 418-434, 2018.
Xingyi Yang, Daquan Zhou, Songhua Liu, Jingwen Ye, and Xinchao Wang. Deep model reassembly. Advances in neural information processing systems, 11-25739-257533 2022.
Runpeng Yu, Weihao Yu, and Xinchao Wang. Kan or mlp: A fairer comparison. arXiv preprint arXiv:2407.16674 2024.
Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang. Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 10819 10819-10829, 2022.
Weihao Yu, Chenyang Si, Pan Zhou, Mi Luo, Yichen Zhou, Jiashi Feng. Shuicheng Yan, and Xinchao Wang. Metaformer baselines for vision. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
14

---

Published as a conference paper at ICLR 2025
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference computer vision, pp. 6023-6032, 2019.
Hongyi Zhang. Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net 2018. URL
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmenta- tion. Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 13001-13008, 2020.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. Proceedings of the IEEE conference on computer vision and patter recognition, pp. 633 641, 2017.
A APPLICATION ON DOMAIN BEYOND VISION
In addition the vision task we presented in the paper, in this appendix, we also provide experiments on applying KAT to tabular data and graph data.
Tabular Experiment. We choose Trbnrassforme (Huang al., 2020 as our baseline for tabular classification. We test 15 publicly available binary classification datasets from UCI dataset (Bache & Lichman 2013). AutoMI. challenge (Guyon et al. 2019) and Kaggle. We keep all experiment setup in the paper, but only modify architcture by replacing MLPs to GR-KANs.
The results are presented in Table 9 Our KAT, when integrated with TabTransformer, improves the AUC score by an average of 0.9. This demonstrates the potential of KAT in enhancing the performance of tabular data models.
Table Comparison between KAT and the baseline TabTransformers datasets, The evaluation metric AUC in percentage.
Dataset
TabTransformer
TabTTransforer=KAAT
albert
75.7
76.9(+1.2)
90.6
91.1(+0.5)
dota2games
63.3
64.1(+0.8)
hedr_main
75.1
76.2 (+1.1)
adult
73.7
75.3 (+1.6)
mankenigg
93.4
93.8 (+0.4)
blastchar
83.5
83.1(-0.4)
insurance
74.4
75.9(+1.5)
jasmine
85.3
85.6(+0.3)
online_shoee shoppers
92.7
93.2(+0.5)
philippine
83,4
84.3 (+0.9)
_io
91.8
92.6 (+0.8)
seismicbumps
75.1
77.8(+2.7)
shrutime
85,6
86.4(+0.8)
spambase
98.5
98.6(+0.1)
Average
82.8
83.7 (+0.9)
Graph Experiments. We apply our methods on top of Graph Transformer (Dwivedi & Bresson 2020) and see if it improves the performance. We replicate its experiments ZINC (Irwin et al.
2012) for graph regression and PATTERN (Abbe) 2017) and CLUSTER (Dwivedi et al. 2023) for node classification. We keep all setup the same its original paper and only modify the MLP layers. The results are shown in Table TO KAT, again, demonstrates improved performance on the graph datasets.
15

---

Published as a conference paper at ICLR 2025
Table 10: Comparison of graph dataset against the Graph Transformer baseline with 500k model parameters.
Dwivedi & Bresson 2020
Model
ZINC
CLUSTER
PATTERN
Graph Graph-Transtone Transformer + KAT
Graph-Transforme Transformer
0.226 0.014
73.169 = 0.622 74.089 = 0.600
84.808 = 0.068 85.293 = 0.065
0.211 + 0.013
B GR-KAN OUT OF TRANSFORMER
Our proposed GR-KAN can be applied outside the scope of transformer. In this section we apply on two tasks. One on simple function fitting and the other solving partial differential equation.
Function fitting. In our function fitting experiments, we selected functions based on examples presented in the KAN paper. We adopted a simple neural network architecture consisting of three layers: an input layer with 2 neurons, hidden layer with 5 neurons, and an output layer with neuron, denoted as a [2
- 1] structure. The models were trained for 1000 epochs using the Adam optimizer with learning rate of 0.001.
The performance of each method evaluated using the mean squared error (MSE), where lower values indicate better function approximation. We compare the results for three models: a standard Multi-Layer Perceptron (MIP), the KAN model, and our proposed GR-KAN model. The MSE values for each function are presented in Table
Table 11: Mean Squared Error (MSE) for Function Fitting. Lower values indicate better performance.
Mothod
exp(sin(x" y """}
exp(sin(1rz) +
exp(Ja + x5)
.2y
(.r M) + az
MLP
0.4307
180.3786
43.4192
80,8309
0.0766
0,0303
194.7122
GR KAN
0.0034
19.3789
20.04033
90.5357
0.0016
0,0221
The results clearly show that our GR-KA KR- model achieves significantly lower MSE values in most cases compared to both the MLP and KAN models. This demonstrates the improved ability of GR-KAN to fit complex functions effectively, highlighting its robustness and accuracy in regression tasks.
PDE Solving. For solving the PDE, we focus one-dimensional damped harmonic oscillator. This system is governed by the equation:
deu u
du dl.
m
+
+ ku 0,
(18)
di
where m represents the mass, set in our experiments. The term μ denotes the damping coefficient, while & corresponds to the stiffness constant. We define the initial conditions as u(0) = 1 and u' (0) 0, providing a complete setup for the problem.
The exact analytical solution of this oscillator can be expressed as:
u(t) = dt (Acos(wt + d)),
where d = p/2wo vk v = arctan(-d/w) are derived from the parameters of the equation and the initial conditions. To approximate this solution, we three methods: standard Multi-Layer Perceptron (MLP), the KAN model, and our proposed GR-KAN model.
Our experimental results, presented in Table 12 compare the performance of different methods based on the L2 error and training loss between predicted and true solutions over a specified time interval.
The KAN model achieves the best performance but requires more parameters and significantly longer training time 20 minutes). In contrast, the GR-KAN model trains much faster minutes) while outperforming the pure MI.P in solving this type of PDE problem.
16

---

Published as a conference paper at ICLR 2025
Table 12: Comparison of 1.2 Loss and Predictions for MI.P and KAN
Type
L2 error (last step)
Prediction
Loss Curve
10g 10
MLP (GELU)
2.0216 10-4
05
DAM 1300
UOW 201
rxtiy Jodivs
GR-KAN
6,3909 10
-34
KAN
1.6125 x 10
W0
50
Ammm
-10
MLP (GELU)
1.1805 10-1
kmmm
-12
0.20
GR-KAN
5.2515 x 10
2.20
-00
KAN
3.7762 10-2
17

---

Published as a conference paper at ICLR 2025
C DETAILED ABLATION STUDY
We conducted detailed ablation study to evaluate the contributions of three key components and parameter grouping in KAN. These experiments were performed using the KAT-Tiny model on ImageNet, training for 300 epochs. Specifically, we analyzed:
Base Function:
Replacing the rational base function with the B-spline used in KAN
Group-wise Computation: Replacing group-wise weight sharing with distinct parameters for each channel.
Initialization: Replacing the proposed initialization with PyTorch's default random initial- ization.
The results ablation study are summarized in Table 13 including Top-1 accuracy, training time, and MACs
Ablation 1: Base Function We replaced the rational base function with B-splines, implemented using the De Boor-Cox algorithm in PyTorch, no CUDA implementation is available.
The results show that the base funetion slightly impacts performance but significantly affects runtime. According to Exp 3 and Exp 6, the rational function provides small performance improvement over B-splines. Although the MACs differ only slightly, the PyTorch-based B-spline implementation is much slower than the CUDA-optimized rational function.
Ablation 2:
Group-wise Computation We replaced group-win wise weight sharing with distinet parameters for each channel.
As shown in Exp 4 and Exp 6, group-wise computation is critical for efficiency. While it results in minor accuracy drop from 74.8% to 74.6%, it reduces training time drastically from 38 hours to 12 hours, making the most significant factor for efficiency improvements.
Ablation 3: Initialization We replaced the proposed initialization with PyTorch's 's default initializa- tion.
As shown in Exp 6 and Exp 5, proper initialization is crucial for achieving fast and reliable convergence and improving performance, particularly for rational functions with higher-order terms. Without proper initialization, terms of different orders are initialized at similar scales, leading to instability. This issuc more severe for rational functions compared to B-splines, as highlighted by the comparison between Exp 2 and Exp 5.
KAN with Parameter Grouping In Exp 2, we explored parameter sharing KAN by applying parameter grouping to the original ViT+KAN architecture. This modification significantly improved the training speed, reducing the time from 43 hours to 20 hours. However, it resulted in a performance drop of 2.7%.
Exp ID
Rational
Group
Initialization
Top-1 (%)
Train Time
MAC
x
X
X
64.9
43h
1.78G
x
62.2
20h
1.15G
3
x
/
73.0
20h
1.15G
4
/
x
74.8
38h
1.76G
53.2
12h
1.13G
74.6
12h
1.13G
Table 13: Ablation study on KAT-Tiny with different components.
D VARIANCE ANALYSIS OF THE KAN LAYERS
In this section, we discuss why the original initialization in KAN fails to maintain activation variance across layers.
18

---

Published as a conference paper at ICLR 2025
Specifically, the B-spline coefficients c₁ initialized NN o²) with 0.1, and w, w,=1 and 6
W/ U
Ado vd..-dout are initialized according to the Xavier initialization (Glorot & Bengio 2010p. The combined output variance of the model can be expressed as:
Varlo Var[o(x)] = Varlu tarlw, [[ssilu((]]+ Var (er[w, sus,ppligg spline(z) = 3E[silu"(1)) tlsplinee (19)
Zero-th Order Case. If we assume the input is normally distributed, x: (1, consider zero-th order spline, the variance of sp1 (n)()) at any point x simply:
E[spline" (x)] = Σ VVarr = - o² Var( o² 0.01
(20)
For the SiLU activation function, although exact variance calculations complex, numerical estimations indicates E[silu"(x)] = 0.355a Combining these, we find Var[ø(z)] = 0.01 1.064o f Var[æ]]
This indicates that, under zero-th order spline, Var[ø(<)] # Var(a].
Higer Order Case. With higher-order splines, the variance instability might increase. This because, increasing the order of a spline leads to excessive smoothing. This smoothing effect reduces the variation in the function values, causing V Varlol [=)] become smaller.
The B-spline is defined recursively as:
t - t; li-p L, Btp- (t) + l+pp11 -
(t)
Bi+1.p-1(t)
li+p+1 li+1
shows that each higher-order basis function B weighted summation of a weighted average of two lower-order basis functions (()) and (1+1+-1(t). as the order p increases, the B-spline becomes wider and smoother.
In the extreme case, as p + oo, the smoothing effect causes the basis functions to become nearly uniform across the domain, i.e., (t) = C, constant. Consequently, as, when -> co, the variance of the output converges to zero, Var[ø(z)] - 0. This excessive smoothing leads to instability in the activation variance, effectively flattening all variations. Regardless of weight initialization, the function becomes unstable for use in neural networks.
Based on all analysis above, the default initialization opposes the essential variancee-preserving principle.
E EXPLICIT GRADIENT CALCULATION
n our CUDA implementation, we compute the gradients 6F and
using the following ap- 5am. 5b 8z
6F
proach:
8F
6F
A(x)
P(x) and
8F 8P(x) 1 8Q(x) P(x:)
ar
(21)
8am
Q(x))
8bn
[A(x)] Q(x)²'
dx δx ((x)) 8x Q²
where A(x) = 21æ ....
..... + bn
6P(m)
6z = + 20z + mam and sQ(x) A(z)
(b: + 2b2x
[A(=]
nb,,," 1)). )
F INITIALIZING am, bn
Given ground-truth function g(-) and parameterized rational function F(:; {am), (br)) H), we run linear least square to determine } {bn). Specifically, we optimize the following function:
N Eg(r.)) F(an (am), {ba)))²
min (Am (.h...
2
We uniformly sample 1000 points x; from the interval [-3,3]. {am}: (ba} are randomly initialized. In practice, solve this using the Bveenbegg-arqquaa algorithm, available in the MINPACK package.
In the end, this easily done by calling scipy. .oy.op wpt.miimizii ze. curve fit.
19

---

Published as a conference paper at ICLR 2025
G WHY USING RATIONAL FUNCTION?
In this section, we explain why we chose rational functions as our base function. The main reasons are efficiency, prior successful use, and strong theoretical properties.
Efficiency Perspective. Evaluating polynomials involves simple operations that are well-suited for parallel computing. This makes rational functions computationally efficient for large-scale models.
Practical Perspective. Rational activations have already been successfully used as activation functions in neural networks (Boullé al. 2020 Molina al. 2020p.
Theoretical Advantage. Rational functions can approximate wider range of functions Lunclions-includingg those with singularities or sharp variations-mort -ooon efficiently and accurately than polynomials (Walsh) 1935 Baker Jr & Gammel] 196T). Since B-splines are essentially sums of local polynomials, rational functions offer a theoretical advantage over B-splines for modeling complex behaviors.
Given these reasons, adopt rational functions the base functions in our KAN layers to enhance the model's expressiveness, stability, and computational efficiency.
H DERIVATION AND CAL.CULATION OF FLOPS
Given the function:
+ a1T + + am x.
F(x)
1 + |b + 2n
H.1 PLAIN COMPUTATION
NUMERATOR
The numerator is polynomial of degree m:
Multiplications:
There are m(m-1) multiplications for computing powers of and m
multiplications for coefficients 0g, giving m(m-1) + m.
2
Additions: There are m additions to sum up the polynomial terms.
DENOMINATOR
The denominator involves the absolute value of polynomial of degree nı:
Multiplications:
multiplications for powers of x and n multiplications for coefficients b; giving a(x-1) + n.
There are m(n-1) 2
Additions: There are n additions for polynomial terms and additional addition after the absolute value operation.
Absolute value operation: 1 absolute value calculation.
Division. There is division operation for the final computation of F(r).
Total FLOPs. The total FLOPs for any m and n are:
Multiplications: m(m-
nn 1) +m+n+1, Additions: m+n++1
Absolute Value: 1,
Division:
2
2
In case m = 5 and n = 4, there are totally 16 multiplications, 10 summations, absolute value and 1 division. In total 28.
H.2 HORNER'S S METHOD
Using homer's method, for polynomial of order m, need summations and multiplications.
20

---

Published as a conference paper at ICLR 2025
Thus, for numerator, we need summations and multiplications. For denominator, need + 1 summations and n multiplications. total, we need m + n + summation, m + n multiplications, absolute value, and division.
In case m = and n = 4, there are a total of FLOPs, comprising 9 multiplications, 10 summations, 1 absolute value, and 1 division.
I HYPER-PARAMETERS FOR KAT MODEL
The hyper-parameter for training KAT model on ImageNet-1k 1k is shown in Table 14
Table Hyper-parameters KAT on ImageNet image classification.
KAT
Tiny
Small
Base
Input resolution
2242
Epochs
300
Batch size
1024
Optimizer
AdamW W
Adam
1 10-8
Adam (31, (32)
(0.9, 0.999)
Learning
10-3
Learning rate decay
Cosine
Gradient clipping
None 5
Warmup epochs
Weight decay
0.05
Rand Augment
9/0.5
Repeated Augmentation
off
Cutmix
1.0
Mixup
0.8
Cutmix-Mixup switch prob
0.5
Random erasing prob
0.25
Label smoothing
0.1
Peak stochastic depth rate Random erasing prob
0.1
0.1
0.4
0.25
EMA decay rate
0.9999
Model Variant. We select the configurations of KAT to be identical with those used in ViT (Dosovit skiy et al. 202T), as summarized in Table T5 All variants use an input patch size of 16 16.
Model
Layers
Hidden Size D
MLP Size
Heads
Params
KAT-Tiny
12
192
768
5.7M
KAT-Small
12
384
1536
6
22.1M
KAT-Base
12
768
3072
12
86.6M
Table 15: Details of KAT model variants.
I.1 ABLATION AND ANALYSIS
This section outlines the key hyperparameter settings used for KAT, determined through ablation studies and preliminary experiments.
Number of Groups. We conducted an ablation study with the KAT-Tiny model to determine the optimal number of groups. The results showed that accuracy improved slightly up to 8 groups, with no further gains beyond that. Based on these findings, we set the number of groups to 8, which balances simplicity and performance.
21

---

Published as a conference paper at ICLR 2025
Table 16: Impact of group numbers KAT-Tiny Top-1 accuracy.
Group Number
4
16
32
KAT-Tiny Top-1 Accuracy 74.2
74.3
74.6
74.7
74.6
Maximum Order Rational Function. We set the rational order (m = 5, = 4) in the paper, following the default configuration in the PAU paper. We conduct a ablation experiment with the KAT-Tiny model to confirm this choice
Table 17: Impact of rational order on KAT-Tiny Top-1 accuracy.
Order (m,
(3, 2)
(5,
(7, 6)
KAT-Tiny Top-1 Accuracy
74.2
74.6
74.6
The experiment show that increasing the order beyond (m 5, n = provides no additional accuracy gains, making practical and efficient choice.
Benefit of CUDA Implementation. To evaluate the efficiency improvements introduced by our CUDA implementation discussed in Section 4.2 we conducted experiments to measure both forward pass speed and peak memory usage. Specifically, we compared our CUDA implementation against two alternative methods. The first called Torch Looped, which loops over each channel group. applies the rational function, and then concatenates the results. The second is called Torch Vectorized. In this method, the input tensor reshaped according to the channel groups, the rational function applied in vectorized manner, and the tensor is reshaped back to its original form. We compare these three implementation on A5000 GPU, under 1) different group number E {1,2, 4, 8, 16). 2) different input dim D {128, 256, 512, 1024, 2048)
Turch Looped
brrch Vedtailbed
CUBA Jours)
Terch Loosed
Tirch Vecto fbed
CUDA 80un)
Thrnughpat (batche))
+007
1800
(a) Throughput (batch/s) for Different Group Sizes. Larger the better.
(b) Peak Memory (MB) for Different Group Sizes. Smaller the better.
Figure 6: Comparison of Throughput and Peak Memory Different Methods and Group Sizes. Input size fixed to (64, 1000, 512].
The results, presented in Figure 6 and Figure and significantly outperforms both the Torch Looped and Torch Vectorized implementations, offering superior speed and memory efficiency.
clearly demonstrate that our CUDA implementation
Visualization of Trained Functions An important aspect to examine is the behavior the trained rational functions. As shown in Figure we plot the functions for KAT-S with across all 12 layers. The results indicate that within each layer, the rational functions exhibit similar trends, while the functions across different layers tend differ from each other. Another interesting finding is that most of the fitted rational functions are (near)-identity This indicates that the model learns to perform relatively simple operations rather than overly complex ones.
22

---

Published as a conference paper at ICLR 2025
Torch sooped
Vecirrtedd
CUDA 10us)
Loopez
Tonch Vectomaze
(Dus)
Pos Aibomoo AND
Throughppu
Input Shape Diinension)
Input Shape Dimersion)
(a) Throughput (batchls) for Input Dimension Sizes. Larger the better.
(b) Peak Memory (MB) for Input Dimension Sizes. Smaller the better.
Figure 7:
Comparison of Throughput and Peak Memory Different Methods and Input Dimension Sizes. Group size fixed to 8.
J DISCUSSION AND FUTURE WORK
Discussion, Our study highlights KAT's potential as good alternative to MLP--ased transformers, especially large-scale vision tasks. This integration introduces exciting opportunities for broad applications, For example, employing KAT architectures might help development of language models.
However, KAT is without its challenges. primary concern is running speed. Even with the CUDA optimized code, the rational function is still slower than plain activation like ReLU and GELU. Another issue the stability when using rational functions neural networks. The higher order gradients for am and b, can become unstable because of their dependence on the input power. Integrating these functions into the backpropagation process could introduce complications.
Additionally, it is important to acknowledge that our GR-KAN represents hybrid model. On the one hand, GR-KAN a KAN layer with shared edges and a rational base function. On the other hand, can be interpret MLP with redesigned activation placed before the linear layer. It leverages the computational simplicity of MLPs but maintains some characteristics of KANs. However, GR-KAN is not pure KAN model. Instead, it merges advantages from both systems to enhance overall functionality.
Future Work. There multiple directions of KAT for future research. One potential area of exploration to find alternative base functions to further improve computational efficiency and compatibility with emerging hardware architectures. Currently, rational functions serve as one option, but other possibilities exist. These include Fourier transformations (Noesis 2024), Wavelet
transforms (Bozorgasl & Chen 2024b), and Gaussian radial bases (Li 2024
Additionally, expanding the applicability of KAT to other domains beyond vision tasks, such as natural language processing or reinforcement learning, could unlock new opportunities for performance gains, Further research could also investigate hybrid models (Yang et al., 2022 Yu et al.) 2023), or adaptive mechanisms for dynamically selecting between KAN and MLP layers based on the complexity the task, thereby optimizing resource utilization. Finally, addressing the remaining scalability challenges, particularly in terms of memory footprint and inference speed, will be crucial for deploying KAT in real-world applications at scale.
23

---

Published as a conference paper at ICLR 2025
TAMES
Dom 2ar2
Figure 8: Fitted rational functions for KAT-S model, with 12 layers and groups
24


Reviews:
[{"review_id": "tL6u93km3J", "reviewer": "Reviewer_yXCb", "summary": "There has been a resurgence in Kolmogorov Arnold Networks (KANs) recently as an effective alternative to MLPs. This work carefully studies the major issues with scaling of the standard KAN architecture and proposes suitable modifications to create what they define as Group-Rational KAN (GR-KAN). The GR-KAN is then used as a replacement for the MLP layer in the transformer architecture in this work, which is then called the Kolmogoorv Arnold Transformer (KAT). The utility of the KAT architecture is demonstrated through experiments on various vision tasks such as Image Recognition, Object Detection, Instance Segmentation and Semantic Segmentation.", "questions": "Q1) As per my understanding, Kolmogorov Arnold Representation Theorem and the setup mentioned here involves learning a multivariate function, why do *lines 146-147* in the *section 2.2* mention *learn a univariate functions on the edge*. The dimension of $f(x)$ would be \n$d_{out}$ making it multivariate?\n\nQ2) Do we have a theoretical understanding of the behaviour outlined in lines *316-317*? How are the parameters adjusted in case of B-splines or any other type of functions?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1) Existing works using KANs in-place of MLP modules in various models (such as feedforward NNs, CNNs, Transformers etc) have had limited success due to various issues. This work goes a step further and proposes an alternative to the original KAN architecture called Group Rational (GR-KAN) solving the issues for inefficiencies in base functions, parametrizations and base initializations reducing the parameters and FLOPs from the original KAN architecture.\n\nS2) The GR-KAN employs rational activations as the base function which are efficient and suitable for GPU computation replacing the B-Splines which are not optimized for GPU or parallel processing. (which is one of the core strengths of this work amongst related KAN literature)\n\nS3) The KAT architecture improves the performance over standard transformer architectures such as ViT within the same computational budget.", "weaknesses": "W1) This work doesn’t include results on using KANs in solving PDEs (as the original work did, albeit the architecture won’t be transformers for this task) and KATs for language processing (mentioned in future work). The experiments are majorly limited to vision tasks. (Table and Graph Classification results have also been included.)\n\nW2) Ablations on the function which parametrizes the learnable activation are not included. Ablations are mainly with respect to alternative MLPs and fixed activation functions. Ablations could have included alternatives to rational functions, B-splines etc.\n\n**Presentation/Typos/Corrections**\n\nI believe that the *Experiments* section can be shortened or moved to appendix and the *Appendix F Discussion and Future Work* be moved to the main paper as it packs a lot of relevant quality content.\n\n*Appendix D Section D.1*\nThis is a minor correction, the number of multiplications in exponent will be $\\frac{m(m-1)}{2}$ and correspondingly $\\frac{n(n-1)}{2}$ instead of $\\frac{m(m+1)}{2}$ and $\\frac{n(n+1)}{2}$ respectively. \nAs an example case for $m=1$, the numerator becomes $a_0 + a_1x$ which involves 0 multiplications in the exponent computation and a single multiplication from the multiplication for the coefficients.\nThe total multiplications will be in fact $\\frac{m(m+1)}{2}$ which is $\\frac{m(m-1)}{2}+m$\nNote that with this change the values in the Table 1 and the appendix D will change accordingly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J1yH9uEFSd", "reviewer": "Reviewer_h5qV", "summary": "The paper introduces the Kolmogorov–Arnold Transformer (KAT), where they replaces MLP layers in transformer with Kolmogorov-Arnold Network (KAN) layers. B-spline functions in KANs were replaced with rational functions to improve compatibility with modern GPUs. Group KAN was used to reduce computations and variance preserving initialization was employed. Introduces Group-rational KANs for intergation into ViT. Extensive experiments on image classsification has shown improved performance of GR-KAN.", "questions": "How do you explain the contradiction of saying KANs require less paramters and then saying KAN requires more parameters?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Indentified issues with integrating KANs with ViT.\nOvercame the recursive computation requirement for using B spline curves in KANs and replaced them with faster rational functions.\nIntroduced variance preserving initialisation to stabilize training.\nCreated an efficient an fast CUDA implementation of rational function for faster running in GPU\nImproved performance gains by grouping and sharing parameters\nExtensive experimetation is provided", "weaknesses": "Replacing MLP in transformers with KAN is an obvious choice and does not warrant any novelty.\nphi_in matrix in eq. 2  Top row right element should show phi_1,d instead of phi_1,n\nIn line 46, the paper says KANs require fewer parameters than MLPs, then goes on to show in Line 176-189 than KAN requires more paramters.\nThe violation of variance-preserving nature in higher order B-splines is not clear.\nThe basic novelty is the use of rational function instead of B-splines", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bXp3VeqOKn", "reviewer": "Reviewer_cLED", "summary": "This work presents the Komlogorov-Arnold Transformer, a transformer architecture that replaces the typical transformer layer's concluding 2-layer MLP with (modified) Kolmorogov-Arnold layers. The authors explain why a simple substitution of KAN layers will fail to both perform well and to scale with the hidden dimension of the network, and provide fixes (via different base functions, weight sharing across grouped edges, and improved initialization) that improve performance and speed. Improvements are shown in experiments across a variety of machine learning tasks in vision domains and beyond.", "questions": "1) Can you explain very specifically the novelty of your method? You break your contribution into three pieces, can you say which portions of these are novel and which are not?\n2) Much of the presentation about scalability is presented through FLOPs computation. Is there a reason that you chose this representation, rather than actual training/inference time?\n3) A major part of this work is the CUDA implementation of the layer. How do you plan to make this available? Will the layer also be eventually made available through frameworks like pytorch or jax? I appreciate the work that goes into a direct CUDA implementation, but I think it is crucial that there are plans for wide availability in popular frameworks in order to have this work translate to impact in the community.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1) The paper is generally well written: it has a clear narrative, each new section is well-motivated, and for the most part, technical discussions are nicely balanced with takeaways and intuition.\n2) The technical analysis and contributions of the paper directly lead to practical improvements. The entire chain of derivations/modifications creatively connect different lines of research (KAN networks + rational neural networks, for example) and add some new techniques as well. In particular, I found the re-expression of the grouped parameters in Section 4.3 into a specialized MLP to be quite elegant.\n3) The experiments are comprehensive, all of the important training details are made available, and the results show that using these layers does lead to improvement in accuracy with relatively little increase in training time.", "weaknesses": "1) I felt some details about the method and novelty were missing. In my opinion this is mostly clearly true in the initialization section (which seems to be where a lot of the improvements/convergence come from). A lot of Section 4.4 is spent showing that choosing these initialization values is difficult and non-obvious - which I agree with! - but then there is no space left to actually show how you pick values for a and b (I know you say that you \"determine a and b such that F fits established activations\", but a small algorithm or set of equations would do a lot to disambiguate this).\n\n2) The suite of experimental results is impressive, but there are comparatively few ablations studies. I am left wondering which parts of your method are responsible for the improvement over standard ViTs. This is particularly disappointing because you clearly break down your improvements step-by-step, so it is easy to imagine what such an ablation would look like. I realize that some of your modifications are meant to address scalability, so a large set of experiments might be infeasible. But something like accuracy & total training time of a ViT-T with different parts of your method (i.e., with/without your initialization scheme, with/without grouping, etc) seems like it should be possible.\n\n3) Similarly to the above: after switching the base functions to rationals, the GR-KAN layer in practice resembles the rational layers from Boulle et al much more than it resembles the original KAN layers. But there are no direct comparisons to those; I would also find this experiment useful.\n\nMore minor pieces of feedback:\n\n4) On L248: I don't think it's quite right to say that this gradient computation is \"Similar to Monila et al\"; it is identical to their calculation. I also don't think the details of this calculation are used anywhere later in the paper, unless I am mistaken, so they could also be relegated to the appendix.\n5) Table 1: This caption could use more detail - is this just the forward pass, or forward and backward passes? Also, a comparison to an analogously-sized MLP layer would be additionally useful.\n6) L433: \"ViT-L + KAN fails to converge\" - I think this should be ViT-B?\n7) Figure 3: I think I understand what this figure is trying to communicate (i.e., that the choice of values for $a$ and $b$ do a good job of fitting the activation functions), but in practice it just ends up looking like... graphs of activation functions. Something quantitative would probably be more effective is communicating how well the rationals approximate the activation functions, and it would take up less space as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kFvOz9UH3N", "reviewer": "Reviewer_DyMH", "summary": "In this work, authors propose Kolgomorov-Arnold Transformer (KAT) as an improvement to standard transformer architecture, which uses MLP networks, by replacing MLPs with KAN networks. Authors propose a number of improvement over vanilla KAN architectures to make them scalable in practical settings and achieve better results: 1.) they swap b-spline functions with rational function for speed 2.) they make use of parameters sharing (grouping) 3.) they employ a variance preserving initialization.\nResults show that KAT achieves better results than ViT on certain benchmarks.", "questions": "- In Eq. 5, 6, and 7 what are MSA and LN? \n- Missing text in paragraph name in line 485 \n- In Tab. 4 KATDet-S is reported as best (41.5) but ConvNext and SwinT are higher (41.7 and 41.6)\n- I feel like the change you propose can be applied to KANs also outside of the transformer architecture. Perhaps it could be useful to add a comparison between KANs and GR-KANs on toy problems", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The topic is highly relevant, as literature has not focused much on improving the MLP aspect of transformers\n- KANs have emerged as an interesting alternative to MLPs, however, they present issues (e.g. scaling) which make them unfavorable w.r.t. MLPs\n- The proposed approach is scalable, faster, and requires less parameters than KANs, making it usable in practical settings and on large-scale datasets such as ImageNet-1k or MSCOCO\n- The experimental evaluation comprises different tasks such as classification, object detection and segmentation.", "weaknesses": "Here are my doubts about this work: \n- While the results for ImageNet are convincing, the proposed KAT still lags behind traditional architectures such as ConvNext or even transformers such as SwinB in segmentation. \n- A proper ablation study is missing (e.g. combinations of KAN + rational function + grouping + initialization). It is difficult to evaluate the contribution of each proposed change quantitatively. Also I wonder how simple KAN perform with parameter grouping in terms of MACS/FLOPS and results? \n- It is not clear how you choose the hyperparameters for KAT such as number of groups and maximum order of rational function (m)\n- (More of a philosophical doubt) KAN are based on the Kolmogorov-Arnold representation theorem, which states that a multivariate function can be approximated by a composition of univariate functions, while your proposed method only uses one irrational function. Can you still say it's based on the Kolmogorov-Arnold theorem? \n- (minor) some typos and missing notation are present in the paper (see questions below)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JFzKrZ0rRU", "reviewer": "Reviewer_s19p", "summary": "The author proposes and implements Kolmogorov-Arnold Transformer (KAN) architecture. Especially, they propose rational activation functions, group KAN, and variance-preserving initialization, demonstrating improved performance over conventional vision transformers across various tasks.", "questions": "* Could you specify where in the report it states that KAN is \"10x slower than MLPs, given the same number of parameters\"?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "* Comparative experiments are conducted on image classification, object detection, instance segmentation, and semantic segmentation.\n* Challenges and solutions are clearly explained, supported by experimental results.\n* The proposed method is sufficiently general to be applicable across different transformer architectures.", "weaknesses": "* In Tab 1, given the efficient CUDA implementation, experimental results of GPU computation are needed. Specifically, the inference time on a specified GPU should be provided.\n* The choice of the number of groups in group-rational KAN is not discussed. What is the reason for the current choice of group number? An experiment analyzing the effect of different group numbers would be beneficial for better understanding.\n* For consistency, it would be helpful to include the GPU information used for image classification task as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)