Improving Memory Efficiency for Training KANs via Meta Learning
Authors: Zhangchi Zhao
Jun Shu
Deyu Meng
Zongben Xu
Venue: ICML 2025 poster (2025)
Keywords: Hypernetwork;Kolmogorov-Arnold networks;Memory efficiency;Scalability and Extensibility
Primary area: deep_learning->algorithms
TL;DR: We propose an alternative technique for improving scalability and extensibility of KANS via Meta-Learner

Abstract: Inspired by the Kolmogorov-Arnold representation theorem, KANs offer a novel framework for function approximation by replacing traditional neural network weights with learnable univariate functions. This design demonstrates significant potential as an efficient and interpretable alternative to traditional MLPs. However, KANs are characterized by a substantially larger number of trainable parameters, leading to challenges in memory efficiency and higher training costs compared to MLPs. To address this limitation, we propose to generate weights for KANs via a smaller meta-learner, called MetaKANs. By training KANs and MetaKANs in an end-to-end differentiable manner, MetaKANs achieve comparable or even superior performance while significantly reducing the number of trainable parameters and maintaining promising interpretability. Extensive experiments on diverse benchmark tasks, including symbolic regression, partial differential equation solving, and image classification, demonstrate the effectiveness of MetaKANs in improving parameter efficiency and memory usage. The proposed method provides an alternative technique for training KANs, that allows for greater scalability and extensibility, and narrows the training cost gap with MLPs stated in the original paper of KANs. Our code is available at \url{https://github.com/Murphyzc/MetaKAN}.

Improving Memory Efficiency for Training KANs via Meta Learning
Zhangchi Zhao
Jun Shu
Deyu Meng 12 Zongben Xu'
Abstract
ral network architectures, ranging from MLP (Cybenko, 1989) to CNN (LeCun et al., 1989), RNN (Elman, 1990), LSTM (Hochreiter & Schmidhuber, 1997), and Transformer (Brown et al., 2020) architectures. While these architectures generally use fixed activation functions, which limit their interpretability and flexiblity.
Inspired by the Kolmogorov-Amold representa- tion theorem, KANs offer a novel framework for function approximation by replacing traditional neural network weights with learnable univari- ate functions. This design demonstrates signif- efficient and interpretable
icant potential alternative to traditional MLPs. However, KANs are characterized by substantially larger num- ber of trainable parameters, leading to challenges in memory efficiency and higher training costs compared to MIPs. To address this limitation, we propose to generate weights for KANs via a smaller meta-learner, called MetaKANs. By train- ing KANs and MetaKANs in end-to-end differ- entiable manner, MetaKANs achieve comparable or even superior performance while significantly reducing the number of trainable parameters and maintaining promising interpretability, Extensive experiments on diverse benchmark tasks, includ- ing symbolic regression, partial differential equa- tion solving, and image classification, demon- strate the effectiveness of MetaKANs in improv- ing parameter efficiency and memory usage. The proposed method provides an alternative tech- nique for training KANs, that allows for greater scalability and extensibility, and narrows the train- ing cost gap with MIPs stated in the original paper of KANs. Our code is available at https: //github. www.//m///////. wwwp////pyyee///ettaata.oo
Recently, Kolmogorov- Networks (KANs)(Liu et al., 2024), inspired by the Kolmogorov-Aroold representation theorem(Kolmogorvv, 1957), introduced novel neural ar- chitecture. Rather than employing traditional fixed activa- tion functions, KANs utilize learnable, pplin-paramettrzedd univariate functions on edges, enabling a more flexible and interpretable framework for function approximation, Re- cent research on KANs has focused on modifying the B- spline functions to improve their performance. For instance, Chebyshev KAN (SS et al., 2024), GramKAN, and KALN used Chebyshev polynomials, Gram polynomials, and Leg- endre polynomials, respectively. to enhance the non-linear function approximation capabilities. Similarly, (Bozorgasl & Chen, 2024) used wavelet basis functions to improve the model's ability of capturing both high- and low-frequency components of data. Additionally, (Yang Wang, 2024) and (Li, 2024) replaced B- B-splines with rational polynomi- als and radial basis functions to improve the computational speed of KANs. To enhance KANs' capability in image clas- sification tasks, (Bodner et al., 2024) proposed KA, while (Drokin, 2024) extended ConvKAN to more convo- lutional structures. Although these KAN variants improve approximation and computational efficiency, the presence of learnable activation functions leads to large number of trainable parameters. This design significantly impacts memory efficiency and increases training costs, posing chal- lenges for addressing larger datasets and complex tasks.
1. Introduction
With the rapid development of deep learning, artificial in- telligence (AI) has achieved remarkable progress in vari- ous fields (Brown et al., 2020; Jumper et al., 2021). This progress is largely due to advancements of various neu-
To address these challenges, we propose MetaKANs, a memory-efficient framework that improves training effi- ciency for KANs through meta learning. Our approach builds on the key observation: KANs decompose high- dimensional functions into dimension-wise univariate func- tions (learnale,, spline spline-parametrian parametrized activation functions), while these univariate functions follow a common functional class F, in which they share the same trainable parameters setting rule. Inspired by current meta learning (Kong et al., 2020; Shu et al., 2023) and in-context learning (Brown et al., 2020; Garg al., 2022) techniques, we propose to
'School of Mathematics and Statistics, Ministry of Education Key Lab of Intelligent Networks and Network Security, Xi'an Jiaotong University "Pengeheng Laboratory. Correspondence to: Jun Shu <junshu mmi..xjuu.duu.c..
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Improving Memory Efficiency for Training KANs via Meta Learning
Input
Weight generation
Backward pass
Forward pass
KAN
KAN
XEX
Loss
yEY
Loss
yev
and Ha eu Eer aud 6:10
Ma
(a) KANs
(b) MetaK ANs
Figure/ 1. Overview of the architectures for KANs and MetaKANs. The connections marked red means trainable. The trainable parameters are w for KANS and 0 2/, for MetaKANs.
Lesend
(2) Proposed method is model-agnostic, which could be applied to various KAN variants. The memory efficiency improvements upon FastKAN (Li, 2024), ConvKAN (Bod- ner et al., 2024), and WavKAN (Bozorgasl & Chen, 2024) demonstrate the universal applicability for MetaKANs.
Clasuitationn (NU
(3) We experimentally demonstrate that MetaKANs improve training efficiency in terms of reducing memory load across various benchmarks, including symbolic regression, solv- ing PDEs, and image classification. Experimental results substantiate the capability of MetaKANs on enhancing the scalability and extensibility of KANs.
2. Related Work
2.1. Kolmogorov-Arnold Networks (KANs)
Figure 2. Parameter-accuracy tradeoff comparison between KANs and MetaKANs across six benchmark datasets. Marker color indicates dataset, shape represents model type (circle: KANs, square: MetaKANs). MetaKANs demonstrates superior memory efficiency while maintaining competitive performance, The arrow indicates that MctaKANN not only reduce the number of parameters but also maintain performance.
KANs (Liu et al., 2024) introduce a novel network structure by replacing fixed activation functions in traditional MLP networks with learnable activation functions. This provides more flexible and interpretable framework for function ap- proximation tasks, Recent studies on KANs can be broadly categorized into three research directions: parameterization strategies of basis functions, domain applications, and theoretical foundations.
use a smaller meta-learner to simulate the shared parameter generation rule, and thus the large number of trainable pa- rameters for activation functions reduce to the parameters of the smaller meta-learner. Specifically, the parameters of MetaKANs consists of two parts: (1) learnable input vari- able (i.e.. prompts) serving as the unique identifier for each activation function and (2) the parameters of meta-learner learning the shared weight setting rule for dimension univariate functions. The diagram is shown in Figure 1.
To enhance expressivity and computational efficiency, di- verse basis functions have been explored beyond the origi- nal B-spline formulation. These include polynomial bases such as Chebyshev, Legendre, and Gram polynomials (SS et al., 2024), rational functions (Aghaei, 2024), radial basis functions (Li, 2024), wavelets (Bozorgasl & Chen, 2024). Besides, KANs have been integrated into diverse architec- tures and tasks. In computer vision, ConvKAN (Bodner et al., 2024) and KAT (Yang & Wang, 2024) enhance CNNs and Transformers, respectively. For time-series, frequency- aware models like TIMEKAN (Huang al., 2025) and -ugmentd variants (Genet & Inzirillo, 2024) im- prove long-term forecasting. U-KAN (Li et al., 2024) ap- plies KANs to U-Net for medical imaging, while KAN- ODES (Koenig et al., 2024) model dynamical systems. In graphs, KA-GNNs (Bresson et al., 2024) and Kolmogorov-
In a nutshell, we make the following contributions:
(I) We propose novel memory efficiency training method for KANs, called MetaKANs, which generates weights of KANs via smaller meta-learner instead of directly optimiz- ing parameters of KANs. As shown in Figure 2, proposed method achieves comparable or even better performance with only 1/3 to 1/9 parameters of the number of trainable parameters for KANs.
2

---

Improving Memory Efficiency for Training KANs via Meta Learning
Arnold Attention (Fang et al., 2025) boost expressive ca- pacity. Other extensions include continual learning classi- fiers (Hu et al., 2025) and generative modeling via learned Schrödinger bridges (Qiu et al., 2025). Theoretical work has begun to formalize KANs capacity and generalization prop- erties. (Zhang & Zhou, 2024) established generalization bounds, quantifying KAN complexity based on activation properties. While expressiveness studies show that KANs can outperform standard MLPs under large-grid regimes and exhibit weaker spectral suggesting better modeling of high-frequency components (Wang et al., 2024).
3. Methods
3.1. Preliminary
3.1.1. KOLMOGOROV-ARNOLD REPRESENTATION
THEOREM
The KA representation theorem (Kolmogorov, 1957) states that any multivariate continuous function can be decom- posed into finite sum of univariate functions. Specifi- cally, for continuous function f [0, 1] R. there exist continuous univariate functions dour R + R and iin [0, 1] + R such that:
Although these developments have advanced KANs signifi- cantly, the presence of learnable activation functions results in large number of trainable parameters and challenges in memory efficiency than MLP network, posing challenges for addressing larger datasets and complex tasks. Unlike these prior works, our proposed MetaKANs leverage a meta- learner and learnable prompts corresponding to activation functions, achieving a trainable parameter count comparable to that of MLPs with similar structures.
2n-1
f(x) = f(x), Tn)
Σ
dout Σ
in ((() (I)
This theorem could be formulated as follows:
f(x) T out o in o x,
where Pin and Dout are defined as:
(f)
(-)
2.2. HyperNetworks
Pin-
din (2 ()) d (-)
HyperNetworks use a smaller auxiliary network to generate the weights of a larger main network, offering a mechanism to reduce trainable parameter counts while maintaining com- petitive performance. First introduced in (Ha et al., 2017), hypernetworss achieved comparable accuracy to typical CNNs and RNNs with significantly fewer parameters, lay- ing the foundation for broad range of follow-up research. Recently, Mahabadi et al. (2021) applied hyperneworrs to Transformers for efficient fine-tuning across multiple tasks
4 out [dq (.)
22n+a())1.
donnt
3.1.2. KOLMOGOROV- NETWORKS
KANs implement the KA representation theorem using a parameterized neural network. Each activation function in KANs is represented by basis function and its associated parameters. In practice, Liu al. (2024) explicitly parame- terize each univariate function d(-) as B-spline curve, with learnable coefficients of local B- B-spline basis functions,
In meta-learning, (Zhao et al., 2020) reformulated the cemplate-sharing mechanism as a hypernetwork guided by leamable input vectors. (Navon al., 2021) extended this idea to multi-objective settings, using task embeddings to condition a shared hypernetwork, allowing task- specifcc weight generation without duplicating full networks. Hy- pernetworks have also been tailored to specific domains. For instance, (Alaluf et al., 2022) adapted the concept to StyleGAN, enabling efficient, real-time image editing by modulating generator weights with compact hypernetorkss.
G+
&(1; w) wH + Σ B.(,)),
(2)
where SiLU(!) I w [105, C, C1, CG+k]] and (Bt(t), E [G + k|} is the B-spline basis functions de- termined by the grid size G, spline order k. The param- eters wh controls the overall magnitude of the activation function, while e; are trainable coefficients. We denote B() = (SiLU()), (1), ( so the learnable univariate activation function can be written as:
Prior work typically employs heuristic parameter generation strategies, such as using shared bypemetworks for entire lay- ers (Ha et al., 2017 7) or task-agnostic templates (Savarese & Maire, 2019). In contrast, propsoed MetaKANs is grounded in the understanding of working mechanism of KANs, in which the weights learning could be regarding as the multi- task basis representation learning from a shared functional family. Instead of heuristically predicting weights through hypemetworks, we explicitly model the shared weight gen- eration rules through a meta-learner, where the learnable prompts as the identifiers for activation functions.
o(t: w) w B().
The general KANs consist of T. layers, with the structure [no, R.,..., RzI, where nı denotes the number of nodes in the 1-th layer. The outputs of 1-th KAN layer is defined as
x (+1)
Σ
(1
(1) D (2 w whm
())
W
2 W
(4)
3

---

Improving Memory Efficiency for Training KANs via Meta Learning
Table I. Comparison of parameter number among different models with the architecture [na, R1,, reL]-
ARCHITECTURES
PARAMS.
MLP
i-n (n x n++1)
KAN
Ei=o (TL n x
METAKAN
Ja (ny M++1+CC x (dhidden + +1) (G + +k+1)
FASTKAN
>i= (ni X 1x
METAFASTKAN
S1-1 (nr x n441 144+)++ (dhiaden + 1) x
WAVKAN
El= (mi ((+1)) x3
METAWAVKAN
El (mu x m441) (dhidden + 1) x
where x0) E R is the input of [-th layer, and 4 (2) activation matrix consisting of 11+1 X activation functions o wo!) )) which is given by:
wa is
(q) These parameters collectively account for the total number of learnable parameters in KANs, which can be expressed by Et-1 (my x n1+1) x (G + k++ +1) 1).
For comparison, the forward computation in an MLP can be expressed follows
(:.(::l.)) (1)
$1,m (:3 (::W.. win)
Φ Wa)
MLP(x) ooll.))ooo....W w .
Omiia 1.1 (5 1 Sn 1 (33 w Wii
The (-th KAN layer is parameterized by W
(1) E IT. Here we denote [n] {1....n) n} and the index set for the activation functions as Iz = [n]] x [m++1] for / E
where W E R X represents the linear transforma- tion, and a denotes the nonlinear fixed activation func- tion. Thus an MLP with the same structure requires only Σls (n: n4+1) parameters. This shows that the param- eter count for a KAN is approximately (G + k)x larger than that of an MLP (refer to Table 1). Such an increase in parameter count introduces challenges for KANs, including greater memory usage and potential scalability issues.
[L] - 1. Thus KAN with structure [na, m,,..., has a set of parameters denoted by:
L-1 w U wO Ünm e R
L-1
a | (5)
Given input vector x E Ro, the output of the KAN is
3.2. Improving Memory Efficiency via Meta Learning
KAN(X; =
(1)
&
W (x).
(0)
(6)
3.2.1. UNDERSTANING WORKING MECHANISM OF 2-LAYER KANS
way
It can be seen that the KA representation theorem in Eq. (1) corresponds to a 2-layer KAN with shape [n. 2n 1,
Consider an n-dimensional input x = (x1,..., In) and a target function f(x) represented by the Eq. (1) accord- ing the KA representation theorem. Thus f(x) could be approximated by 2-Layer KANs with shape [n, 2n + 1,
KANs have been demonstrated superiority in fucntion fit- ting than MLP network (Liu et al., 2024). Suppose the target function f(x), then the learning objective can be formulated
2n+1
n
f(x)
1))
Σ
Pio
(1:)
W = arg min E Px W), (7)
2n+1
where é denoted the loss function, and we set MSE for regression and cross-entropy loss for classification.
(1)
Σ
wiy "
(1)
~
Pio
(I
1=1
KAN(x: W).
3.1.3. MEMORY INEFFICIENCY
Here we could understand the learning weights W ((0) UW()) as a multi-task learning problem. Specifically, for the l-layer weights (0)), the weights learning
KANs differ from MLPs by placing learnable coefficients in each activation function, while MLPs place fixed activation function in the architecture design. This could increase the number of parameters compared with MLPs, especially as the growth of network's width and depth.
learn x (2n 1) regression tasks belonging to the shared fune- ()),())), tion class Fi = (z))) w T )) w E RG+++1];
and the lcarning of 2-layer weights ( is to learn (2n + 1) x regression tasks belonging to the shared func- tion class F2 = (f) = w B()()), w e IG++++)), } where B-)) defined in Section 3.1.2. In fact, each regres-
From the above, it can be concluded that for KANs with the structure [no. n.,..., nn), a B-spline function of order k with G grid points require parameters W as presented in

---

Improving Memory Efficiency for Training KANs via Meta Learning
sion task could be expressed by
Algorithm 1 The MetaKANs Algo. for Shallow KANs
Input: Training dataset D batch size n, max iterations
: RG+++
(8)
T. KAN and MetaKAN models,
Output: Updated meta-learner parameters ((T)) and
where T denotes the task information to identify which activation function it is, and the output w E RG+K+ corre- sponds the learnable coefficients for the activation func- tion. The regular learning of KANs in (7) overlooks the underlying common weights learning rule, and optimize the weights in a brute-force way. This learning strategy signifi- cantly impacts the computation and memory efficiency of KANs, posing challenges of KANs for addressing larger datasets and complex tasks compared with MLP network.
prompts Z(T)
Initialize meta-learner parameters g()) and learnable prompts g
for = 0 to T 1 do
Sample mini batch fxp. yo from D of size n
Compute the output of MetaKAN using Eq. (3.2.2)
Compute training loss with Eq. (12)
Update ((++1) and ( via gradient descent
end for
3.2.2. PROPOSED METAKANS FRAMEWORK
special instance. Thus the number of trainable parameters of MetaKANs is
To improve the memory efficiency of KANs, the key insight is to delicately employ the commom weights learning rules in (8). To this aim, we proposed a Meta-KANs frame- work as shown in Figure to directly predict weights of KANs inspired by current advances on meta-learning (Kong et al., 2020; Shu et al., 2018; 2023) and in-context learning (Brown et al., 2020; Xu et al., 2024). Specifically, we design meta-learner to parameterize the weights learning rules in (8) as follows
2 10 (n m4+1) + (dhidden 1) x (G+k+11)
(=U To obtain the meta-learner, we can optimize the following objective function derived from (q)
0 arg min P(x) [<(MotaKAN(x: 2,0) f(x)) 2,0
(12)
During the training process, both the parameters of the meta- learner and the learnable prompts Z are updated simulta- neously by gradient descent algorithm. The overall training algorithm for MetaKANs formulated in Algorithm.
Ma R GG
(9)
Here use learnable task prompt E R. identify which activation function it is as popular studied in current large language models (Brown et al., 2020), and Mg is formulated as two-layer MLP with hidden dim equals to dhidden. This choice is motivated by the MLP's capability as a universal function approximator, as used in previous meta-learning methods (Ha al., 2017; von Oswald et al., 2020: Shu et al., 2019) and its simplicity and computational efficiency.
3.2.3. SCALING TO DEEP KANS
Based on the analysis in Section 3.2.1, we could further understand the weights learning for a multi-layer KAN de- fined in Eq. (6). Specifically, for the l-layer weights ( the weights learning is to learn n4 X 1u++1 regression tasks belonging to the shared function class Fi = (z)(((())))))))))))))))))))))))) = w B(x()), w E G++++1). }. In other words, the weights learning of deep layer of KANs is akin to that of the shallow layer, while the input task information is shift and evolution- ary accross different layers. This mismatch task distribution makes the weighting learning rule in Eq) different across layers. Though we introduce the learnable prompts Z as input information for each task, is hard to mine the proper layer information use a shared meta-learner, and thus bring inferior performance as demonstrated in Table 9.
Based on this formulation, the learnable activation function in Eq. (3) now becomes:
(10) and then the 2-layer KAN in Eq. (8) now becomes:
@ (t:2, 2, 0) (o(=)TB(()))
MetaKAN(x: 2,6)
2n+1
n Σ
(a (o(2)) Mo(:())) 1i
(0)
To address this issue, straightforward way is to assign different meta-learners for every layer. This design allows (1) for (-th layer to effectively capture the
Note that the learnable parameters in activation function of KANS are converted into the parameters (2, 6) of the meta-learner, where Zi is defined as (see Sec.4.4)
the meta-learner M layer-dependent task information, which could potentially help extract proper weights learning rules for different layers. However, such setup would introduce significant parameter overhead because that the total parameters are scaling with the number of layers L, which tends to further raise their memory inefficiency for deep KANs with many layers.
L-1 Z - Uf e E a Iı - [md] x [n]-]]. (11)
1=0
This definition corresponds to the general case of a L-layer KAN, and the previously discussed case with I, a
5

---

Improving Memory Efficiency for Training KANs via Meta Learning
Algorithm 2 Clusters Determination Algo. for Deep KANs
Algorithm 3 The MetaKANs Algo. for Deep KANS
1: Input: Output-channel sizes n = [ RL-1]; num-
I: Input: Training dataset Dir; KANs with L layers ar-
ber of clusters C
chitecture and MetaKANs; Layer clustering ( [Le
2: Output: Intervals ('sinnt ( Jm
obtained using Algorithm 2, where Le | lemii
3: labels t KMEANS(n, C)
Batch size n; Max iterations T
4: for 1 to C do
Output: Updated meta-learner parameters (
5: Le + {1] labels[[] = c}
and prompts Z(T)
6: 1saatt <- min Le,
lond + max Le
3: Initialize meta-learner parameters O = ={0 [0(0) [00) 1e
7: end for
8: return {1 (emi)) )
c=1
and learnable prompts 2
4: for 0 to T do
5: Sample mini-batch (x6. from DDr of size n
To make balance between expressive capacity and memory efficiency, we extend the MetaKANs framework by parti- tioning the deep KAN's L layers into C' distinct clusters, {In...., [. each assigned a dedicated meta-learner M(e) This partitioning guided by the input and output channels of the layers, strategy that typically groups consecutive layers L. [ [end]] which exhibit similar channel dimen- sion, as illustrated for a ConvKAN in Sec. C.4. The detailed
6:
for - 0 to L -- do
7: Generate weights for l-th layer using Eq3
8: Propagate through /-th KAN layer via Eq. (14)
end for
10: Compute training loss with Eq. (15)
11: Update ( and 2(++) via gradient descent
12:
end for
algorithm please refer to Algorithm 2.
Compared with case for shallow KANs, the increased num- ber of learnable parameters for deep KANs attributes the additional number of meta-learners (C 1 degenerates to the case discussed in Section 3.2.2). Their parameters C x (dhidden + + (G + k + 1) may remain constant due to the fixed architecture of meta-learners Mo (z), E [C]. While the term Σt (ny x ni+1), which accounts for the learnable prompts Z, will grow with the size of the KANs's architecture. Then the total parameter count of MetaKANs is approximately by
Based on this novel formulation, the parameters in Eq. (5) are generated with proposed MetaKANs:
w!O MB (=(0) VIE a I
(13)
where (()) represents the cluster-specific meta-learener pa- rameters. And we will validate this design in Sec. C.4. Then the output of the 1-th KAN layer in Eq. (4) is given by:
(1+1)) (1)
xx(1)
((0)
L-1
|2| R Σ (nz x n4+1),
Σ 1 (x, (1),
More (.)).....
11
(14) which is equivalent to that of a standard MLP network with the same structure. The detailed comparison of the number of trainable parameter among KAN, MLP and MetaKAN is shown in Table 1. And thus the proposed MetaKANs framework is potentially promising in improving memory efficiency of training KANs, and reduced memory cost ap- proximating to MLP network. In Section 4, we experimen- tally demonstrate that proposed MetaK. ANs could achieve competitive performance than orginal KANs.
Dimmir (ar Mo (2 (=(
. Ie.
The forward procedure of MetaKANs now becomes:
MetaKAN(x; 0) = (L-1) 21b-11,0 110(
T (0)
200) (1))) (x),
where θ = ]0 and the objective function is
z+,
3.3. Extension to Other KANS Variants
(15)
arg min Ex (MetaKAN(x: z, 2, -(x)) 2,0
It is worth emphasizing that proposed MetaKANs frame- work model-agnostic, which is capable of applying to recent KANs variants, e.g., WavKAN(Bozorgasi & Chen, 2024), FastKAN(Li, 2024) and ConvKAN(Drokin, 2024), etc. To illustrate this, we propose MetaKAN variants that apply our meta-lcarning framework (Section 3.2) to these architectures including MetaWavKAN, MetaFastKAN, and MetaConyKANN Detailed descriptions and technical details are provided in Appendix A. And we empirically verify the
The overall training algorithm for MetaKANs predict weights of deep KANs is formulated in Algorithm.3.
Memory efficiency analysis. The number of learnable parameters in MetaKANs for deep KANs with the structure [no, 71, nz] is given by:
|2++0 =
(nu ++ (dridden +1) (G+++1).
0=0
6

---

Improving Memory Efficiency for Training KANs via Meta Learning
Table 2. Performance Comparison Between KANs and MetaKANs on Feynman Dataset with Different Grids.
5
G = 20
NAME STRUCTURE
KANS
METAKANS
KANS
METAKANS
MSE
# PARAM
MSE
# PARAM
MSE
# PARAM
MSE
# PARAM
16.20A
[1.2.1.1]
5.94 X 10T4
58
3.88 x 10 4
218
1.01 X 10
134
5.48 X 10 3
653
16.20
[2.2.1.1]
6.44 X 10
76
2.67 x 10
210
6.00 x 10
193
4.15
x 10
439
16.20B
[3.5,5.5.1]
4.16 x 10-3
646
3.68 x 10 10-3 3
1083
3.79 X 10
1976
3.04 x 10
1,798
18.4
[4.5,5.1]
1.30 x 10-2
979
3.76 x 10-3
461
1.16 x 10-1
1.461
8.33 x 10-2
1,994
19918
16.5,5.5.1]
2.39 x 10-3
781
1.70 X 10-3
993
1.62 X 10
2,556
4.00 X 10-3
2,029
1.1.2
14.3.3.11
2,83 10-3
223
1.84 X 10
480
1.19 X 10
751
2.03 X 10
672
1.2.4
[3.3,2.1]
1,87 x 10
70
1.04 X 10
159
4.55 X 10
550
6.13
10
665
L.2.5
12,2,11
1.32 10
57
1.16
10
30
2.97 10
201
2.24
10 10
330
1.1077
13,4,4.1.1]
6.96
10
307
5.98
10
115
9.77 10
1,132
4.00
10 10
1,329
112.11
[5.2.2.1]
8.73 X 10
149
8.55
10 10
85
2.58 10
565
1.35
10
1,312
L13.4
[4.2,1,1]
4.22 10
103
3.52
10
55
1.84 10
400
1.21
10
1,955
L.3.12
[5,4,4,1]
3.41
10
369
2.83
10
100
1.77 10
1,489
3.09
10
1,336
I.14.3
[3,2,1]
4.53 10
75
4.01
10
40
x.66 10
307
5.73
10
332
1.14.4
12,2,1.11
1.27 10
2.08
10
35
1.16 1.16x10
277
1.69
10
319
1.1533X
[4.3,3.1]
1.57
10
223
5.38 x 10
2.59
x 10
967
2.33
10
181
I11.110
[3.3,2.1]
8.45
10
159
7.79 x 10
7.93
10 10
703
8.38
10
1,313
L18.4
[4,4.3.1]
2.21
X 10
287
1.55 x 10
90
4.63
10
1,310
8.19
10
189
effectiveness of MetaWavKAN (Table 7), MetaFastKANN
slightly more parameters than KAN, despite MetaKANs often maintain competitive performance.
(Table 6), and MetaConVKAN (Table 3), indicating the uni-
versal applicability of MetaKANs framework.
Memory efficiency analysis. For a general KANs model, the additional parameter count arises from the learnable univariate functions (>x; w), with the total parameters for network structure (ng, R1. nz. given by: El= (m1 X m++1) x dim(w), (w), where dim(w) G ++ for standard KANs. With meta-learner, the parameter count for KAN variants is reduced by approximately 1 1/dim(w) (dim(w). For exam- plc, the dimensionality dim(w) (c)) is for FastKAN and 3 for WavKAN. The meta-learner generates activation parame- ters, reducing the total parameter count to approximately as (3.2.3) which is comparable to that of an MLP. This makes MetaKANs and its variants significantly more memory efficient while retaining their flexibility and perfor- mance. Detailed comparisons are provided in Table 1,
4. Experiments
We conduct extensive experiments to evaluate the per- formance of our method, including function fitting tasks in both low-dimensional (Sec.4.1) and high-dimensional scenarios (Sec.C.1), solving partial differential equations (Sec.C.3), and image classification tasks using KANs with both fully-connected (Sec. C.2) and convolutional architec- tures (Sec.4.2). Additionally, we provide the analysis of memory usage in Sec.4.3, and perform ablation study in Sec.C.4 to demonstrate the scalability of our approach with respect to prompt dimensions and layers of KANs.
4.1. Function Fitting Task
4.1.1. 1. EXPERIMENTAL SETUP
3.4. More Discussion the Memory Efficiency
To demonstrate that MetaKANs reduce the overall number of trainable parameters without compromising the inter- pretability and imncionn-ittting capabilities of the original KANs model, we conducted a function-fitting task on the Feynman dataset (Udrescu & Tegmark, 2020) as done the original KAN paper. The comparison focused on the Mean Squared Error (MSE) performance.
It's important to understand the conditions that MetaKANs improve memory efficiency of KANs. KAN's parameters scale as E/= (T4 x m++1) x (G + k + 1), reflecting total number of activations multiplied by the number of param- eters per spline. MetaKANs generate these spline param- eters with one prompt per activation (Et-) (nı x n4-1) total) but adds fixed meta-learner cost (s= C'dridden + () + & 1)). Parameter reduction occurs when this fixed cost is less than the total spline parameters saved. This advantage is more pronounced for larger and deeper net- works. By simple computation, we roughly require that G+* El- (ni X n1+1) to ensure the mem- ory efficiency of MetaKANs (see Figure 4). Consequently, for very small KANs in Table 2 with few total activations, the meta-learner's fixed cost can cause MetaK to have
In the experiments, employed the LBFGS optimizer with an initial learning rate set to 1, consistent with the original paper. Additionally, to better balance fitting performance and model complexity, the hidden layer nodes of the meta- learner of MetaKANs were set to 32, 64. The number of grid points in set to G 5, 20. The architecture of the KANs, including its depth and width, was designed based the complexity and characteristics of each target function, following the principles outlined in (Liu et al., 2024).
drsdden 2 G+K+1
7

---

Improving Memory Efficiency for Training KANs via Meta Learning
Table 3. Performance Comparison on MNIST, CIFAR1O, and CIFAR 100 (4 and 8 Layers).' on 8-layer MctaKANConv and MetaFast AN- Conv means we set the dimension of learnable prompts 2 4 respectively and using the multiple meta- retaaleersees.
MODEL
MNIST
CIFAR-10
CIFAR-100
PARAM
ACC.
PARAM
ACC.
PARAM
ACC.
4 LAYERS
KANCONY
3,489.774
98.43 0.46
3,494,958
41.924 1.87
3,518,088
7.69 0.34
METAKANCON
391,255
96.03+ 1.15
392,887
45.97 6.22
416,017
9.71+ 1.02
FASTKANCONY
3,489.292
99.36+
3,494,480
68.12+ 2.85
3,517,610
34.64+
METAFASTKANCONN
391.829
98.541
392,409
66.69 1.11
415,539
32.11
KALNCONY
1,940.330
8..641
1,943,210
32.98 4.25
1,966,340
10.481
METAKALNCONY
391,119
97.641 1.82
391,919
42.46 11.00
416,393
11.781 3.08
KAGNCONY
1,940,346
09.15+ 0.11
1,943,226
72.08 0.69
1. 1,966,356
36.61 0.49
METAKAGNCONV
391,359
99.21
392,383
73.49 0.95
415,513
35.54
8 LAYERS
KANCONV
40,694,018
99.50 0.08
40,696,610
67.24
40,742,780
35.97
METAKA NCONV
99.46 0.08 4,5
9,053,531
72.20
9,099,371
44.17" + 3.19
FASTKANCONY
40,693.052
40,695,648 0.07
0.46
40,741,818
50 0.18
METAFASTKANCONY
4.539,652
99.42 +: 0.08
9,051,033
78.73 1.59
18,142.821
52.02* + 1.07
KALNCONV
22,611,642
69.02 10.54
22,613,082
29.144 1.67
9.10 22,659,252
METAKALNCONV
4,529,727
99.53+ 0.05
4,530,015
70.23+
1,5
21.94
KAGNCONY
22,611.674
99.581 0.06
22.613.114
84.6
22.659.284
56.41
METAKAGNCONV
4,543,682
99 99.461:00 0.20
4.530,495
84.25 0.36
4.576,665
0.57
structure [4, 5, 5, 1)) respectively on fitting the function f(x) exp (1 (sin (= (1f + =2)) + sin (1 (x3 =))))) in Figure 9. From the figure, we observe that the functional class learned by MetaKANs is significantly more compact compared to that learned by KANs. This indicates that fit- ting the target function requires a much smaller function space, whereas KANs learn a larger and redundant func- tion space, leading memory inefficiency. More detailed analysis please refers Section B.
(a) KANs
(b) MetaKANs
Figure Function fitting on L.2.5 (f(I1 = x122). KANs
learn the formula by (2), - (2= =2)2² +(-21 222)²
4.2. Convolutional Architecture Experiments
2m1 while MetaKANs learn the formula by 322 =
(-I1- 1- (xî + =²) 2x122.
4.2. EXPERIMENTAL SETUP
Extending the framework from (Drokin, 2024), we imple- ment four core modules: KANConv, FastKANCoNv, KAG- NConv, and KALNConv. The baseline architecture contains two configurations::
4.1.2. RESULTS
The comprehensive evaluation in Table demonstrates MetaKANs' significant advantage in balancing parameter efficiency and approximation accuracy. Across 16 Feynman equations, MetaKANs consistently outperform the baseline KANs model under identical grid configurations while us- ing substantially fewer parameters. At G 5, MetaKAN achieves lower MSE in 11 out of 17 functions with train- able parameter reductions ranging from 21% to 47%. A notable example is the 1.12.5 ( = x122), where MetaKANs attains an order-of-magnituue improve- ment MSE (1.16 10-4 vs 1.32 x 10- 10-3) while requiring only 30 parameters compared to 57 parameters of KANs (learned symbolic functions refer to Fig. 3). The results persist higher grid resolutions, with MetaKANs main- taining superior accuracy in 13/16 cases at G = 20 despite using 18% fewer parameters on average. To further illus- trate the insight behind our meta-learner design, we also visualize the parameters' inner dot similiarity matrix of each activation function for KANs and MetaKANs (with
4-layer (shallow) and 8-layer (deep) net- works. Channel progression follows [32, 64, 128, 512] for 4- layer and [2, 64, 128, 512, 1024, 1024, 1024, 1024] for 8- layer architectures, with 3x3 kernels, stride 1, and padding 1. And for 8-layer MetaKANConv and MetaFastK we set the dimension of learnable prompts to and respec- tively and using the multiple meta-learners, The details of clustering meta-learner is stated in Sec. C.4.
The optimization strategy employs three AdamW optimiz- ers: learnable prompts (1 = 10 4) meta-learner (n = 10-3), and main network (17 = 10 10-4). Training details in- clude random horizontal flipping and cropping for CIFAR datasets, exponential decay learning rate scheduling, and Dropout (p 0.2) regularization.
8

---

Improving Memory Efficiency for Training KANs via Meta Learning
nounced for KANs with large grid sizes, such G 80 and above. In these high-resolution scenarios, MetaKANs provide substantial memory savings compared to standard KANs, even when the MetaKANs use larger hidden di- mensions for the meta-learner. This makes it well-suited for developing scalable, high-resolution KANs model. For KANs with medium grid sizes, roughly G = 20 to = 50, MetaKANs can also effectively reduce memory, especially if the meta-learner is configured with small to moderate hidden dimension, like 16 or 32. Similarly, in settings with small grid sizes, such as G 5 to G = 15, memory savings are achievable if meta-learner with a very small hidden dimension (e.g., 16) is employed. Therefore, MetaKANs offer a valuable strategy for making KANs more memory- efficient, particularly for high grid resolution. For practical suggestion, the hidden dimension of the meta-learner should be chosen in consideration of the KAN's grid size.
300
Netatak
250
200
1.5
100
200K
400k
600K
BOCK
1.24
Number Connectioos
(a) Memory usage vs. number of connections
1700
Mou (homde ParsoN Iic 122)
120
4.4. Physical Meaning of Learned Prompt Embedding
Figure 10 visualizes the learned task prompt 2 defined in (q.(1 understand whether Z can distinguish activation functions with different task properties for basis functions. The heatmap (right) shows the similarity of different task prompts 20 a E [ny] [n++1] (left). We can see that learned task prompts for similar activation functions more similar (darker regions in the heatmap) than that for dissimilar activation functions. This suggests that learnable task prompt could identify which activation function it is, validating the rationality for the design of the meta-learner. This further supports that proposed MetaKANs are capable of learning proper weighting learning rules for KANs, and thus improving memory efficiency.
200
(b) Memory usage vs. grid sizes
Figure 4. Memory efficiency of MetaKANs compared KANs. Peak memory allocation (MB) vs. network connections for grid size G = 5 (Top) and vs. grid sizes for hidden dim daidden 64 (Bottom). MetaKANs show substantial memory savings.
4.2.2. EXPERIMENTAL RESULTS
The convolutional architecture comparisons (shown in Ta- ble 3) reveal significant parameter efficiency gains while maintaining competitive accuracy across datasets. For 4- layer models, MetaKANCony achieves 45.86% accuracy on CIFAR-10 with 89% parameter reduction (392,887 vs 3,494.958 parameters), outperforming the bascline by 6.03 percentage points. MetaFastKANNonv demonstrates partic- ularly strong results on CIFAR-100, attaining 34.81 % accu- racy (vs 28.84% baseline) using only 12% of original param- eters (415,539 vs 3,517,610). The 8-layer configurations show enhanced performance scaling, with MetaFastKAN Conv reaching 81.53% on CIFAR-10 while reducing param- eters by 78% (9,051,033 vs 40,695,648).
5. Conclusion
In this paper, we introduced MetaKANs, a memory-efficient framework for training KANs through meta-learning. By leveraging a smaller meta-learner to generate weights for KANs, MetaKANs significantly reduce the number of train- able parameters while maintaining or even improving per- formance. Our experiments across various benchmark tasks, including symbolic regression, partial differential equa- tion solving, and image classification, demonstrate that MetaKANs achieve comparable or superior performance with only a fraction of the parameters required by tradi- tional KANs. Furthermore, MetaKANs are agdelsagnostic, making them applicable wide range of KAN variants, such as FastKAN, ConvKAN, and WavKAN. In conclusion. MetaKANs provide promising direction for improving the training efficiency of KANs, enabling their application to larger datasets and more complex tasks. Besides, how to transfer the weights prediction rules for helping improve memory efficiency of novel KANs training tasks poten- tially valuable future research direction.
4.3. Benefits to Memory Usage
A primary goal of MetaKANs is reducing KAN's high training costs, particularly peak GPU memory consump- tion. Figure 4 shows the peak memory usage of MetaKANs and KANs under different network sizes and grid settings. This demonstrates that MetaKANs are significantly more memory-efficient, especially as the size of network grows larger or grid size increases. MetaKANs scale better and keep memory consumption lower compared to KANs.
Its effectiveness in saving memory is particularly pro-

---

Fang, T., Gao, T., Wang, C., Shang, Y., Chow, W., Chen, Improving Memory Efficiency for Training KANs via Meta Learning
Acknowledgements
In Proceedings of the 34th International Conference on Neural Information Processing Systems, pp. 1877-1901, 2020.
Authors acknowledge the constructive feedback of review- ers and the work of ICML'25 program and area chairs. This work was supported by the National Key Research and Development Program of China (2022YFA 10000000 and in part by the National Natural Science Foundation of China (62476214, 14, 12326606, U24A20324 and 32430017), Tianyuan Fund for Mathematics of the National Natural Science Foundation of China (Grant No. 12426105) and the Major Key Project of PCL under Grant PCI.2024A06.
Cybenko, G. Approximation by superpositions of sig- moidal function. Mathematics of control, signals and systems, 2(4):303-314, 1989.
Drokin, I. Kolmogorov-arrold convolutions: Design
principles and empirical studies. arXiv preprint arXiv:2407.01002, 2024.
Elman, L. Finding structure in time. Cognitive science, 14(2):179-211, 1990.
Impact Statement
In this work, we explore to understand the working mech- anism of KANs and reveal the potential inefficiency of weights learning for KANs. Based on this observation, we improve memory efficiency of training KANs by meta- learning techniques, and achieve almostly the same trainable parameters with MLP network. We believe such attempt will inspire future research toward bridging the gap of train- ing cost between KANs and MLP network. It is meaningful to make KANs easier to learn with limited resources while keeping their interpretability and better performance. How- ever, we note that both of our algorithm and KANs may perform worse on image classification benchmarks than ex- isting baselines. Some explanations for this phenomenon and possible strategis for addressing the issue are worth- while to explore.
L., and Yang, Y. Kaa: Kolmogorov- attention for enhancing attentive graph neural networks. arXiv preprint arXiv:2501.33556 2025.
Garg, S., Tsipras, D., Liang, P. S., and Valiant, G. What can transformers learn in-context? case study of sim- ple function classes. Advances in Neural Information Processing Systems, 35:30583-30598, 2022.
Genet, R. and Inzirillo, H. Tkan: Temporal kolmogorov- arnold networks. arXiv preprint arXiv:2405.07344 2024.
Ha, D., Dai, A. M., and Le, Q. V. Hypernetworks. In International Conference on Learning Representations, 2017.
Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural computation, 1735:1735-17800 1997.
References
Hu, Y., Liang, Z., Yang. F., Hou, Q., Liu, X., and Cheng, classifier for continual learning. arXiv preprint arXiv:2503.21076 2025.
Aghaei, A. A. rkan: Rational kolmogorov-arnold networks. arXiv preprint arXiv:2406. 140000...449.. 14495, 2024.
M.-M. Kac: Kolmogorov
Alaluf, Y., Tov, O., Mokady, R., Gal, R., and Bermano, A. Hyperstyle: Stylegan inversion with hyperntworks for real image editing. In Proceedings of the IEEE/CVF conference on computer Vision and pattern recognition, pp. 1851 1-1221 2022.
Hu, Z., Shukla, K., Karniadakis, G. E.,, and Kawaguchi, K. Tackling the curse of dimensionality with physics- informed neural networks. Neural Networks, 176: 106369, 2024.
Huang, S., Zhao, Z., Li, C., and Bai, L. Timekan: Kan- based frequency decomposition learning architecture for long-term time series forecasting. arXiv preprint arXiv:2502.06110, 2025.
Bodner, A. D., Tepsich, A. S., Spolski, J. N., and Pourteau, S. Convolutional kolmogorov- amood networks. arXiv preprint arXiv:2406. 13155, 2024.
Bozorgasl, Z. and Chen, H. Wav-kan: Wavelet kolmogorov- arnold networks. arXiv preprint arXiv:2405.1288 /2832, 2024.
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Zidek, A., Potapenko, A., et al. Highly accurate protein structure prediction with alphafold. nature, 596(7873) 2021.
Bresson, R., Nikolentzos, G., Panagopoulos, G., Chatzianas- tasis, M. Pang, J., and Vazirgiannis, M. Kagnns: Kolmogorov-arnold networks meet graph learning. arXiv preprint arXiv:2406. 18380, 2024.
Koenig, B. C., Kim, S., and Deng, S. Kan-odes: Kolmogorov-arnold network ordinary differential equa- tions for learning dynamical systems and hidden physics. Computer Methods in Applied Mechanics and Engineer- ing, 432:1 432:4112399 17397, 2024.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are fhw-shot learners.
10

---

Improving Memory Efficiency for Training KANs via Meta Learning
Kolmogorov, A. N. On the representation of continuous functions of many variables by superposition of contin- uous functions of one variable and addition. In Dok- lady Akademii Nauk, volume 14, pp. 953-956. Russian Academy of Sciences, 1957.
Shu, J., Meng, D., and Xu, Z. Learning explicit hyper- parameter prediction function conditioned tasks. Jour- nal of machine learning research, 24(186))1-74, 2023.
SS, S., AR, K., KP, A., et al. Chebyshev polynomial based kolmogorov-arroll networks: An efficient architecture for nonlinear function approximation. arXiv preprint WrXii:2405007200 2024.
Kong, W., Somani, R., Song, Z., Kakade, S., and Oh, S. Meta-learning for mixed linear regression. In Interna- lional Conference on Machine Learning, pp. 5394-5404. PMLR. 2020.
Udrescu, S.-M. and Tegmark, M. Ai feynman: A physics- inspired method for symbolic regression. Science ad- vances, 6 6(16):eaay663, 2020.
LeCun, Y., Boser, B., Denker, J. S.,, Henderson, D., Howard, R. E., Hubbard, W., and Jackel, L. D. Backpropaga- tion applied to handwritten zip code recognition. Neural computation, 1(4):541-551, 1989.
von Oswald, J., Henning, C., Grewe, B. F., and Sacra- mento, Continual learning with hypernetworks. In 8th International Conference on Learning Representa- tions (ICLR 2020) International Conference on Learning Representations, 2020.
Li, C., Liu, X., Li, W. Wang. C., Liu, H., Liu, Y., Chen, Z., and Yuan, Y. U-kan makes strong backbone for medi- cal image segmentation and generation. arXiv preprini MrXiv:2406.02918 2024.
Wang, Y., Siegel, J. W., Liu, Z., and Hou, T. Y. On expressiveness and spectral bias of kans. arXiv preprint arXiv:2410.01803, 2024.
Li, Z. Kolmogorov-arnold networks are radial basis function networks. arXiv preprint arXiv:2405.06721, 2024.
Xu, 7., Shu, J., and Meng, D. Simulating learning methodol- ogy (slem): an approach to machine learning automation. National Science Review, 11(8):mwae277, 2024.
Liu, Z. Wang, Y., Vaidya, S., Ruehle, F., Halver- son, J., Soljačić, M., Hou, T. Y., and Tegmark, M. Kan: Kolmogorov-arnold networks. arXiv preprint arXiv:2404. 19756, 2024.
Yang, X. and Wang, X. Kolmogorov-arnol transformer. arXiv preprint arXiv:2409. 10594, 2024.
Mahabadi, R. K., Ruder, S., Dehghani, M., and Henderson, Parameter-efficien multi-task fine-tuning for transform- ers via shared hypernetworks. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers),
Zeng, S., Zhang, Z., and Zou, Q. Adaptive deep neu- ral networks methods for high-dimensional partial differential equations. Journal of Computational Physics, 463:111232, 2022. ISSN 0021-9991. doi:
URL http:::/www.scleencddeeeeett cciene//Ariiae//////022992222222
pp. 565-576, 2021.
Navon, A., Shamsian, A., Fetaya, E., and Chechik, G. Learn- ing the pareto front with hypernetworss. In International Conference on Learning Representations, 2021.
Zhang, X. and Zhou, H. Generalization bounds and model complexity for kolmogorov-ande networks. arXiv preprint 1rXiv:2410.08006, 2024.
Qiu, X., Yang, M., Ma, X., Li, F., Liang, D., Luo, G., Wang. W., Wang, K., and Li, S. Finding local diffusion schr)," odinger bridge using kolmogorov-arnold network. arXiv preprint arXiv:2502. 19754, 2025.
Zhao, D., Kobayashi, S., Sacramento, J., and von Oswald, J. Meta-lcarning via hypemetworks. In 4th Workshop on Learning at NeurIPS 2020 (MetaLearn 2020). NeurIPS, 2020.
Savarese, P. and Maire, M. Learning implicitly recurrent cnns through parameter sharing. In International Confer- ence on Learning Representations, 2019.
Shu. J., Xu, Z., and Meng, D. Small sample learning in big data era. arXiv preprint arXiv: 045724 2018.
Shu, J., Xie, Q. Yi, L., Zhao, Q., Zhou, S., Xu, Z., and Meng, D. Meta- detaa-eiight-eet. Learning an explicit mapping for sample weighting. Advances in neural information processing systems, 32, 2019.

---

Improving Memory Efficiency for Training KANs via Meta Learning
A. Extension to Other KAN Variants
A.2. FastKAN
A.I. WavKAN
FastKAN (Li, 2024) proposes an efficient approximation method for B-splines of order & with G' intervals by em- ploying Radial Basis Function (RBF) network. An RBF network approximates a target function using weighted sum of radial basis functions centered at specific points. The general form of an RBF network can be expressed as:
In WavKAN, (Bozorgasl & Chen, 2024) propose replac- ing the activation functions in KAN with mother wavelet functions. This modification aims to enhance the model's multi-scale resolution capability, allowing it to effectively capture features across different scales. Additionally, the use of wavelet functions reduces the overall parameter count compared to traditional KAN, as the wavelet basis functions inherently introduce sparsity and compactness into the rep- resentation.
G+
f(x; w) = Σ
wi (||= - cill) = -ww (x)
(18)
where
[w1, WG++| T. and B(x) o (1lz - ......... cG++
Specifically, taking the Mexican hats wavelet function as an example, it follows as:
Gaussian RBF is specifically chosen as the basis function due to its smoothness and universal approximation capabili- ties. The Gaussian RBF defined as:
2
t2
((:o)) π. 71/4//3
-1
exp
(16)
2
<(r) = exp
2h
where a shows the adjustable standard deviation of Gaus- sian, As for each activation function
where r is the radial distance, and is a parameter that controls the width or spread of the function.
By selecting G+ k centers for the Gaussian RBFs, FastKAN efficiently approximates a series of B-spline basis functions, offering both computational efficiency and strong function approximation capabilities. For the consistency of the for- mulation, B-spline basis function in Eq. (2) could be substi- tuted with Eq. (18).
5(I; w) wrb(1- - 14; o)
(17)
where w is a learnable scaling factor controlling the ampli- tude of the wavelet function, and j represents the translation parameter that adjusts the center of the wavelet. The param- eters are compactly represented w [w, μt, o,)"" Thus, each activation function in WavKAN is parameterized by just three learnable parameters w E R3, For the WayKAN, is easily to replace the activation function in Eq. (2) with the (11 w) = wnb(1 - o)
Although FastKAN improves computation efficiency via RBF approximation, it still requires (G + k)x parame- ters count compared with MLP. Thus similar to MetaKAN and Meta WAYKN, we propose using meta-learner, named MetaFastKAN, to generate parameters w in the Eq. 18.
As a result, the parameter count for WavKAN is significantly reduced compared to KAN. Specifi- cally, for structure [no, (M,,,,,,..,,1,, Nr.), WavKAN requires O 33x El=o (n, x m441) parameters, which is approxi- mately 3x the parameter count of standard MLP with the same structure.
A.3. ConyKAN
Kolmogorov-Amool convolutions, introduced by (Bod- ner et al., 2024) and extended by (Drokin, 2024), which replace traditional convolutional activation functions with Kolmogoroy-Aroold based transformations, such as spline (2), RBF (18), or wavelet (17).
Although WavKAN reduces the parameter count compared to KAN, it still introduces additional parameters relative to MLPs, especially when the network structure becomes more complex. To address this, we propose meta-learner, referred to as Meta AANKN, to generate the parameters of WavKAN, further reducing the overall parameter count. Specifically, the meta-learner Ma(z) : R + R gener- ates the three learnable parameters w(2, f) [w, p. o M(2; 0) for each activation function with one learnable scalar z. The meta-learner Me(z) is the same two-layer MLP used MetaKANs. Consequently, MetaWavKAN maintains the same total parameter count as MetaKAN, as described in Sec. (3.2.3).
Unlike conventional convolution kernels, Kolmogorov- Arnold (KA) kernels consist of set of univariate non- linear learnable activation functions. Given an image in- put X E Rexhxw and a KA kernel K E R N the KAA. convolution operation can be expressed as:
Vij =
Da (T.. Wa,.a),
(19)
d=
where & represents a univariate non-linear learnable func- tion with trainable parameters w. This function can take
12

---

Improving Memory Efficiency for Training KANs via Meta Learning
forms such as B-splines (KANConv), wavelet functions (WavKANConv), Gaussian RBFs (FastKAAConyy, de- pending the chosen architecture. However, similar to conventional convolutions, the inclusion of additional pa- rameters w leads to inefficiency in parameter storage.
relatively larger and redundant function class, leading memory inefficiency.
For KANs, it achieves the MSE error 4.58c - 02, while MetaKANs achieve MSE error 3.54e 02. After applying node pruning and symbolic regression as introduced in (Liu et al., 2024), we can formulate the learned symbolic function. For KANs, the result
To address this inefficiency, we propose introducing a meta- learner to in-context learn the function class and generate the parameters. Specifically, the learnable parameters consist of prompts Z for each layer, well as the kernel size and the number of input and output channels.
KAN(x; W) = 0.0031 3g + 0.2892.x4 + 0.6304)
0.0118 (I3 + 0.4825%, 0.3973) 2
For a ConvKAN with L layers structured as [no, P1,...., ...,., ,nLl, we introduce a set of prompts Z for each layer, denoted as:
0.6505 00.550923 0.0147x4
0.3847 (0n90.99 - 7.7876)
2
U [=1
! a (i.j,abb) [md] x [m==1] x [Ki]]
(20)
0.1353 sin22.02662,, 1.5882)
sin (2.281 (in(1.401lr + 1.576)
- -2.2739 sin1144022z - 1.572) + +.59942)
where K2 represents the kernel size of the 1-th layer, and a and b are the kernel indices. These prompts z! are used by the meta-learner to generate the weights w for each convolutional layer. The number of learnable parameters depends the kernel size K₁ and the input-output channel dimensions.
+ 0.6822 + 0.04x²
+ + 8.2417) 0.8938) 2
+ 0.33499smm1.4000.. + 1.576)
0.3478 1.3478sinn1.44042: 1.572)
si62..92n00 7. 7.7876)
Specifically, for each layer, the weight generation process is formulated as:
0.2155 0.2155 1.5882)
+ 1.322sin(-0.0044 + 0.04x²
L-1 -Moc!" -M(oo) a E J
= ( 1=0
(21)
0.1144r3 + 0.06392, + 7.8417) - 0.4717,
where Z = [m]] x [++1] x [K]], 1 E 14]4-11 The
-
while the result of MetaKAN is
hypernetwork Mo maps the embedding scalars (1) Rijjaa,b to the
corresponding convolution kernel parameters W.j.a,b"
MetaKAN(x; W = 1.019 sin
0.5264
As discussed in Section 3.3, the parameter count of Meta- ConvKAN can be reduced by factor of //dimw) com- pared to KANConv, resulting in a parameter efficiency ap- proximately equivalent that of standard convolutional networks. If we set the dimension of the from to dim(2), ), the parameter count can be reduced by factor of dim(z)/dim((( compared to KANConv.
(3.1711 STA)²
+ 3.175500.0000 2zz)² ) 12.5921)
+ 0.525 sin (3.159700.0004 - x22)
+ 3.1578(0.0005 - x1)² 12.5819)
- 9.8227 1.8769.
B. Illustration of the Learned Function Class
We train KANs and MetaKANs (with structure [4, 5, 5, 1) respectively on fitting the function f(x) exp ( (sin (IT (af + +z)) sin (i (23 + +2)))) and the parameter inner dot similarity matrices are represented in Figure 9. From the figure, we observe that the function class learned by MetaKANs is significantly more compact compared to that learned by KANs. This indicates that MetaKANs require a smaller set of function class to fit the target function, whereas KANs learn a
The results demonstrate that MetaKAN not only achieves better accuracy but also learns a more compact and inter- pretable representation of the target function. Specifically, MetaK AN's formula is significantly simpler, with fewer terms and a clearer structure. This highlights MetaKAN's advantage in learning concise and generalizable representa- tions, which is crucial for both interpretability and memory efficiency.
13

---

Improving Memory Efficiency for Training KANs via Meta Learning
Table 4. Parameter counts and MSE for KANS and MetaKANs at various dimensions for three different functions.
FUNCTION
f1(x)= exp ( Σ sin
(τ)
fz(x) = Σ + x
fa(x) = exp -
DIMENSION
KAN
METAKAN
KAN
METAKAN
KAN
METAKAN
STRUCTURE [N.1.1] # PARAM
MSE
PARAM
MSE
# PARAM
MSE
PARAM
MSE
# PARAM
MSE
PARAM
MSE
10
101
5.92F-4
97
5.85k-4
191
3.11F-3
119
7.06E-4
101
1.47k-4
97
3.39F-4
191
4.63E-4
108
4.97E-4
371
2.63E-3
141
2.26E-4
191
1.11E-5
3.42E-5
50
461
5.52E-4
130
1.40E-3
911
NA
185
1.15E-3
461
3.46E-4
174
3.19E-4
100
911
7.00E-3
229
1.45E-3
1811
1.74
361
1.16E-2
911
5.12E-4
229
6.47E-4
500
4,511
2.46E-1
339
4.62E-3
9011
1.83E+2
889
5.52E-2
4,511
3.69E-2
669
8.838-3
1000
9,011
4.14E-1
713
1.48E-2
18,011
1.68E+2
1329
1.43E-1
9,011
9.70E-2
2,759
3.94E-3
C. Extended Experiments
Mecanin 19541
C.1. High Dimensional Function
cAN
C.1.1. EXPERIMENT SETUP
In this experiment, we evaluated the performance of the KANs and MetaKANs models on three different high- dimensional functions. The target functions were defined as follows:
f((x)= exp -
Σ sin
exxp
2
f2(x) = Σ x² x²
Figure 5. Function fi Accuracy and trainable parameter count under different dimension
fa(x) exp (-1 Σ
Training Test Comparison
2.2
PacuOA (Fai)
For each function, the models were evaluated at six dif- ferent dimensions: n 10, 20, 50, 100, 500, 1000. Ex- periments span dimensions
2.30
10 to 1000 using min- imal [n, 1, 1] architecture. For MetaKAN, the hypernet- work's hidden layer width systematically explored from 2 to 64 neurons through grid search, with optimal widths selected as {8, 16, 32, 32, 64, 64) corresponding to dimensions {10, 20, 50, 100, 500, 1000} respectively. This controlled expansion ensures the hypernetwork's capacity grows proportionally with input dimension while maintain- ing parameter efficiency.
Epach
Figure 6. Training and test Loss comparison for KANs and MetaKANs on SVHN dataset (G
The training conducted using the LBFGS optimizer, and the results were measured in terms of parameter count and MSE. The tables display the parameter counts and corre- sponding MSE values for both KAN and MetaKAN models across all dimensions for each function.
vs 9,011). This efficiency advantage amplifies with dimen- sionality for the radial function f3. MetaKAN's parameter count grows 28 x from 10D to 1000D (97 to 2,759) versus KAN's 89x x increase (101 to 9,011), enabling MetaKAN to limit error growth to 26.8x compared to KAN's 658x MSE escalation. Critical limitations emerge for KAN: failure converge for f2 50 dimensions, and error amplification 1.68 10 MSE 1000D with 18,011 parameters, versus MetaKAN's stable 1.43 10-1 MSE using 1,329 parame- ters. These patterns are visually corroborated in Figure 5, demonstrating MetaKANs consistently perform better on different dimension settings while require less parameter count than KANs.
1.2. RESULTS
Table 4 demonstrates MetaKAN's superiority in high- dimensional function approximation across three key pects. Across all function types and dimensions, MetaKAN demonstrates superior accuracy with significantly fewer rameters. For the cubic function f2 at 500 dimensions, MetaKAN achieve a 330x lower MSE (5.52 x 10-2 vs 1.83 102) while using only 10% of KAN's parameters (889
14

---

Improving Memory Efficiency for Training KANs via Meta Learning
Table 5. Classification accuracies and parameter counts for KAN and MetaKAN models across different settings.
G=3
G=5
G=10
DATASET
HIDDEN
KAN
METAKAN
KAN
METAKAN
KAN
METAKAN
PARAM
ACC.
PARAM
Acc.
# PARAM
Acc.
PARAM
ACC.
# PARAM
ACC.
# PARAM
Acc.
SVHN
32
529.130
51.16
77,376
78.34
680,298
52.35
78,144
75.35
831.466
59.67
79,424
63.46
FMNIST
32
177.898
84.59
27,456
87.28
228,714
85.59
27,968
87.73
355.754
85.65
29,248
86.7
KMNIST
32
177,898
77.64
27,456
83.93
228,714
78.28
27,968
82.82
355,754
73.52
29,248
73.22
MNIST
32
177,898
94.11
27,456
96.3
228,714
93.60
27,968
96.69
355,754
92.76
29,248
95.43
CI
32
529,130
39.36
77,632
47.98
680,298
36.75
78.144
49.67
1.058,218
39.86
79,424
48.36
CIFAR-100
32
549,380
4.45
80,512
21.09
706,308
8.75
81,024
20.85
1,098,628
5.66
82,304
19.35
SVHN
32,32
536,330
60.31
78,656
78.10
689,546
62.54
79,168
77.01
1,072,586
58.71
80,448
59.51
FMNIST
32,32
185,098
85.91
28,480
87.6
237,962
86.12
28,992
86.16
370,122
86.73
30,272
85.81s
KMNIST
32,32
185,098
80.46
28,480
84.26
237,962
80.23
28,992
83.44
370,122
81.44
30,272
80.07
MNIST
32,32
185,098
93.80
28,480
95.91
237,962
91.22
28,992
94.43
370,122
93.32
30,272
94.87
CIFAR-10 10
32,32
536,330
42.16
78,656
49.75
689,546
47.15
79,168
49.4
1,072,586
43.79
80,448
44.6
CIFAR 100
32,32
556,580
2.95
81,536
20.49
715,556
8.8
82,048
14.42
1,112,996
6.48
83,328
18.23
C.2.2. RESULTS
The comparative results across different architectures reveal consistent patterns in parameter efficiency and performance gains. For standard KANs (Table 5), MetaKANs achieve 85 ~ 94% parameter reduction while maintaining compet- itive accuracy, particularly evident in SVHN (78.34% vs 51.16% at G 3) and FMNIST (87 28% vs 84.59%). We visualize the training procedure the SVHN (Figure 6) and the t-SNE visualization (Figure 7).
(a) KAN
(b) MetaKAN
Figure 7. T-SNE visualization on the SVHN test datset. The
MetaKAN performs clearer classification boundaries
The dual-hidden-layer configuration (32,32) demonstrates improved performance on complex datasets, with MetaKAN reaching 49.75% accuracy on CIFAR-10 compared to 42.16% 16% for baseline KAN G 33
C.2. Fully Connected Architecture Experiments
C.2.1. EXPERIMENTAL. SETUP
The FastKAN experiments (Table 6) demonstrate MetaFastKAN's parameter efficiency with 84-92% reduction across configurations while maintaining or improving accuracy. For CIFAR-100 with dual hidden layers (32,32), MetaFastKAN achieves 23.17% accuracy at c = 13 using only 8.3% baseline parameters (92,565 vs I, 1,117,867). The architecture shows particular strength in simpler datasets, attaining 96.31% 31% accuracy on MNIST (c 6) versus baseline's 94.99% with 15.7% parameter load. Notably, the dual-hidden- configuration achieves superior results on complex tasks, improving SVHN accuracy to 79.52% at while maintaining 91.7% parameter reduction. Shallow architectures (single hidden layer) preserve competitive efficiency, reaching 88, 88.29% accuracy on FMNIST (c 6) with 16% of original parameters.
We systematically evaluated three base architectures (KAN, WavKAN, and FastKAN) along with their meta-learning variants. The network architecture employs single-hidden- layer (32 neurons) and double-hiddee hiable--hddee-llyeerr (32x32 neu- rons) configurations. For activation function configura- tion: KAN uses B-spline basis functions with grid points WavKAN
G (3.5, 5, 10} and polynomial order k adopts three mother wavelet functions (Mexican Hat, Mor- let, and Derivative of Gaussian); FastKAN sets the number of centers E {,, 8, 13} align with the B-spline basis count in KAN.
13 (vs 75.51% baseline)
The meta-learner design involves hidden layer dimensions selected from {32, 64, 128) through grid search. Out- put dimension adaptation follows specific rules: for KAN, dout (G+ k 1); for WavKAN, dout corresponding to the three wavelet bases; for fastkan, dous c 1. To training MetaKANs. we use separate Adam optimizers for the meta-learner M (7) E {10-4, 10 10-3)) and learn- able prompts (722 E (10-3, 10 10-2)). Training configuration employs cosine annealing for learning rate scheduling and batch size 128.
baselet-based architectures (Table 7) demonstrate mother function-dependent performance, with Mexican Hat wavelet achieving 73.62% accuracy on SVHN compared to 66.1%% for DoG wavelet in MetaWavKAN The dual-hidden-llyer configuration improves generalization on CIFAR- 10, where Meta WavKAN reaches 49.52% accuracy versus 46.34% in
15

---

Improving Memory Efficiency for Training KANs via Meta Learning
Table 6. Classification accuracies and parameter counts for FastKAN and MetaFastKAN models across different settings.
C=6
C=8
C=13
DATASET
HIDDEN
FASTKAN
METAFASTKAN
FASTKAN
METAFASTKAN
FASTKAN
METAFASTKAN
PARAM
ACC.
PARAM
ACc.
PARAM
ACC.
PARAM
ACC.
PARAM
ACC.
PARAM
ACC.
SVHN
32
533,910
72.94
81.523
77.00
685.082
66.19
81,081
73.19
1,063,012
70.24
82,440
70.27
FMNIST
32
179,542
86.64
28,211
88.29
230.362
87.41
29,881
87.93
357,412
88.9
35,272
88.44
KMNIST
32
179,542
75.99
28,211
81.67
230,362
80.38
29,881
81.02
357,412
82.87
35,272
84.22
MNIST
32
179,542
92.98
28,211
96.29
230,362
95.16
27,769
95.31
357,412
94.82
29,128
95.72
CI
32
533,910
47.99
82,675
51.73
685,082
47.05
83.193
50.77
1,063,012
50.57
81,523
50.27
CIFAR-100
32
554,160
22.28
87,859
23.19
711,092
22.36
88,889
23.5
1,103,422
21.56
87,368
22.26
SVHN
32,32
541,180
72.43
82,617
77.20
694,402
71.25
82,881
74.69
1,077,457
75.51
89,685
79.52
FMNIST
32,32
186,812
87.13
32,761
88.13
239,682
88.71
29,569
87.92
371,857
88.9
30,229
87.96
KMNIST
32,32
186,812
82.62
29,305
82.96
239,682
85.82
33,793
85.46
371,857
85.04
36,373
84.9
MNIST
32,32
186,812
94.99
29,305
96.31
239.682
96.37
29,569
95.99
371,857
95.47
30,229
96.37
CIFAR-10
32,32
541,180
48.78
83,769
51.55
694,402
48.32
87,105
49.38
1,077,457
50.57
89,685
51.16
CIFAR 100 32,32
561,430
18.5
86.649
22.6
720.412
17.87
89,985
22.47
1,117,867
21.08
92,565
23.17
Table 7. Classification accuracies and parameter counts for WavKAN and MetaWayKAN models across different settings.
DoG WAVELET
MORLET WAVELET
MEXICAN HAT WAVELET
DATASET
HIDDEN
WAVKAN
METAWAVKAN
WAYKAN
METAWAVKAN
WAvKAN
METAWAVKAN
# PARAM
ACc.
# PARAM
Acc.
PARAM
Acc.
PARAM
Acc.
PARAM
Acc.
# PARAM
ACC.
SVHN
32
226,836
45.65
76,951
66.13
226,836
19.08
76,311
25.69
226,836
67.05
76,951
73.62
FMNIST
32
76,308
83.86
26.775
87.66
76.308
83.11
26.135
82.1
76.308
87.13
25,815
87.23
KMNIST
32
76,308
77.61
26,775
78.76
76,308
77.08
25,815
79.98
76,308
83.37
25,731
82.17
MNIST
32
76,308
94.58
26,775
95.3
76,308
93.14
25,815
96.17
76,308
96.78
30,531
95.9
CIFAR-10
32
226,836
33.33
76,951
46.34
226,836
13.48
76,311
23.11
226,836
47.17
75.991
48.09
CIFAR-100
32
235,656
11.11
80,011
20.96
235,656
1.5
79,371
5.71
235,656
20.51
83,587
21.27
SVHN
32,32
306,580
46.27
78,039
69.42
229,972
19.47
77.079
55.20
229,972
60.83
81,731
76.55
FMNIST
32,32
79,444
83.72
27,863
84.73
79,444
76.34
27,223
80.02
79,444
86.52
27,863
86.81
KMNIST
32,32
79,444
79.87
27,863
80.63
79,444
28.44
26,903
82.89
79,444
84.59
31,555
84.11
MNIST
32,32
79,444
96.44
36,678
96.74
79,444
26.25
26.903
96.8
79,444
96.65
36,678
96.74
CIFAR-10
32.32
229.972
49.12
78.039
49.19
229.972
10.67
77,399
26.94
229.972
44.81
77.399
49.52
CIFAR- 100
32.32
238.792
21.86
84,611
21.97
238,792
1.09
80,459
4.08
238,792
16.4
81,099
20.87
single-hidden-laye setups. Parameter compression remains effective across wavelet types, maintaining 87-91 % reduc- tion while preserving accuracy margins.
where NO, 1), and x E Bd (the unit ball).
Next, we define the three PDEs.
Poisson Equation: The first PDE is the Poisson equation, defined as:
archttare analysis reveals consistent in-context learning advantages: 1) Average parameter savings of 89%+4 across all variants; 2) Accuracy improvements of 3.2-14.6 percentage points on small-to-large datasets. These outcomes validate meta-learner's 's capability to achieve ef- fective balance between parameter reduction and accuracy maintenance across varied basis function implementations (B-spline basis, wavelet and RBF) and architectural depths.
(n(x)=g(2)) 2 E
(22)
where g(x) Allessa
Allen-Cahn Equation: The second PDE is the Allen-Cahn equation, defined as:
Au(x) + u(xx u(x)3 g(x), T Bd
(23)
C.3. Solving PDEs
where g(x) Auexnet (2) Uexoct Mexect (z)
C. I. EXPERIMENTAL SETUP
Sine-Gordon Equation: The third PDE is the Sine-Gordon equation, defined as:
We first introduce the exact solution for the PDEs following the (Hu et al., 2024):
Au(x) + sin(u(x)) = g(x), a x Bd,
(24)
Mexac(()) (2)) = (1 ||x1))
Σ
cos(x;;)
where g(x) A ) sin(Uexxc(
16

---

Improving Memory Efficiency for Training KANs via Meta Learning
Table 8. Relative & error and parameters for KANs and MetaKANs at different dimensions.
DIM
POISSON
ALLEN-CAHN
SINE-GORDON
KAN
METAKAN
KAN
METAKAN
KAN
METAKAN
MSE
PARAM
MSE
PARAM
MSE
PARAM
MSE
PARAM
MSE
PARAM
MSE
PARAM
20
5.24E-4
24,480
2.96E-3
4.137
1.36E-3
24.480
3.66E-3
4.137
2.66E-1
24,480
2.69E-1
4,137
50 D
2.14E-3
33.120
4.85E-3
5,097
2.36E-3
33.120
6.10E-3
5,097
6.08E-2
33,120
5.88E-2
5,097
100 D
3.77E-3
47,520
4.59E-3
6,697
6.53E-3
47,520
3.37E-3
47,520
1.91E-2
47,520
2.60B-2
6,697
Table Ablation study of embedding dimensions and the number of meta-leamer (C) CIFAR- 100 dataset.
METAKAN
C=1
C=3
C=5
C=7
DIM
ACC
#PARAM
ACC
#PARAM
ACC
#PARAM
ACC
#PARAM
29.90= 7.27
4,578,565
4,584,215 4.30
37.75+ 2.01
4,579,305
38.00 3.60
4,580,731
40.40
9,097,397
34.58 6.02
9,097,799
39.34
9,098,585
44.17
9,099,371
44.13 + 2.33
18,139,285
44.37
18,139,751
41.85+ 5.53
18,140,665
42.04 + 7.87
18,141,579
METAFASTKAN
C=1
C=3
C=5
C=7
DIM
ACC
#PARAM
ACC
#PARAM
ACC
#PARAM
ACC
#PARAM
40.44 7.68
1.5
37.56+ 1.48
4,575,861
46.05 0.41
4,576,583
44.16+ 1.01
4,576,073
2
38.62 1.93
9.096.051
3..0
9,096,837
46.101 1.85
9,097.623
49.23
9,098.409
42.09 6.33
18,137,875
42.38 1.53
18,138,789
49.03 1.52
18,139,703
52.02 1.07
18,140,617
These three PDEs are solved using the KAN and MetaKAN models. We apply the exact solution Mexact (x) for the exper- iments, where the boundary conditions are set to u = 0 on dB
Both KAN and MetaKAN models have the structure [n, 32,32,32,1J. During training, the same number of initial points, boundary points, and interior points are used.
(a) KANs
(b) MetaKANs
Figure 8. The relative & point-wise error visualization for Allen- Cahn equation (d = 100).
We use a physics-informet loss function based on PINNs, formulated as:
competitive accuracy while significantly reducing the train- able parameter count. The relative l error visualization for Allen-Cau equation (d 100) is shown in Figure 8.
losspde = aloss + loSS
where loss represents the residual loss at interior points, which discretized and evaluated at N; sampled points. Similarly, losSand denotes the boundary constraint loss, which is discretized and evaluated No sampled points. The weighting factor a is used to balance the contributions of these two losses in the overall loss function. Following (Zeng et al., 2022), N; is set to 2000, 4000, 8000, 12000 for different dimensions d e {20, 50, 100}. while N, is set to 100 points per boundary, resulting in a total of 100d boundary points. a is 0.01.
C.4. Ablation Study
We conduct ablation study 8-layer MetaKANConv and MetaFastK architectures systematically analyze the effects of prompt dimensions and meta-leamner configu- ration on CIFAR-100 100 dataset. Based on channel dimension patterns in different layers, we cluster the 8 layers into C= 1, 5, groups for meta-learner assignment with Al- gorithm2.
Table 9 demonstrates that increasing dim generally en- hances accuracy, with optimal gains dim 2=4 for both architectures. While larger dim increases parameters (c.g., +13.5M from dim 2=1 to 4 at N=l), the accuracy improvements justify this tradeoff. More significantly, ex- panding the number of meta-learners (C) yiclds substantial performance gains across both architectures. For MetaKAN,
C.3.2. EXPERIMENTAL RESULTS
Table 8 summarizes the performance of the KANs and MetaKANS for solving the Poisson, Allen-Cahn, and Sine- Gordon equations at different dimensions. The table reports the relative le eтror and parameter count for various dimen- sions. These results suggest that MetaKANg can maintain
17

---

Improving Memory Efficiency for Training KANs via Meta Learning
increasing C from 1 to dim z=4 improves accuracy from 43%% 44.37%, while at dim 2=2, scaling C from 1 to delivers even more pronounced gains (40.40% to 44.17%). The performance progression is particularly re- markable for MetaFastKAN: at dim 2=4, accuracy increases steadily from 42.09% (C=1) to 52.02% (C=7), representing a 23.6% relative improvement. This consistent positive cor- relation between meta-leamer count and accuracy highlights the importance of specialized meta-leamer for distinct layer clusters.
These results validate our layer clustering strategy in Sec. 3.2.3, demonstrating that task-specific meta-learners (C > 1) with sufficient prompt dimensions consistently outperform single meta-learner baselines. The observed per- formance scaling with C' underscores the effectiveness of grouped meta-learner approach, where dedicated meta- learner for different layer clusters progressively enhance model capacity. MetaFastKAN in particular shows near- linear improvement with increasing C, achieving its optimal configuration at C=7 and dim 2=4 with 52.02% accuracy, while MetaKAN reaches peak performance at C=3 and dim 2=4 (44.37%). The minimal parameter overhead (typically :0.01 %) during this C' scaling further confirms the efficiency of our approach.
18

---

Improving Memory Efficiency for Training KANs via Meta Learning
3.75
920
-143
(a) KAN
25
020
22222222
035
-22
am c ai aee oy loker Jew Agl a 1
1000
(b) MetaKAN
Figure 9. Comparion of (a) KANs and (b) MetaKANs in terms of learned activation functions (left) and cosine similarity of learned spline coefficient vectors (Right). They are learned for KANs with the structure [4,5,5,1] trained fit the function f(x) = exp (2 (sin (x (af + x3)) + sin (xt (x3 + 2)) The LHS shows the actual shapes of activation functions learned by KANs and MetaKANs, visualized from the weights of KANS learned by original KANs and predicted by the trained meta- heaa-le MetaKANs. The RHS displays the cosine similarity between these generated spline coefficient vectors reveal structural similarities in learned function shapes. The different color (positive sign or negative sign) can be used to group the activation functions into different classes. We can that MetaKANs could extract more compact structure of the learned function class spaned by fewer B-spline basis functions, while KANs leam a relatively redundant function class.
19

---

Improving Memory Efficiency for Training KANs via Meta Learning
+09
(2)
.01
.00
Figure 10. Visualization of learned task prompt embedding similarities. Left: Schematic of the hierarchical model structure with illustrative function shapes at different layers. Right: Heatmap depicting pairwise absolute distances between the learned scalar embeddings (labeled =o a E [ru] x [m)+1] corresponding to the univariate functions. Darker regions indicate smaller distances (higher similarity in embedding values). KANs fit the function (x) exp (1 (sin (I (2f + =3)) sin ( (xž + =z)))) with structure [4. 2, 1].
20


Reviews:
[{"review_id": "ucjSq8MKQy", "reviewer": "Reviewer_vpTT", "summary": "This paper concerns a meta-learning approach to training Kolmogorov Arnold Networks (KANs) that enables a reduction in the number of trainable parameters in KANs that is substantially larger than that of standard deep learning models like MLPs. The meta-learning approach is quite standard; it proceeds by assuming that part of the trainable parameters are outputs to a hypernetwork. This output of the hypernetwork then couples together the activation-function-related parameters, significantly reducing the number of parameters (cubic to quadratic). Numerical benchmark illustrates that the thus introduced MetaKAN still achieves good performance while having significantly fewer parameters.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "C1:Why use the MLP as the meta-learner?\n\nA1: We understand the reviewer's interest in the theoretical motivation and deeper implications of employing an MLP as the hypernetwork, particularly in comparison to works like Low-Rank PINNs where architectural choices may have stronger connections to the underlying problem structure (e.g., conservation laws).  We acknowledge that our paper could expand more on this aspect. Our choice of an MLP was primarily driven by several considerations: Firstly, as a universal function approximator, the MLP has been widely adopted in the meta-learning field to learn the common task distribution, such as [1-3] . Secondly, the MLP architecture is relatively simple, computationally efficient, and straightforward to implement and train.\n\n[1] HyperNetworks, ICLR 2017.\n\n[2] Meta-weight-net, NeurIPS, 2019.\n\n[3] Continual learning with hypernetworks, ICLR, 2020.\n\nC2:Clarify Figure 3\n\nA2: We apologize for any lack of clarity in Figure 3. To clarify: the LHS does show the actual shapes of activation functions learned by MetaKAN, visualized from the weights generated by the trained hypernetwork. The RHS displays the cosine similarity between these generated spline coefficient vectors to reveal structural similarities in the learned function shapes. We could utilize the different color (positive sign or negative sign) to group the activation functions into different classes and see the compact structure of the learned function family with only a few function classes.\n\nC3:How small can you make the hypernetwork? How does the architecture constrain the weights? What is the extent it can be reduced?\n\nA3: We thank the reviewer for these insightful questions regarding the hypernetwork's design and its impact. These three aspects – minimum size, architectural constraints, and the extent of reduction – are closely related and central to the MetaKAN approach.\n\nThe smallest effective hypernetwork is task-dependent. On simpler tasks, small hidden dimensions (e.g., 4–16) suffice. More complex tasks typically need larger dimensions (e.g., 32–128) to maintain expressivity. The choice reflects finding an point on the efficiency-performance trade-off for the specific application.\n\nThe hypernetwork architecture  constrains the generated weights by acting as a shared meta-learner for all activation function coefficients. Instead of learning separate parameters, MetaKAN learns this shared rule  and a unique, low-dimensional embedding Z for each function, as mentioned in our approach. The hypernetwork takes this small embedding Z as input and uses the learned rule to generate the much larger set of spline coefficients (W). Because all coefficients originate from the common function family. The hypernetwork's  hidden dimension restricting the variety and complexity of coefficients that can be produced from the input embedding Z. \n\nThe extent of parameter reduction achievable by MetaKAN primarily scales with the ratio of the chosen embedding dimension  to the original spline coefficient dimension, as the total parameters become dominated by the embeddings in larger networks. As illustrated in table, for small networks (e.g., [4,2,2,1]), the hypernetwork's own size , related to hidden dimension significantly impacts the total count, making a compact hypernetwork essential for savings. However, for larger networks, increasing hidden dimension(from 32 to 64) adds negligible parameters (0.1% increase) compared to the vast overall reduction (89%) achieved relative to KAN. Thus, while hypernetwork size matters for small models, the principal reduction factor in practical, larger-scale scenarios is determined by the embedding dimension.\n\n| Structure   | Model                | # Param |\n| ----------- | -------------------- | ------- |\n| [4,2,2,1]   | KAN                  | 117     |\n|             | MetaKAN, d_hidden=4  | 58      |\n|             | MetaKAN, d_hidden=16 | 166     |\n| [784,32,10] | KAN                  | 228,672 |\n|             | MetaKAN, d_hidden=32 | 25,705  |\n|             | MetaKAN, d_hidden=64 | 25,993  |\n\n\nC4:Improvement with KAN coefficient reduction.\n\nA4: We strongly agree that combining MetaKAN with other KAN coefficient reduction techniques holds significant potential. Our generative approach (reducing learned parameters) is largely orthogonal to methods like pruning, quantization, or coefficient sharing  (reducing final coefficient complexity or count). Integrating these strategies could lead to further efficiency gains and is a promising avenue for future research.\n\nC5:Physical meaning of latent parameters Z.\n\nA5: In MetaKAN, the Z embeddings identify the specific activation functions needed per edge. These embeddings organize meaningfully in latent space: nearby Z values produce similar activation shapes. This was verified by comparing similarity matrices of Z and the generated coefficients, which showed strong alignment (as in Figure 3). Thus, Z effectively encodes functional similarity.", "claims_and_evidence": "The authors claim that the MetaKAN requires fewer parameters while achieving similar accuracy.", "methods_and_evaluation_criteria": "The numerical test are standard benchmarks that allow a straightforward comparison with other KANs (INR-type tasks plus standard ML classification tasks). The setup is reasonable and the results look convincing.", "theoretical_claims": "I noticed no substantial theoretical claims, other than parameter counts which appears correct.", "experimental_designs_or_analyses": "The experiments are standard, I did not notice any serious issues.", "relation_to_broader_scientific_literature": "The paper aligns well with recent works in literature attempting to find low-dimensional structures within DL architecture via meta-learning. I believe this is one of the first papers attempting so for KANs, as far as I am aware.", "essential_references_not_discussed": "More broadly, especially regarding KANs' use in solving PDEs or representing solutions to PDEs, there are other meta-learning approaches that the author can mention. For example, there are well-known papers in the field,\n\nMichael Penwarden, Shandian Zhe, Akil Narayan, Robert M. Kirby, \"Physics-Informed Neural Networks (PINNs) for Parameterized PDEs: A Metalearning Approach\"\n\nShaowu Pan, Steven L. Brunton, J. Nathan Kutz, \"Neural implicit flow: a mesh-agnostic dimensionality reduction paradigm of spatio-temporal data\"\n\nWoojin Cho, Kookjin Lee, Donsub Rim, Noseong Park, \"Hypernetwork-based Meta-Learning for Low-Rank Physics-Informed Neural Networks\"", "other_strengths_and_weaknesses": "Strengths\n\nThe paper is well-written and propounds the major points concisely and clearly. The tests seem reasonable.\n\nWeaknesses\n\nIn this reviewer's point of view, the paper does not expand sufficiently upon the implications of the specific meta-learning set up. I understand some of this is due to heuristics (ie.. hindsight; some set up works, some not) but are there any settings where the use of MLP arises naturally? For example, Low-Rank PINNs meta-learning approach has some theoretical connections to conservation laws (D. Rim, G. Welper, \"Low Rank Neural Representation of Entropy Solutions.\"", "other_comments_or_suggestions": "Some suggestions:\n\n- Figure 3, which takes up a significant space, was hard to decipher for this reviewer. Does the diagram on the LHS shows the actual activation functions that were learned? Or are they simply for diagram representation? The cosine similarity on the RHS also appears puzzling... if the MLP pre-output hidden-state is low dimensional, it would already be clear that the output parameters feeding into the activations will also be low-rank. I suggest showing the amplitude. Does the sign in the cosine similarity matter here? If not, perhaps showing its magnitude in log-color scale will be better?", "questions_for_authors": "- How small can you make the hypernetwork? How does the architecture constrain the weights? What is the extent it can be reduced?\n\n- Is there potential in jointly reducing the coefficients as well as alongside this MetaKAN approach?\n\n- Does the latent parameters Z represent anything physical upon training?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2uyblg9YU1", "reviewer": "Reviewer_A7wa", "summary": "The paper proposes a novel memory optimization method for Kolmogorov-Arnold Networks known as MetaKAN.  MetaKAN leverages the use of meta-learners—2-layer neural networks that generate B-spline coefficients on-the-fly—to reduce the parameter count of KANs to a level comparable to that of MLP’s.  Across various experiments, ranging from simple function fitting to deep learning tasks, MetaKAN consistently achieves an average parameter count reduction relative to KAN ranging from 18% to nearly 90%, while maintaining and often exceeding base KAN performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "C1: Complexity comparison and Training time between KAN and MetaKAN.\n\nA1: We thank the reviewer for the insightful question. Below is a summary of theoretical complexity\n1. Complexity Analysis\nnotations see https://github.com/icmlrebuttal25/Append\n\n| Model   | Total FLOPs |\n| ------- | ----------------------------------------- |\n| KAN     | $F_{act} N_{nodes} + N_{edges} C_{spline} + 2N_{node}(K^2 + GK) + 3N_{edges}D_{spline}$ |\n| MetaKAN | $F_{act} N_{nodes} + N_{edges} C_{spline}$ + $N_{edges} C_{meta}$ + $(2N_{edges} + 4d_{hidden}) D_{spline}$ |\n\n\n\n2.  Advantage condition\n\nMetaKAN becomes cheaper when $N_{edges} > 4d_{hidden}$, typically when $N_{edges} \\sim 10^4$ or higher. This is due to trading forward-pass overhead for lower backward complexity. In practice, with >4 layers and >500 neurons per layer ($N_{edges} \\sim 10^6$), MetaKAN training time becomes comparable to KAN. For deeper networks (8+ layers, width >1000), MetaKAN can be faster due to backward efficiency gains.\n\n3. Empirical Comparison\n\n| HiddenWidth x depth| Model   | # Trainable Params | Train Time(s) | Peak Mem.(MB) |\n| ------------ | ------- | ------------------ | ------------- | ------------- |\n| 1024×8       | KAN     | 211M               | 5396          | 7508          |\n| 1024×8       | MetaKAN | 8M                 | 5161          | 2519          |\n| 512×6        | KAN     | 45M                | 1399          | 4108          |\n| 512×6        | MetaKAN | 1.7M               | 1435          | 1212          |\n| 512×4        | KAN     | 31M                | 1006          | 2854          |\n| 512×4        | MetaKAN | 1.2M               | 1060          | 931           |\n\n4. Potential\n\nThese results align well with our theoretical FLOPs analysis. Although MetaKAN introduces a modest overhead in the forward pass due to the hypernetwork, it significantly reduces the cost of the backward pass—a major bottleneck when training large models.  Furthermore, MetaKAN achieves this with drastically fewer trainable parameters and significantly lower memory usage. These advantages indicate that MetaKAN is not only efficient in current setups but also holds strong potential for scaling up to even larger KAN models—scenarios where traditional KANs may become impractical due to memory and gradient computation overhead.\n\n\nC2: Performance and training time comparison between MLP and MetaKAN\nA2:\n\n1. For deeper structure, the theoretical parameter count of MetaKAN scales mainly with the number of edges and the number of meta-learners leads to a few parameter count increasing.\n\n| Name                            | # Params.                                                    |\n| ------------------------------- | ------------------------------------------------------------ |\n| MLP                             | $N_{edges}+\\sum_{l=1}^L n_l$     |\n| KAN                             | $N_{edges} \\times (G + k + 1)$   |\n| MetaKAN(Single meta-learner)    | $N_{edges} + (d_{\\text{hidden}} + 1) \\times (G + k + 1)$ |\n| MetaKAN(Multiple meta-learners) | $N_{edges} + C(d_{\\text{hidden}} + 1) \\times (G + k + 1)$ |\n\n2. We compare the two models on two tasks: function fitting and image classification.\n\nOn the function fitting task $f_1(x)=\\exp\\left(\\sum_n(1/n)\\sin^2(\\pi x/2)\\right)$ with $n=1000$, MLP trains faster, but its accuracy is significantly lower. MetaKAN retains the inductive bias of spline-based models, achieving both compactness and high accuracy, making it ideal for tasks requiring interpretable and function representations.\n\nFor image classification (MNIST), MLPs again train faster and achieve slightly better accuracy. However, MetaKAN still maintains strong performance with substantially fewer parameters. \n\n| Task                 | Model   | Structure          | # Params | Training Time(s) | Metric | Peak Memory(MB) |\n| -------------------- | ------- | ------------------ | -------- | ---------------- | ------ | --------------- |\n| Function Fitting     | MLP     | (1000,3000,3000,1) | 12.7M    | 2.53             | 0.7900 | 193.89          |\n|                      | MetaKAN | (1000,512,512,1)   | 1.6M     | 12.37            | 0.0116 | 372.35          |\n| MNIST Classification | MLP     | (784,16000,10)     | 12.7M    | 282.19           | 97.41% | 195.24          |\n|                      | MetaKAN | (784,2048,10)      | 1.6M     | 717.64           | 97.47% | 2012.32         |\n|                      | MLP     | (784,1800×3,10)    | 7.9M     | 310.55           | 98.11% | 121.33          |\n|                      | MetaKAN | (784,512×3,10)     | 0.93M    | 593.61           | 97.95% | 640.75          |\n\n3. In summary, while MetaKAN trains slower than MLPs, it retains the core strengths of KANs—strong inductive bias and interpretability—while dramatically reducing parameter count and memory usage compared to KAN. This allows MetaKAN to close the training cost gap with MLPs, offering a practical and scalable alternative that balances efficiency with symbolic inductive biase.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes.  I reviewed the justification for the theoretical parameter count of MetaKAN.", "experimental_designs_or_analyses": "Yes.  I reviewed the experimental designs presented in Section 4 related to function fitting and various tasks using variations on the base convolutional architecture.", "relation_to_broader_scientific_literature": "MetaKAN innovates on the original KAN architecture presented in Liu et al., 2024 to address some of its inherent complexity issues.  Various prior approaches have attempted to address these issues by replacing the B-splines utilized in Liu et al., 2024 with various alternatives, such as Chebyshev polynomials (SS et al., 2024), rational functions (Yang & Wang, 2024), wavelets (Bozorgasl & Chen, 2024), and radial functions (Li, 2024).  MetaKAN retains the B-spline functions and reduces overall parameter count by generating B-spline coefficients via a meta-learner (inspired by Kong et al., 2020), rather than training those coefficients directly.", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "Strengths:\n- Addresses part of the problematic complexity of KAN by reducing overall parameter count to a level comparable to that of MLP.\n- Achieves performance comparable to or exceeding that of base KAN variants on a variety of tasks, including symbolic regression, partial differential equation solving, and image classification.\n- Method is portable to several prominent KAN variants, yielding reduced parameter counts and competitive performance.\n\nWeaknesses:\n- One of the primary motivations stated in the paper is to reduce the training cost of KAN’s relative to MLP’s; however, the paper does not provide sufficient support for its claim that training costs are reduced.  While a reduction in parameter count may reduce the overall training time, energy demands, etc., increased computational complexity can offset those reductions, and the paper does not provide sufficient evidence to suggest computational complexity or training time are comparable or reduced.\n- While the paper identifies in Figure 12 that the theoretical parameter count of MetaKAN is comparable to that of MLP, it does not specify the theoretical parameter count of the modified MetaKAN architecture set forth in Section 3.4 for use in deeper KAN’s.  Given that the parameter count disparity between KAN and MLP is most acute in the deep network context, a discussion of such theoretical complexity is warranted.  While the paper does present empirical evidence of parameter count in several deep network experiments to demonstrate the relative parameter efficiency of MetaKAN in relation to KAN, it does not include comparison results against MLP models.\n- The paper does not address in sufficient detail the impact of the MetaKAN innovations on computational complexity / effective computation time relative to MLP, KAN, or any of the KAN variants.  Although the MetaKAN architecture reduces the overall parameter count relative to KAN, it is unclear what the impact of this is on computation time.  One of the primary issues with KAN is its training time relative to MLP, and the paper does not address the impact of MetaKAN on this concern.", "other_comments_or_suggestions": "- In Section 3.1.2, Figure 3, the term in the bottom left corner of the matrix uses d_out rather than n_(l+1).  For consistency with the remainder of the figure and the surrounding discussion, I believe this should be n_(l+1).\n- In Section 3.2.2, Figure 7, the indexing notation for l is inconsistent with that of Figure 4.  In particular, the union ranges from l=1 to l=L-1, which does not align with Figure 4's range from l=0 to l=L-1.  I believe Figure 7 should be updated to align with Figure 4.\n- The parameter counts for MetaKAN provided in Table 1 vs. in Figure 12 are inconsistent.  I believe the Table 1 count is accurate, as it is the parameter count of a 2-layer MLP with dimensions din = 1, dhidden, dout = G + k + 1.", "questions_for_authors": "- What is the computation time of MetaKAN relative to KAN and MLP?  You have demonstrated competitive performance with the reduced parameter count in MetaKAN, but it is unclear what the effect of these changes are on computational complexity.\n- Have you performed any experiments testing the performance of MetaKAN relative to MLP (for example, comparing performance with similar architecture dimensions or parameter counts)?  Some discussion on the performance of MetaKAN relative to MLP would be helpful in evaluating its utility in the contexts for which testing has been conducted.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "katwnIeMXA", "reviewer": "Reviewer_tj4z", "summary": "In this work, the Authors combine KANs with Hypernetworks and show the improved/comparable accuracy at the lower parameter counts.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "C1: Show a task solvable by MetaKAN but not by KAN\n\nA1:We thank the reviewer for this important suggestion. In fact, our experiments in Section C.1 (High dimensional function) and Table 4 (function fitting task) already demonstrate a key scenario where standard KANs fail while MetaKAN succeeds:\n\nFor function $f₂(x) = ∑x² + x³$ at dimension $d=50$, standard KANs completely fail to converge (marked as \"NA\" in Table 4), while MetaKAN achieves stable performance (MSE=1.15×10⁻³). At $d=1000$, KANs show severe error amplification (MSE=$1.68×10²$), whereas MetaKAN maintains reasonable accuracy (MSE=$1.43×10⁻¹)$ with 7× fewer parameters.\n\nFrom the theoretical FLOPs analysis in A1 (response to Reviewer A7wa), as network size increases, the computational complexity of KAN grows rapidly with the number of edges. In contrast, MetaKAN benefits from reduced backward-pass complexity, which can result in faster training times on large-scale tasks.\n\nFor instance, in the empirical comparison table provided in A1, MetaKAN not only consumes just 30% of the peak memory used by KAN, but also achieves faster training time in the largest network stucture. These results highlight MetaKAN’s strong potential for scaling to even larger models—scenarios where traditional KANs may become impractical due to memory limitations and the high cost of gradient computations.\n\nC2: Line 197: functions -> function Line 215: formalizes -> formalize\n\nA2:We have corrected the typos.\n\nC3: Table 2: There are some apparent inconsistencies. On Line 362, the text says that, at G = 5, MetaKAN achieves a lower MSE in 14 out of 16 functions with parameter reductions. None of these numbers match the table: There are 17 functions, 10 of them correspond to the parameter reduction (by the way, why is that the case?), and all of them are highlighted by the bold font implying the improvement in the MSE. Could you please explain this inconsistency?\n\nA3:We apologize for the confusion in Table 2. The inconsistencies arose because:\n\n1. The text mistakenly stated \"16 functions\" (now corrected to 17).\n2. For detailed comparasion of the parameter count, We take $G=5,k=3$, and the network struncture is $[n_0,n_1,...,n_L]$, the parameter count of KAN is $\\sum_{l=0}^{L-1}(n_l\\times n_{l+1})\\times (G+k+1) =9\\sum_{l=0}^{L-1}(n_l\\times n_{l+1}) $, and the parameter count of MetaKAN is $\\sum_{l=0}^{L-1}(n_l\\times n_{l+1})+(d_{hidden}+1)\\times (G+k+1) = \\sum_{l=0}^{L-1}(n_l\\times n_{l+1})+9(d_{hidden}+1)$.The condition for MetaKAN to have fewer parameters than KAN is:  $9\\sum_{l=0}^{L-1}(n_l\\times n_{l+1})>\\sum_{l=0}^{L-1}(n_l\\times n_{l+1})+9(d_{hidden}+1)$, which implies the $d_{hidden} < \\frac{8}{9}\\sum_{l=0}^{L-1}(n_l\\times n_{l+1})-1$. So as for some simple network structure (where $\\sum_{l=0}^{L-1}(n_l\\times n_{l+1})$is small), it does not have the parameter reduction cases.\n3. Bold font was incorrectly applied to all MSE improvements, even without parameter reduction. We have now highlighted only cases with  both MSE improvement and parameter reduction.", "claims_and_evidence": "The claims are supported by the evidence: the proposed model is designed in a standard way, as far as hypernetworks are considered, and is tested on the datasets previously used in the field of KANs.", "methods_and_evaluation_criteria": "Benchmark datasets here are the same as used in prior works in the field of KANs, enabling the direct comparisons of the results here with prior models. Such comparisons are also provided in the paper.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental design is standard for the field of Hypernetworks, therefore, it is validated and justified.", "relation_to_broader_scientific_literature": "The work extends two lines of research: on Hypernetworks and on KANs, combining them for the first time and showing the utility of such a combination on up-to-date models from literature.", "essential_references_not_discussed": "The work thoroughly discusses prior literature on hypernetworks, KANs, and benchmarks.", "other_strengths_and_weaknesses": "Strength: the text is very well written and the experiments are well-documented.\n\nWeakness: the evaluation is only performed on the tasks already solved with KANs (which is a necessary step but may not be a sufficient step). It would be interesting to see MetaKANs solving a task that was not possible to solve with the regular KANs. Such a demonstration would strengthen the work by showing a unique capability of the proposed model.", "other_comments_or_suggestions": "Line 197: functions -> function\n\nLine 215: formalizes -> formalize", "questions_for_authors": "Table 2: There are some apparent inconsistencies. On Line 362, the text says that, at G = 5, MetaKAN achieves a lower MSE in 14 out of 16 functions with parameter reductions. None of these numbers match the table: There are 17 functions, 10 of them correspond to the parameter reduction (by the way, why is that the case?), and all of them are highlighted by the bold font implying the improvement in the MSE. Could you please explain this inconsistency?\n\nCould you please provide an example task that is intractable with KANs but tractable with MetaKAN?\n\nI’m happy to revisit my score based on responses to these questions.\n\n________________________\n\nPost-rebuttal: the typos are fixed and the comments are addressed. Raising my score.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "adBrE3Zkkv", "reviewer": "Reviewer_rDdN", "summary": "This paper proposes MetaKAN, which leverages a hypernetwork to generate the B-spline coefficients of KANs. Each edge function is associated with a learnable prompt (usually 1D) so the G+K+1 number of coefficients is compressed down to a single scalar, achieving parameter reduction. They apply MetaKANs to function-fitting tasks (Meta + KANs), PDE solving and image cognition tasks (Meta + Convolutional KANs).\n\n## Update after rebuttal\nThe authors have addressed my questions. Overall this is a great paper, but 5 is a bit of a stretch, so I raised my score from 3 to 4.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "C1: Some algorithmic choices seem arbitrary\n\nA1: We appreciate the reviewer's point regarding the clarity of our algorithmic choices. We acknowledge that the rationale behind certain design choices, such as the specific architecture of the meta-learner (e.g., a two-layer MLP) and the initial dimension of the learnable prompts (e.g., d=1), may not have been sufficiently stated in the original manuscript. In the revised version, we have added explanations for these choices in  Section 3.2.2. We selected a two-layer MLP as the meta-learner because it balances expressive power and parameter efficiency, which is a common choice in many meta-learning and hypernetwork studies [1-3]. For the prompt dimension d=1, we chose it as the most  parameter-minimal  for experimentation. We also explored the impact of different prompt dimensions in the ablation study in  Appendix C.4. The results show that increasing the dimension can improve performance but also comes with additional parameters and potential training instability. This confirms the rationale behind our initial choice of d=1 as the baseline setting, while allowing for flexibility to adjust the dimension when needed. We strive to ensure that all design choices are grounded in empirical observations or inspired by related work, and we will clarify these considerations more explicitly in the revised manuscript.\n\n[1] HyperNetworks, ICLR 2017.\n\n[2] Meta-weight-net, NeurIPS, 2019.\n\n[3] Continual learning with hypernetworks, ICLR, 2020.\n\nC2：Why MetaKANs have more parameters than KANs in Table 2.\n\nA2: We sincerely appreciate the reviewer's careful attention to these important details. You're absolutely correct to note that MetaKANs can have more parameters in certain configurations. Please refer the details about the analysis of the situation to the A3.2 for reviewer tj4z. Besides, the potential benifits of MetaKANs are scaling up to even larger KAN models (the results of large KANs in Table 2; please refer to response for Reviewer A7wa).\n\nAnd we will reformat all tables to highlight only cases with both MSE improvement and parameter reduction.\n\nC3: Why does depth matter? \n\nA3: \n\nThe depth of MetaKAN motivates our design choice of using multiple meta-learners (hypernetworks) in deeper architectures. In deep networks, features become more abstract with depth—early layers capture low-level patterns, while deeper layers model higher-level semantics. Accordingly, the optimal form of activation functions may vary across layers.\n\nUsing a single global hypernetwork to generate all activation functions may be overly restrictive, as it must capture diverse functional shape  through input embeddings. Instead, grouping layers and assigning a separate meta-learner to each group allows better specialization, aligning with the hierarchical structure of deep features.\n\nOur ablation study (Appendix C.4) supports this: increasing the prompt dimension from $d=1$ to $d=4$ improves accuracy but also increases variance. Using multiple meta-learners per layer group reduces this variance, especially in deeper networks, suggesting enhanced stability. We plan to further explore the theoretical basis of this effect in future work.\n\nC4: Visualize how B-spline functions vary with prompt dimension\n\nA4: We appreciate the reviewer’s suggestion regarding visualization. In response, we conducted an experiment by varying the promp dimension from 1 to 4 and visualized the corresponding learned B-spline functions  trained on the dataset $f=\\exp(\\sin(x_1^2+x_2^2)+\\sin(x_3^2+x_4^2))$ . These visualizations reveal how increasing the embedding dimension enriches the expressivity of each spline function. \n| Dimension of Embedding | MSE ($10^{-3}$)    |\n|-----------------------|-----------|\n| 1                     | 3.60  |\n| 2                     | 3.45|\n| 3                     | 3.64 |\n| 4                     | 4.21 |\n\n\nWith a lower embedding dimension, the splines exhibit relatively simple, smooth transformations. As we increase the embedding dimension to 3 and 4, the spline structures become more complex, reflecting the model’s enhanced capacity to capture more complex functions. For the detailed visualization, please refer to the link https://github.com/icmlrebuttal25/Append\n\nC5: Combining with network pruning\n\nA5: Thank you for your interest in combining our meta-learning approach with network sparsification/pruning. Since MetaKAN focuses on generating activation parameters without altering network structure, pruning can be applied independently. For example, in symbolic regression tasks (Appendix B), we applied sparsification by removing low-activation edges and using symbolic simplification. These results show MetaKAN works well with sparsification techniques.", "claims_and_evidence": "The claim about parameter saving (in conclusion) should be tuned down (add \"most of the times\" or \"for large-scale KANs\") since results in Figure 2 show that sometimes KANs are more parameter efficient than MetaKANs, especially when KANs themselves are small.", "methods_and_evaluation_criteria": "Yes. They compare models based on model size and performance (loss or accuracy).", "theoretical_claims": "Not applicable. No theoretical claims in the paper.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "KANs (Liu et al.) demonstrated the superior performance of KANs over MLPs in small-scale tasks. KANs can become quite parameter-inefficient when they are scaled up. In particular, given the same shape (width and depth), KANs have (G+k+1) times more parameters than MLPs due to the modeling of the B-splines. As a result, it is urgent to reduce the number of parameters for KANs to make them practically useful. This paper presents one interesting solution via meta-learning.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "**Strengths**\n* The paper is nicely written and well-motivated\n* The paper proposes an interesting and effective solution (probably, see my questions below) to reducing the size of KANs.\n* Experiments are diverse -- functional fitting, PDE solving and image recognition. \n\n**Weaknesses**\n* Some algorithmic choices seem arbitrary\n* In some cases metaKANs appear to have more parameters than KANs. This is not well discussed and acknowledged as a limitation.", "other_comments_or_suggestions": "* Boldfaces are not consistently used in tables. For example in Table 2, the comparison on # of parameters for KAN/MetaKAN is not shown for G=5.", "questions_for_authors": "* I'm not sure why depth (Sec 3.4) makes the story different. Can we simply increase the prompt dimension (usually one) to promote diversity?\n* In Table 2, there are a few cases when MetaKANs have more parameters than KANs. why is this the case? Also, boldfaces are not used consistently in Table 2.\n* I would love to see this visualization: changing the prompt dimension, how does the B-spline function change? This gives us a sense of what the family of functions looks like. \n* Does this meta-learning trick still work when sparsification is added to prune the network?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces MetaKANs, a memory-efficient training framework for Kolmogorov-Arnold Networks (KANs) that replaces direct optimization of activation parameters with a smaller shared meta-learner. The method achieves substantial parameter reduction while maintaining or improving performance across a range of tasks, including symbolic regression, PDE solving, and image classification.

The reviewers were broadly positive. Reviewers vpTT and tj4z praised the clarity of the writing, soundness of the experimental setup, and the novelty of applying meta-learning to KANs. Reviewer A7wa initially raised concerns about the lack of computational complexity analysis and comparisons to MLPs, but was satisfied by the authors' detailed rebuttal and raised their score. Reviewer rDdN similarly appreciated the clarifications and updated their score positively. The paper was generally viewed as well-motivated and technically solid, with some reviewers requesting more extensive ablations or discussion of architectural choices, which were also addressed in the rebuttal.
Decision: Accept (poster)