MLP-KAN: Unifying Deep Representation and Function Learning

## Abstract

Abstract Recent advancements in both representation learning and function learning have demonstrated substantial promise across diverse domains of artificial intelligence. However, the effective integration of these paradigms poses a significant challenge, particularly in cases where users must manually decide whether to apply a representation learning or function learning model based on dataset characteristics. To address this issue, we introduce MLP-KAN, a unified method designed to eliminate the need for manual model selection. By integrating Multi-Layer Perceptrons (MLPs) for representation learning and Kolmogorov-Arnold Networks (KANs) for function learning within a Mixture-of-Experts (MoE) architecture, MLP-KAN dynamically adapts to the specific characteristics of the task at hand, ensuring optimal performance. Embedded within a transformer-based framework, our work achieves remarkable results on four widely-used datasets across diverse domains. Extensive experimental evaluation demonstrates its superior versatility, delivering competitive performance across both deep representation and function learning tasks. These findings highlight the potential of MLP-KAN to simplify the model selection process, offering a comprehensive, adaptable solution across various domains. Our code and weights are available at https://github.com/DLYuanGod/MLP-KAN .

## 1 Introduction

In recent years, deep learning has evolved from early neural network concepts to sophisticated architectures, such as transformer networks (Vaswani, ), driven by advancements in computational resources and the availability of large datasets, thereby achieving remarkable performance across diverse applications. Along with important technological breakthroughs, representation learning (OpenAI, ; Anthropic, ; OpenAI, ; Touvron et al., ) and function learning (Narayan et al., ; Zhang et al., ; Wu et al., ) moments of prominence and have been extensively explored and utilized in various research and application tasks related to data and learning nowadays. At the same time, the focus of function learning research has shifted from simple function fitting to deep learning (Cuomo et al., ; Cai et al., ), which excels in tasks requiring precise function approximation and has seen new advancements, particularly in its applicability to univariate function tasks. The key difference between representation learning and function learning lies in their objectives: representation learning aims to extract features from data to understand its underlying structure (Bengio et al., ), while function learning focuses on creating direct mappings between inputs and outputs, making it more suited for tasks requiring precise functional relationships (Zupan et al., ).

In this paper, we introduce MLP-KAN, a novel framework that unifies two distinct learning approaches into a cohesive system, utilizing the Mixture of Experts (MoE) methodology Jiang et al. (). Within the architecture of MLP-KAN, Multi-Layer Perceptrons (MLP) (Rumelhart et al., ) function as representation experts, while Kernel Attention Networks (KAN) (Liu et al., ) are designated as function experts. The MoE mechanism efficiently routes inputs to the appropriate expert, significantly enhancing both efficiency and performance across a diverse range of tasks. MLP-KAN was developed to address the problem users encounter when determining whether to apply representation learning or function learning models across diverse datasets. By integrating MLPs and KANs within a mixture-of-experts framework, this architecture dynamically adapts to the specific characteristics of the task, ensuring optimal performance without requiring manual model selection. The main challenge in our method is effectively integrating MLPs and KANs, ensuring the right model is selected for each task without compromising performance. In additional, balancing the differing training needs of representation and function learning while maintaining efficiency across diverse datasets is complex. The main challenge in our method is effectively integrating MLPs and KANs, ensuring the right model is selected for each task without compromising performance, as shown in Figure 1. In additional, balancing the differing training needs of representation and function learning while maintaining efficiency across diverse datasets is complex.

To address the challenge of effectively integrating MLPs and KANs within the MoE framework, we utilized a soft MoE approach. This method enables dynamic and flexible routing between MLPs for representation learning and KANs for function learning. By incorporating this MoE system within a transformer framework, the model can seamlessly perform deep representation learning or deep function learning, adapting to the specific nature of the task at hand while maintaining efficiency across diverse datasets.

The main contributions of this work are as follows:

- •
We present MLP-KAN, a unified framework that synergizes MLP for representation learning with KAN for function learning. This novel architecture leverages a MoE mechanism to dynamically route tasks between representation and function experts, addressing the challenge of selecting the appropriate learning paradigm for diverse datasets.
- •
We propose a flexible and versatile model by integrating MLP-KAN within the transformer architecture, enabling efficient performance across both representation and function learning tasks. This integration enhances model capability and improves performance across a broad range of tasks, including computer vision, natural language processing, and symbolic formula representation.
- •
We perform extensive experimental evaluations, demonstrating that MLP-KAN consistently outperforms or matches state-of-the-art models such as MLP and KAN on widely recognized benchmarks, including computer vision,nature language processing, and functional dataset. Our approach achieves superior accuracy in representation learning tasks and lower RMSE in function learning tasks, underscoring its universal applicability across diverse domains.

Figure: Figure 1: The comparison between the MLP, KAN, and our proposed MLP-KAN. In the domains of Computer Vision and Natural Language Processing, the goal is to achieve the highest accuracy possible. In contrast, for the Symbolic Formula Representation task, the objective is to minimize the root mean square error (RMSE). The numbers are the average values of the experimental results. MLP-KAN effectively combines the strengths of both, ensuring strong performance in representation and function learning, and eliminating the need for task-specific model selection.
Refer to caption: x1.png

## 2 Related Work

#### Deep Representation Learning.

Deep representation learning has gained significant attention due to its ability to automatically discover hierarchical feature representations from raw data (Butepage et al., ; Zhong et al., ; Long et al., ), outperforming traditional hand-crafted feature extraction techniques. The introduction of deep learning methods, such as MLP based convolutional neural networks (Li et al., ) and recurrent neural networks, enabled breakthroughs in areas like image recognition (Zoph et al., ; He et al., ), object detection (Zhao et al., ; Yu et al., ; Liu et al., ), and natural language processing (Chowdhary & Chowdhary, ; Khurana et al., ) by capturing more abstract and high-level features. Recent advancements in deep architectures, including transformer-based models (Gillioz et al., ), have further pushed the boundaries of representation learning, proving highly effective across diverse domains. For example, generative AI, such as large language models (LLMs) (Yao et al., ; Zhao et al., ), has garnered significant attention for its ability to generate coherent, contextually relevant text and learn deep representations from vast amounts of unstructured data. LLMs like GPT-4o (OpenAI, ) and LLaMA (Touvron et al., ) utilize MLP based transformer architectures, which excel at capturing long-range dependencies in sequential data, allowing them to perform tasks such as text generation, summarization, and translation with remarkable accuracy. Beyond natural language processing, LLMs have also influenced other fields, including code generation (Chung et al., ; Li et al., ), medical diagnosis (Kononenko, ; Amato et al., ), and drug discovery (Drews, ; Sliwoski et al., ), by leveraging their deep learning capabilities to model complex relationships in data. These advancements highlight the growing importance of deep representation learning in not only understanding and generating human-like text but also in solving a wide range of interdisciplinary challenges (Newell et al., ). In these models, MLP play a crucial role as fundamental building blocks, serving as dense layers that transform and learn high-dimensional representations by mapping inputs to deeper abstract features (Donoho et al., ).

#### Deep Function Learning.

Deep function learning focuses on capturing complex mathematical relationships and patterns within data, particularly in scientific and engineering domains (Sarker, ; Shen, ; Karpatne et al., ). Techniques such as Physics-Informed Neural Networks (PINNs) (Raissi et al., ) have emerged as powerful tools for solving partial differential equations (PDEs) (Evans, )
by embedding physical laws into neural network architectures, allowing for accurate modeling of phenomena governed by underlying physical principles (Raissi et al., ; Cuomo et al., ). Beyond traditional neural networks, deep function learning leverages over-parameterized models, which enable the precise interpolation of data, even in the presence of noise, enhancing both generalization and optimization performance (Karniadakis et al., ; Advani et al., ; Chen et al., ). Recent advancements have demonstrated the potential of these methods for tasks such as surrogate modeling (Razavi et al., ), sensitivity analysis (Christopher Frey & Patil, ; Lenhart et al., ), and discovery of new scientific relationships (Wren et al., ; Klahr & Simon, ). KAN are highly effective for function learning due to their ability to capture complex non-linear relationships through learnable spline-based univariate functions, offering superior approximation capabilities and scaling compared to traditional MLP (Yu et al., ; Liu et al., ; Zhang, ; Vaca-Rubio et al., ).

## 3 Preliminary

**Table 1: Comparison between MLP and KAN.**
| Feature | MLPs | KANs |
| --- | --- | --- |
| Activation Functions | Fixed functions (e.g., ReLU, SiLU) | $\varphi(x)=\sum_{i=1}^{k}c_{i}B_{i}(x)$ |
| Weight Structure | Scalar weights | Spline-based weights $\varphi(x)$ |
| Layer Architecture | Standard fixed depth | $\Phi_{q}\left(\sum_{p=1}^{n}\varphi_{q,p}(x_{p})\right)$ |
| Error Scaling | Limited by dimensionality | $\|f-(KAN)\|_{C^{m}}\leq CG^{-k-1+m}$ |
| Scaling Law | $\ell\propto N^{-\alpha}$ with lower $\alpha$ | $\ell\propto N^{-\alpha}$ with higher $\alpha=4$ |
| Expressiveness | Suited for general representation learning | Suited for functional learning |

KAN are inspired by the Kolmogorov-Arnold Representation Theorem (Liu et al., ), which asserts that any multivariate continuous function $f(x)$ can be decomposed into a sum of univariate functions. This is formally stated as:

$$ $f(x)=\sum_{q=1}^{2n+1}\Phi_{q}\left(\sum_{p=1}^{n}\varphi_{q,p}(x_{p})\right)$ (1) $$

where $\varphi_{q,p}(x_{p})$ and $\Phi_{q}$ are univariate functions, summing over $q$ and $p$. Unlike traditional Multi-Layer Perceptrons (MLPs), which use fixed activation functions at each neuron, KANs introduce learnable univariate activation functions on the edges between layers (Vaca-Rubio et al., ; Aghaei, ). Each weight in KANs is replaced by a learnable spline function:

$$ $\varphi(x)=\sum_{i=1}^{k}c_{i}B_{i}(x)$ (2) $$

where $B_{i}(x)$ are basis functions (such as B-splines) and $c_{i}$ are trainable coefficients (Eilers & Marx, ). This spline-based approach allows KANs to better capture non-linear relationships, particularly in high-dimensional tasks where MLPs tend to struggle.

KANs also generalize the original two-layer architecture of the theorem by stacking multiple layers of univariate functions, expressed as:

$$ $KAN(x)=(\Phi_{L-1}\circ\Phi_{L-2}\circ\cdots\circ\Phi_{1}\circ\Phi_{0})(x)$ (3) $$

The approximation capabilities of KANs scale better compared to MLPs, as shown in Table 1. The error bound for KANs with splines of order $k$ and grid size $G$ is $\|f-(KAN)\|_{C^{m}}\leq CG^{-k-1+m}$ where $C$ is a constant, and $m$ represents the order of derivatives considered. Furthermore, KANs exhibit superior neural scaling laws, with the test loss decreasing as $\ell\propto N^{-\alpha}$ where $N$ is the number of parameters and $\alpha$ depends on the spline order $k$. For cubic splines ($k=3$), KANs achieve $\alpha=4$, outperforming MLPs, which often cannot reach these scaling efficiencies. This makes KANs particularly effective for high-dimensional function approximation (Sprecher & Draghici, ; Köppen, ).

## 4 Methodology

### 4.1 MLP-KAN

As shown in Figure 2, our proposed MLP-KAN is composed of $NE$ experts, which can be classified into two types: representation experts and function experts. Representation experts, based on MLP architectures, focus on learning rich feature representations, while function experts, utilizing FasterKAN architectures, specialize in tasks requiring smooth and precise interpolation over continuous data points. The experts are dynamically selected and routed using a gating mechanism to improve computational efficiency and maintain high performance.

Figure: Figure 2: The framework combines a soft mixture of experts (MoE) with a unification of MLPs and KANs, denoted as the MLP-KAN module, to dynamically select experts for each token. The input tokens are passed through a multi-headed self-attention mechanism followed by layer normalization. The routing process involves soft weighting of experts for each slot and token via linear combinations and a softmax layer per slot and token. MLP and KAN experts are arranged in parallel, and based on the input’s characteristics, either MLP or KAN is selected for computation, enhancing the model’s ability to handle diverse representations efficiently. The gating mechanism determines the most relevant expert for each token, improving overall computational efficiency. This architecture retains the residual connections of the traditional Transformer while expanding its capacity to model complex functional and representational data.
Refer to caption: x2.png

#### Representation Expert.

In the context of MLP-KAN models, half of the experts are designed as representation experts, utilizing multi-layer perceptrons (MLPs). These experts excel in tasks requiring the learning of rich feature representations, such as image classification.
Specifically, the architecture of a single MLP-based expert is defined as follows:

$$ $\text{Expert}_{i}=\text{MLP}(\mathbf{X})\quad\text{for }i=1,\dots,\frac{NE}{2}$ (4) $$

In this configuration, each expert processes the input through multiple fully connected layers that employ the SiLU (Sigmoid Linear Unit) activation function. Unlike ReLU (Rectified Linear Unit) (Hahnloser et al., ), SiLU provides smooth gradients and mitigates the issue of dying neurons, enhancing the robustness and efficiency of learning.

The process of forward propagation within each expert is executed as follows:
Given an input $\mathbf{X}\in\mathbb{R}^{B\times N\times D}$, where $B$ is the batch size, $N$ is the sequence length, and $D$ is the feature dimension, the transformation through the MLP involves applying a linear transformation followed by the SiLU activation function:

$$ $\mathbf{h}^{(1)}=\text{SiLU}(\mathbf{W}^{(1)}\mathbf{X}+\mathbf{b}^{(1)}),\ \mathbf{h}^{(2)}=\mathbf{W}^{(2)}\mathbf{h}^{(1)}+\mathbf{b}^{(2)}$ (5) $$

where $\mathbf{W}^{(1)}\in\mathbb{R}^{D\times H}$ and $\mathbf{W}^{(2)}\in\mathbb{R}^{H\times D^{\prime}}$ are the weight matrices, and $\mathbf{b}^{(1)}\in\mathbb{R}^{H}$ and $\mathbf{b}^{(2)}\in\mathbb{R}^{D^{\prime}}$ are the bias vectors of the corresponding layers. The output $\mathbf{h}^{(2)}$ is passed on for further processing.

#### Function Expert.

The other half of the experts in MLP-KAN are defined as function experts to handle specialized data, particularly in functional datasets. These experts are based on the FasterKAN (Delis, ) architecture, which is known for its strong performance in tasks requiring smooth interpolation over continuous data points.

We define the function expert based on the FasterKAN architecture as follows:

$$ $\text{Expert}_{i}=\text{FasterKAN}(\mathbf{X})\quad\text{for }i=\frac{NE}{2}+1 ,\dots,NE$ (6) $$

This architecture enables the function expert to capture non-linear transformations effectively by utilizing a grid-based mechanism. Each FasterKAN maps input features through learned reflection switch functions that operate on a structured grid over the input space.

The transformation of an input $\mathbf{X}\in\mathbb{R}^{B\times N\times D}$ through the expert’s layers follows these steps:

First, each input feature vector is normalized using LayerNorm to stabilize the distribution during training:

$$ $\mathbf{X}_{\text{norm}}=\text{LayerNorm}(\mathbf{X})$ (7) $$

Subsequently, the reflectional switch function $\phi(\mathbf{x})$ computes the differences between the normalized input, predefined grid points and
hyper-parameter denominator, followed by a non-linear transformation to approximate smooth basis functions:

$$ $\phi(\mathbf{X})=1-\tanh\left(\frac{\mathbf{X}-\text{grid}}{\text{denominator} }\right)^{2}$ (8) $$

Lastly, the computed basis values are passed through a spline transformation $\mathbf{W}_{\text{spline}}$ to map the input to the output dimension:

$$ $\mathbf{y}=\mathbf{W}_{\text{spline}}\cdot\phi(\mathbf{X})$ (9) $$

By integrating FasterKAN for half of the experts, MLP-KAN is well-equipped to process functional data, leveraging FasterKAN’s interpolation across a smooth grid representation. The remaining experts can follow alternative architectures, allowing MLP-KAN to dynamically select the optimal model based on the input’s characteristics.

#### Gating Mechanism.

In MLP-KAN, the gating mechanism serves a pivotal function in dynamically routing input tokens to the most relevant experts. This mechanism efficiently selects a subset of experts for each input sequence, reducing computational overhead while maintaining robust model performance.

Given an input sequence $\mathbf{X}\in\mathbb{R}^{B\times N\times D}$, the gating mechanism computes the similarity between the input tokens and a set of learnable slot embeddings $\mathbf{E}\in\mathbb{R}^{NE\times S\times D}$, where $NE$ is the number of experts and $S$ is the number of slots per expert. This similarity is calculated as follows:

$$ $\text{logits}_{b,n,e,s}=\langle\mathbf{X}_{b,n,:},\mathbf{E}_{e,s,:}\rangle, \quad\text{for }b\in[1,B],n\in[1,N],e\in[1,NE],s\in[1,S]$ (10) $$

where $\langle\cdot,\cdot\rangle$ denotes the dot product, and the resulting logits $\text{logits}\in\mathbb{R}^{B\times N\times E\times S}$ represent the unnormalized attention scores between each token and the expert slots.

Next, a softmax function is applied along the expert and slot dimensions to compute the dispatch weights $\alpha\in\mathbb{R}^{B\times N\times E\times S}$, determining the contribution of each token to each expert:

$$ $\alpha_{b,n,e,s}=\frac{\exp(\text{logits}_{b,n,e,s})}{\sum_{e^{\prime},s^{ \prime}}\exp(\text{logits}_{b,n,e^{\prime},s^{\prime}})}$ (11) $$

These dispatch weights $\alpha$ are then used to aggregate the input tokens across the sequence for each expert, resulting in routed inputs $\mathbf{z}\in\mathbb{R}^{B\times E\times S\times D}$:

$$ $\mathbf{z}_{b,e,s,:}=\sum_{n=1}^{N}\alpha_{b,n,e,s}\mathbf{X}_{b,n,:}$ (12) $$

Finally, each expert processes its routed inputs, and the outputs from all experts are aggregated using softmax-normalized combination weights. This ensures that the final output $\text{F}(\mathbf{X})$ is a unified combination of contributions from all experts, based on the initial input $\mathbf{X}$.

### 4.2 Architecture

While the traditional Transformer architecture has shown remarkable success in various tasks, it still encounters limitations in scaling efficiently, particularly when dealing with diverse and complex input distributions. To address these challenges, we draw inspiration from two primary sources: the MLP-KAN paradigm, which allows dynamic routing of tokens to different experts, and the block-sparse operations that enable efficient expert utilization. As depicted in Figure 3 and Equation (13), we replaced the standard MLP layer in the Transformer block with an MLP-KAN-based module to improve the model’s capacity in handling diverse token representations. This modification helps the model better capture complex dependencies while maintaining computational efficiency by selecting only a subset of experts for each token.

$$ $\mathbf{Y}=\mathbf{X}+\mathrm{MHA}(\mathrm{LN}(\mathbf{X}))+\text{F}(\mathrm{ LN}(\mathbf{X}+\mathrm{MHA}(\mathrm{LN}(\mathbf{X}))))$ (13) $$

Where LN denotes the layer normalization (Ba et al., ) applied to the input and intermediate states, respectively, MHA represents the multi-head self-attention mechanism that captures contextual information across the token sequence. Our proposed MLP-KAN replaces the traditional MLP, where experts are dynamically selected based on the input through a gating mechanism, ensuring efficient routing of tokens to the most relevant experts. $\mathbf{X}$ represents the input data after passing through the attention mechanism, and $\mathbf{Y}$ represents the output data after the combined processing of the MoE module and residual connections. This modification allows for more flexible token-wise computations while maintaining the overall structure of the Transformer block.

Figure: Figure 3: Architecture of the transformer encoder with MLP-KAN Integration.
Refer to caption: x3.png

## 5 Experiment

### 5.1 Experimental Setup

#### Datasets.

We have validated the effectiveness of our method on several public datasets.
In representation learning, we have validated the CIFAR-10, CIFAR-100, and mini-ImageNet datasets (Krizhevsky et al., ; Vinyals et al., ) in the field of computer vision, and the SST2 dataset (Socher et al., ) in the field of natural language processing. In function learning, we have validated thirty functions on the Feynman dataset (Udrescu & Tegmark, ).
The CIFAR-10 and CIFAR-100 datasets are the tasks of image classification, both consisting of 50,000 images for the training set and 10,000 images for the test set. However, the former has only 10 categories, while the latter has 100 categories.
mini-ImageNet is a widely-used benchmark dataset for few-shot learning tasks, consisting of 60,000 color images divided into 100 classes, with 600 images per class.
Both CV datasets use top-1 accuracy (top1-acc.) and top-5 accuracy (top5-acc.) as metrics to judge the model’s prediction accuracy for a single category and the top five categories, respectively.
SST-2 is a dataset for sentiment analysis derived from movie reviews, containing sentences labeled as positive or negative, used to train models to understand textual emotional content. Specifically, we use the F1 score (F1) and the accuracy score (Acc) to measure performance.
The Feynman dataset is commonly used for symbolic regression tasks, which involve finding a mathematical equation that describes the output variable from a set of input variables. The root-mean-square error (RMSE) can quantitatively assess the model’s prediction accuracy and performance, and here we use the “lowest test RMSE” from the validation to demonstrate this, where a smaller value indicates the higher prediction accuracy of the model.

#### Training and Evaluation Details.

To comprehensively demonstrate the superiority of MLP-KAN, our experimental setup involved comparisons with MLP and KAN. These extensive experiments demonstrate that our method can be universally applied across various domains and consistently achieves excellent results.
All experiments were conducted using four A100 GPUs. During the training phase, we meticulously tuned parameters to optimize the learning process.
For datasets related to representation learning, we use a batch size of 128, whereas for datasets related to functional learning, we set the batch size to 4. The learning rate was initially set at 5e-5, and the training continues until convergence.
We applied dropout to the output of each MLP-KAN using a dropout rate of 0.1. Regarding the hyperparameters of MLP-KAN, we configured $n=8$ (i.e., 8 experts) and $k=2$ (i.e., top2 experts).

### 5.2 Function Learning

**Table 2: Comparison of losses for Feynman Equations. Results highlighted in bold represent the best performance in the comparison, while those underlined represent the second-best results.**
| Feynman Eq. | Original Formula | Variables | KAN loss | MLP loss | MLP-KAN loss |
| --- | --- | --- | --- | --- | --- |
| $I.6.20a$ | $\frac{e^{-\theta^{2}/2}}{\sqrt{2\pi}}$ | $\theta$ | $\underline{8.82\times 10^{-4}}$ | $1.37\times 10^{-1}$ | $\bf 3.87\times 10^{-4}$ |
| $I.6.20$ | $\frac{e^{-\theta^{2}/2\sigma^{2}}}{\sqrt{2\pi\sigma^{2}}}$ | $\theta,\sigma$ | $\underline{1.42\times 10^{-2}}$ | $1.20\times 10^{-1}$ | $\bf 8.44\times 10^{-3}$ |
| $I.6.20b$ | $\frac{e^{-(\theta-\theta_{1})^{2}/2\sigma^{2}}}{\sqrt{2\pi\sigma^{2}}}$ | $\theta,\theta_{1},\sigma$ | $\underline{1.59\times 10^{-2}}$ | $1.16\times 10^{-1}$ | $\bf 4.99\times 10^{-3}$ |
| $I.8.4$ | $\sqrt{(x_{2}-x_{1})^{2}+(y_{2}-y_{1})^{2}}$ | $x_{1},x_{2},y_{1},y_{2}$ | $\bf 4.58\times 10^{-3}$ | $1.91\times 10^{-1}$ | $\underline{1.23\times 10^{-2}}$ |
| $I.9.18$ | $\frac{Gm_{1}m_{2}}{(x_{2}-x_{1})^{2}+(y_{2}-y_{1})^{2}+(z_{2}-z_{1})^{2}}$ | $G,m_{1},m_{2},x_{1},x_{2},y_{1},y_{2},z_{1},z_{2}$ | $\underline{4.87\times 10^{-3}}$ | $1.40\times 10^{-2}$ | $\bf 3.13\times 10^{-3}$ |
| $I.10.7$ | $\frac{m_{0}}{\sqrt{1-\frac{v^{2}}{c^{2}}}}$ | $m_{0},v,c$ | $\bf 2.04\times 10^{-2}$ | $3.22\times 10^{-1}$ | $\underline{1.46\times 10^{-1}}$ |
| $I.11.19$ | $x_{1}y_{1}+x_{2}y_{2}+x_{3}y_{3}$ | $x_{1},y_{1},x_{2},y_{2},x_{3},y_{3}$ | $\underline{3.37\times 10^{-2}}$ | $9.89\times 10^{-2}$ | $\bf 2.65\times 10^{-2}$ |
| $I.12.1$ | $\mu N_{n}$ | $\mu,N_{n}$ | $\underline{9.22\times 10^{-3}}$ | $3.34\times 10^{-1}$ | $\bf 7.17\times 10^{-3}$ |
| $I.12.2$ | $\frac{q_{1}q_{2}}{4\pi\epsilon r^{2}}$ | $q_{1},q_{2},\epsilon,r$ | $\underline{6.75\times 10^{-3}}$ | $4.75\times 10^{-2}$ | $\bf 3.06\times 10^{-3}$ |
| $I.12.4$ | $\frac{q_{1}}{4\pi\epsilon r^{2}}$ | $q_{1},\epsilon,r$ | $\underline{5.62\times 10^{-3}}$ | $4.87\times 10^{-2}$ | $\bf 3.86\times 10^{-3}$ |
| $I.12.5$ | $q_{2}E_{f}$ | $q_{2},E_{f}$ | $\bf 2.93\times 10^{-3}$ | $3.25\times 10^{-1}$ | $\underline{3.61\times 10^{-3}}$ |
| $I.12.11$ | $q(E_{f}+Bv\sin(\theta))$ | $q,E_{f},B,v,\theta$ | $\underline{6.38\times 10^{-2}}$ | $1.85\times 10^{-1}$ | $\bf 3.56\times 10^{-2}$ |
| $I.13.4$ | $\frac{1}{2}m(v^{2}+u^{2}+w^{2})$ | $m,v,u,w$ | $\underline{2.10\times 10^{-2}}$ | $1.26\times 10^{-1}$ | $\bf 9.68\times 10^{-3}$ |
| $I.13.12$ | $Gm_{1}m_{2}\left(\frac{1}{r_{2}}-\frac{1}{r_{1}}\right)$ | $G,m_{1},m_{2},r_{1},r_{2}$ | $\bf 8.69\times 10^{-3}$ | $3.87\times 10^{-2}$ | $\underline{9.78\times 10^{-3}}$ |
| $I.14.3$ | $mgz$ | $m,g,z$ | $\underline{8.98\times 10^{-3}}$ | $1.64\times 10^{-1}$ | $\bf 2.80\times 10^{-3}$ |
| $I.14.4$ | $\frac{1}{2}k_{s}x^{2}$ | $k_{s},x$ | $\bf 5.13\times 10^{-3}$ | $1.11\times 10^{-1}$ | $\underline{6.79\times 10^{-3}}$ |
| $I.15.3x$ | $\frac{x-ut}{\sqrt{1-\frac{u^{2}}{c^{2}}}}$ | $x,u,t,c$ | $\bf 3.50\times 10^{-2}$ | $3.48\times 10^{-1}$ | $\underline{8.52\times 10^{-2}}$ |
| $I.15.3t$ | $\frac{t-ux/c^{2}}{\sqrt{1-\frac{u^{2}}{c^{2}}}}$ | $t,u,x,c$ | $\bf 3.69\times 10^{-2}$ | $3.44\times 10^{-1}$ | $\underline{7.18\times 10^{-2}}$ |
| $I.15.10$ | $\frac{m_{0}v}{\sqrt{1-\frac{v^{2}}{c^{2}}}}$ | $m_{0},v,c$ | $\underline{2.36\times 10^{-2}}$ | $2.27\times 10^{-1}$ | $\bf 1.47\times 10^{-2}$ |
| $I.16.6$ | $\frac{u+v}{1+\frac{uv}{c^{2}}}$ | $u,v,c$ | $\bf 8.73\times 10^{-3}$ | $1.45\times 10^{-1}$ | $\underline{1.06\times 10^{-2}}$ |
| $I.18.4$ | $\frac{m_{1}r_{1}+m_{2}r_{2}}{m_{1}+m_{2}}$ | $m_{1},r_{1},m_{2},r_{2}$ | $\bf 6.18\times 10^{-3}$ | $2.33\times 10^{-1}$ | $\underline{2.26\times 10^{-2}}$ |
| $I.18.5$ | $rF\sin(\theta)$ | $r,F,\theta$ | $\underline{5.67\times 10^{-2}}$ | $2.03\times 10^{-1}$ | $\bf 4.93\times 10^{-2}$ |
| $I.18.16$ | $mrv\sin(\theta)$ | $m,r,v,\theta$ | $\underline{6.88\times 10^{-2}}$ | $1.02\times 10^{-1}$ | $\bf 3.40\times 10^{-2}$ |
| $I.24.6$ | $\frac{1}{4}m(\omega^{2}+\omega_{0}^{2})x^{2}$ | $m,\omega,\omega_{0},x$ | $\underline{7.99\times 10^{-3}}$ | $6.20\times 10^{-2}$ | $\bf 5.87\times 10^{-3}$ |
| $I.25.13$ | $\frac{q}{C}$ | $q,C$ | $\underline{1.07\times 10^{-2}}$ | $5.17\times 10^{-1}$ | $\bf 8.33\times 10^{-3}$ |
| $I.26.2$ | $\arcsin(n\sin(\theta_{2}))$ | $n,\theta_{2}$ | $\underline{2.74\times 10^{-2}}$ | $4.45\times 10^{-1}$ | $\bf 1.15\times 10^{-2}$ |
| $I.27.6$ | $\frac{1}{1/d_{1}+n/d_{2}}$ | $d_{1},d_{2},n$ | $\bf 5.97\times 10^{-3}$ | $1.42\times 10^{-1}$ | $\underline{6.18\times 10^{-3}}$ |
| $I.29.4$ | $\frac{\omega}{c}$ | $\omega,c$ | $\underline{5.27\times 10^{-3}}$ | $2.26\times 10^{-1}$ | $\bf 3.45\times 10^{-3}$ |
| $I.29.16$ | $\sqrt{x_{1}^{2}+x_{2}^{2}-2x_{1}x_{2}\cos(\theta_{1}-\theta_{2})}$ | $x_{1},x_{2},\theta_{1},\theta_{2}$ | $\underline{8.48\times 10^{-2}}$ | $2.91\times 10^{-1}$ | $\bf 5.31\times 10^{-2}$ |
| $I.30.3$ | $I_{0}\frac{\sin^{2}(n\theta/2)}{\sin^{2}(\theta/2)}$ | $I_{0},n,\theta$ | $\underline{2.24\times 10^{-1}}$ | $4.07\times 10^{-1}$ | $\bf 1.99\times 10^{-1}$ |

The results from Table 2 demonstrate that MLP-KAN significantly outperforms both MLP and KAN across a variety of equations. or simpler equations like I.6.20a, MLP-KAN achieves an RMSE of $3.87\times 10^{-4}$, which is much lower than KAN’s $8.82\times 10^{-4}$ and MLP’s $1.37\times 10^{-1}$. This illustrates our method’s ability to accurately capture basic functional relationships with far fewer errors than MLP, which often over-parameterizes for simple tasks. For more complex equations involving multiple variables, such as I.9.18, MLP-KAN maintains a strong advantage, achieving an RMSE of $3.13\times 10^{-3}$ compared to KAN’s $4.87\times 10^{-3}$ and MLP’s much higher $1.40\times 10^{-2}$. This shows that our MLP-KAN scales effectively and can manage the intricacies of complex interactions that MLP struggles to capture without excessive parameters. Our proposed MLP-KAN demonstrates versatility across different types of equations, such as in I.12.5, where it achieves a lower RMSE ($3.61\times 10^{-3}$) than both KAN and MLP. The results reflect its ability to adapt dynamically to different functional forms, from basic algebraic equations to those involving physical constants and nonlinearities. n physics-based equations like I.15.3t, which involves relativistic transformations, MLP-KAN outperforms both KAN and MLP with an RMSE of $7.18\times 10^{-2}$ compared to KAN’s $3.69\times 10^{-2}$ and MLP’s $3.44\times 10^{-1}$. This indicates the superior ability of our method to generalize across equations that require deep understanding of physical laws. Our proposed achieves superior performance without the excessive parameter overhead required by MLPs, making it computationally efficient. For example, in I.14.4, MLP-KAN achieves an RMSE of $6.79\times 10^{-3}$, far outperforming MLP’s $1.11\times 10^{-1}$, demonstrating that MLP-KAN can achieve better accuracy with fewer resources. Across almost all equations, MLP-KAN consistently outperforms both KAN and MLP, often achieving RMSEs that are orders of magnitude smaller. This consistent superiority highlights MLP-KAN ’s versatility and adaptability to both simple and complex mathematical forms, making it the most robust and efficient solution for function learning across diverse domains.

### 5.3 Representation Learning

**Table 3: Comparison of results in representation learning. Results highlighted in bold represent the best performance in the comparison, while those underlined represent the second-best results.**
| Method | Dataset: CIFAR-10 |  | Dataset: CIFAR-100 |  | Dataset: mini-ImageNet |  | Dataset: SST2 |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Acc1 | Acc5 |  | Acc1 | Acc5 |  | Acc1 | Acc5 |  | Acc | F1 |  |
| KAN | 0.904 | 0.989 |  | 0.731 | 0.933 |  | 0.623 | 0.803 |  | 0.925 | 0.925 |
| MLP | 0.922 | 0.997 |  | 0.752 | 0.958 |  | 0.680 | 0.845 |  | 0.931 | 0.930 |
| MLP-KAN | 0.920 | 0.996 |  | 0.750 | 0.952 |  | 0.679 | 0.843 |  | 0.935 | 0.933 |

As shown in Table 3, our proposed MLP-KAN shows consistent high performance, demonstrating particular strengths across diverse datasets.
Notably, MLP-KAN achieves the second-best results for both top-1 and top-5 accuracy metrics on CIFAR-10, with scores of 0.920 and 0.996, respectively, closely trailing the MLP method. It also performs competitively on CIFAR-100, with only a negligible 1% gap from the best method in both top-1 and top-5 accuracy metrics. Furthermore, MLP-KAN consistently outperforms KAN, which achieves an Acc1 of 0.904 for CIFAR-10 and 0.731 for CIFAR-100.
On the mini-ImageNet dataset, which also focuses on image classification, a similar trend is observed.
In addition, MLP-KAN excels in the NLP task on the SST2 dataset, achieving the best results with an accuracy of 0.935 and an F1 score of 0.933.
This superior performance highlights MLP-KAN’s versatility and robustness in handling not only image data but also text data, making it an excellent choice for representation learning.

### 5.4 Ablation and Analysis

#### Number of Experts.

In this ablation study, we investigate the impact of the number of experts in the MoE component of MLP-KAN on the performance of CIFAR-10 and CIFAR-100. As observed in Table 4, increasing the number of experts from 4 to 10 yields steady improvements in both top-1 and top-5 accuracy across both datasets. Notably, the top-1 accuracy for CIFAR-10 increases from 0.908 to 0.928, while CIFAR-100 improves from 0.742 to 0.755 when the number of experts increases from 4 to 10. However, performance gains begin to diminish after using 8 experts. The difference between using 8 and 10 experts is marginal: The accuracy of the top-1 of CIFAR-10 only increases by 0.8%, and CIFAR-100 sees a mere 0.5% improvement. While the model with 10 experts delivers slightly better results, the computational cost associated with using more experts becomes significant. Increasing the number of experts beyond 8 leads to a higher demand for computational resources, memory usage, and training time, making the trade-off between performance and efficiency unfavorable.

**Table 4: Results of CIFAR-10 and CIFAR-100 accuracy with different numbers of experts.**
| Expert | CIFAR-10 (Acc1) | CIFAR-10 (Acc5) | CIFAR-100 (Acc1) | CIFAR-100 (Acc5) |
| --- | --- | --- | --- | --- |
| 8 | 0.920 | 0.996 | 0.750 | 0.953 |
| \cdashline1-5<br>4 | 0.908 | 0.990 | 0.742 | 0.950 |
| 6 | 0.914 | 0.996 | 0.740 | 0.952 |
| 10 | 0.928 | 0.997 | 0.755 | 0.958 |

#### Number of Top-K.

In this ablation study, we examine the impact of varying the Top-K value on the accuracy of CIFAR-10 and CIFAR-100. As shown in Table 5, we experiment with Top-K values of 1, 2, and 3, measuring their impact on both top-1 and top-5 accuracy across both datasets. Interestingly, we observe that setting Top-K to 2 yields the best performance. For CIFAR-10, both top-1 and top-5 accuracies improve slightly compared to K=1. Specifically, the top-5 accuracy increases from 0.990 to 0.996, while top-1 remains constant at 0.920. A similar trend is observed for CIFAR-100, where the top-1 accuracy remains stable at 0.750, but top-5 accuracy improves slightly from 0.952 to 0.953. On the other hand, when Top-K is set to 3, we notice a decline in performance. Both CIFAR-10 and CIFAR-100 exhibit reduced accuracy, with CIFAR-10 top-1 accuracy dropping to 0.908 and CIFAR-100 top-1 accuracy falling to 0.742. This indicates that increasing Top-K beyond 2 leads to diminished returns, as the additional experts likely introduce more noise or less relevant expertise.

**Table 5: Results of CIFAR-10 and CIFAR-100 accuracy with different Top-k values.**
| Top-k | CIFAR-10 (Acc1) | CIFAR-10 (Acc5) | CIFAR-100 (Acc1) | CIFAR-100 (Acc5) |
| --- | --- | --- | --- | --- |
| 2 | 0.920 | 0.996 | 0.750 | 0.953 |
| \cdashline1-5<br>1 | 0.920 | 0.990 | 0.750 | 0.952 |
| 3 | 0.908 | 0.991 | 0.742 | 0.949 |

## 6 Conclusion

In this paper, we propose a novel approach that effectively enhances both representation learning and function learning. This approach demonstrates excellent performance when integrated with MLP and KAN experts. Additionally, our proposed MLP-KAN can seamlessly replace the existing MLP layers in the transformer architecture. Furthermore, our extensive evaluations confirm that MLP-KAN significantly improves performance in each area.

## Appendix A Additional Implementation Details

Building on the transformer architecture, the input initially passes through the attention layer, where the number of attention heads is set to 8. Furthermore, our proposed MLP-KAN replaces the original MLP layer and consists of 8 experts (4 MLP experts and 4 KAN experts), with 2 experts dynamically selected for computation in each forward pass.
Subsequently, an additive residual connection is applied before the attention and MLP-KAN layers. We also use the normalization layer to ensure a consistent numerical distribution across different feature dimensions. This improves both the stability during training and the overall performance of the model.
We utilized a structure with 12 identical layers. To enhance model generalization, we employ Stochastic Depth (Huang et al., ), which randomly drops certain layers during training.
The process is as follows:

- •
Step 1: Tokenize the input $\mathbf{X}$ into tokens $\mathbf{X}_{i}$:
$\mathbf{X}=[\mathbf{X}_{1},\mathbf{X}_{2},\ldots,\mathbf{X}_{m}];$
- •
Step 2: Apply the multi-head self-attention mechanism (MHA) and layer normalization (LN), obtaining:
$\mathbf{X}^{\prime}=\mathrm{MHA}\left(\mathrm{LN}(\mathbf{X})\right)+\mathbf{X}$
- •
Step 3: Continue processing with MLP-KAN to obtain the following results:
$\mathbf{X}^{\prime\prime}=\mathrm{F}(\mathrm{LN}(\mathbf{X}^{\prime}))+\mathbf
{X}^{\prime}$

Typically, MLP-KAN, denoted as $\mathrm{F()}$, incorporates a Mixture of Experts (MoE) layer comprising multiple feed-forward networks (FFNs). These FFNs form a pool of experts $[\mathbf{e}_{1},\mathbf{e}_{2},\dots]$.
In this work, the MLP and KAN experts represent two distinct implementations within the FFN ensemble, together constituting the complete pool of experts. The gating mechanism, functioning as a linear layer, calculates the probability of each input token being assigned to a particular expert.
Based on the router’s output, the Top-K mechanism most probable experts are selected to process the input, and the outputs of these experts are weighted and summed to form the final result. The final representation is expressed as follows:

$$ $\mathcal{\alpha}_{i}(\mathbf{X})=\frac{\mathbf{e}^{g_{i}(\mathbf{X})}}{\sum_{j }^{E}\mathbf{e}^{g_{j}(\mathbf{X})}},$ $$

where $g(\mathbf{X})=\mathbf{W}\cdot\mathbf{X}$ represents the logit produced by the gate, and the weights are normalized via a softmax function to yield the assignment probabilities for each input token across the experts. Through the Top-K operation, K experts with the highest probabilities are selected to process each input token.

Each selected expert processes the input, and the outputs are weighted according to softmax probabilities. These are then aggregated into a weighted sum to produce the final output, which can be described as follows:

$$ $\text{F}(\mathbf{X})=\sum_{i=1}^{k}\mathcal{\alpha}_{i}(\mathbf{X})\cdot \mathbf{e}_{i}(\mathbf{X}).$ $$

This mechanism allows each token to be effectively processed by only a few relevant experts, thereby achieving efficient computation and expanding the model’s capacity.

## Appendix B Datasets

### B.1 CIFAR-10 Dataset

The CIFAR-10 dataset is a labeled subset of the 80 million tiny images dataset, containing 60,000 32x32 color images distributed across 10 mutually exclusive classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck. Each class contains 6,000 images, and the dataset is divided into 50,000 training images and 10,000 test images. The training images are split into five batches, each consisting of 10,000 images, while the test batch contains 10,000 randomly selected images. The dataset provides a diverse representation of objects, and the classes are non-overlapping; for instance, “automobile” includes small vehicles like sedans and SUVs, while “truck” includes only larger vehicles like big trucks.

Each image is represented by a 1x3072 array of pixel values, where the first 1024 entries correspond to the red channel, the second 1024 to the green channel, and the last 1024 to the blue channel, stored in row-major order. The dataset is widely used for image classification benchmarks, and baseline results using convolutional neural networks have achieved test error rates of 18% without data augmentation and 11% with augmentation. The dataset is commonly accessed in Python, Matlab, or binary formats, with convenient tools for loading and processing the images for machine learning tasks.
The structure of the CIFAR10 dataset as shown in Table 6.

**Table 6: CIFAR-10 Dataset Structure**
| Data | Shape | Description |
| --- | --- | --- |
| train_x | (50000, 32, 32, 3) | Training Samples |
| train_y | (50000, 1) | Training Labels |
| test_x | (10000, 32, 32, 3) | Testing Samples |
| test_y | (10000, 1) | Testing Labels |

### B.2 CIFAR-100 Dataset

The CIFAR-100 dataset shares the same general structure as CIFAR-10 but is more granular, containing 100 classes of objects, each represented by 600 images, with 500 training images and 100 test images per class. The dataset introduces a hierarchical structure where the 100 fine-grained classes are grouped into 20 superclasses (coarse labels). For example, the superclass “aquatic mammal” includes beaver, dolphin, otter, seal, and whale, while the superclass “vehicles 1” contains bicycle, bus, motorcycle, pickup truck, and train.

Similar to CIFAR-10, CIFAR-100 images are stored as 1x3072 arrays, with two label bytes for each image: one for the coarse label and one for the fine label. This dataset is often used for fine-grained classification tasks, presenting a more challenging problem due to its increased number of classes and hierarchical structure. Both the CIFAR-10 and CIFAR-100 datasets have been extensively used in the computer vision community for benchmarking the performance of image classification algorithms.
The structure of CIFAR-100 as shown in Table 7.

**Table 7: Classification Table**
| Category | Subcategory |
| --- | --- |
| Aquatic Mammals | Beaver, Dolphin, Otter, Seal, Whale |
| Fish | Aquarium Fish, Flounder, Ray, Shark, Trout |
| Flowers | Orchid, Poppy, Rose, Sunflower, Tulip |
| Food Containers | Bottle, Bowl, Can, Cup, Plate |
| Fruits and Vegetables | Apple, Mushroom, Orange, Pear, Bell Pepper |
| Household Appliances | Clock, Computer Keyboard, Lamp, Phone, TV |
| Household Furniture | Bed, Chair, Sofa, Table, Wardrobe |
| Insects | Bee, Beetle, Butterfly, Caterpillar, Cockroach |
| Large Carnivores | Bear, Leopard, Lion, Tiger, Wolf |
| Large Man-made Outdoor Things | Bridge, Castle, House, Road, Skyscraper |
| Large Natural Outdoor Scenes | Cloud, Forest, Mountain, Plain, Sea |
| Large Omnivores and Herbivores | Camel, Cow, Chimpanzee, Elephant, Kangaroo |
| Medium-sized Mammals | Fox, Porcupine, Opossum, Raccoon, Skunk |
| Non-insect Invertebrates | Crab, Lobster, Snail, Spider, Worm |
| People | Baby, Boy, Girl, Man, Woman |
| Reptiles | Crocodile, Dinosaur, Lizard, Snake, Turtle |
| Small Mammals | Hamster, Mouse, Rabbit, Shrew, Squirrel |
| Trees | Maple, Oak, Palm, Pine, Willow |
| Vehicles | Bicycle, Bus, Motorcycle, Van, Train |

### B.3 Feynman dataset

The Feynman dataset is a collection of physics equations sourced from the Feynman Lectures on Physics (Feynman, ), designed as a benchmark for symbolic regression tasks. It comprises 120 formulas, primarily drawn from classical physics, including key concepts from mechanics, electromagnetism, and thermodynamics. For our purposes, we focus on the Feynman_no_units subset, specifically equations involving at least two variables, which reduce to one-dimensional splines. An example is the relativistic velocity addition formula, $f(u,v)=\frac{u+v}{1+uv}$, where $u$ and $v$ are sampled from the range (-1, 1), and the network is trained to predict $f$ based on these inputs. The dataset serves to evaluate the ability of neural networks and other symbolic regression methods to model and predict underlying physical laws from empirical data.

### B.4 Mini-InmageNet DATASET

Mini-Imagenet is a small-scale dataset extracted from the ImageNet dataset by the Google DeepMind team in 2016, primarily used for research in the field of few-shot learning. The total size of the dataset is approximately 3GB and contains 60,000 images divided into 100 classes, with 600 images per class. These images are of varying sizes and are saved in .jpg format.

Compared to the full ImageNet dataset, Mini-Imagenet significantly reduces the data volume, making it more accessible for researchers with limited hardware resources. It is suitable for rapid prototyping and evaluating a model’s classification performance, especially in few-shot learning scenarios.

The dataset is structured as follows:

**Table 8: Mini-Imagenet Dataset Structure**
| Directory | Description |
| --- | --- |
| mini-imagenet/ | Root directory of the dataset |
| images/ | Folder containing all the images |
| train.csv | Label file for the training set |
| val.csv | Label file for the validation set |
| test.csv | Label file for the test set |

It is important to note that when this dataset was created, the labels were not evenly sampled from each class, which adds an additional challenge for models designed for few-shot learning. Researchers can use these CSV files to obtain image labels and perform training, validation, and testing.

### B.5 SST-2 DATASET

The Stanford Sentiment Treebank (SST) is a linguistically annotated dataset designed to enable detailed analysis of sentiment composition in natural language. Derived from movie reviews, this dataset includes 11,855 individual sentences, which were parsed into syntactic structures using the Stanford parser. The resulting parse trees consist of 215,154 unique phrases, all annotated by human judges to capture nuanced sentiment at various granularities.

A distinctive feature of the SST dataset is its ability to support research on compositional sentiment analysis, as each sub-phrase in a sentence is independently labeled for sentiment. This allows for a deeper understanding of how sentiment is constructed and expressed through the combination of linguistic elements.

In the context of binary sentiment classification tasks, a simplified version of the dataset, known as SST-2, is often used. In SST-2, neutral sentences are excluded, and the remaining sentences are categorized into either negative or positive classes. This binary classification setup has become a widely adopted benchmark for evaluating sentiment analysis models.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 1
presentation: 1
questions: I think the paper is insufficiently clear to support proper review, and therefore I don't have any questions.
soundness: 1
strengths: 1) The premise of the paper is potentially reasonable; there is evidence in the literature that KANs and MLPs are complementary, and developing a method that adaptively selects which modeling strategy is best for a given problem is a good idea.
summary: The authors hypothesize that KAN networks and MLPs are effective for solving different types of problems: specifically, MLPs are good for representation learning, while KANs are good for function learning.  They propose a modeling strategy that includes both MLPs and KANs, which are adaptively selected based upon the problem setting.
weaknesses: 1) The presentation of the paper is not sufficiently clear to facilitate review.   The paper is generally vague, contains many typos, and the methodology is ultimately unclear.  I provide examples

(ii) Poor presentation.  In Line 47 in the 2nd paragraph of the paper, the authors define KAN as Kernel Attention Network, yet the whole paper seems to be about Kolmogorov-Arnold Network.  Line 51 and 73 in the introduction are nearly identical, and repeat the same idea. 

(ii) Incorrect/unclear methodological description.  For example, in Eq. (5) the dimensions of W and X are incompatible.  The entire architecture of the proposed model is unclear.  For example, it is unclear whether there are multiple MLP-KAN layers?  The authors have a section about "Architecture" which then, without motivation, discusses self attention.   The number of layers in the MLP-KAN are never provided (although it is unclear if there are multiple layers), nor is the overall size of the model discussed. 

(iii) Vague motivation.  The whole premise of the paper is not clearly explained.  While the idea of combining KANs and MLPs seems reasonable, the authors repeatedly argue a theoretical motive based upon MLPs being "representation learning" methods while KANs are "function learning".  This premise for the proposed approach is repeatedly mentioned throughout the paper, yet the difference between these two approaches is never precisely described.  
 
2) Insufficient Experiments.  The experiments are insufficient to demonstrate the efficacy of the proposed approach.  To the best I can discern, the proposed method would significantly increase the number of modeling parameters because we now have multiple KANs and MLPs in a single model, along with some parameters to select among them.  However, the resulting model often performs similarly to other models only composed of a single KAN or MLP architecture (e.g., Table 3).  What would happen if we simply used a single MLP or single KAN model that is the same size (in terms of free parameters) to the MLP-KAN?  Or what if we made a simple fusion model where we interlaced MLP and KAN layers, or added a few KAN layers to the end of a standard MLP?  How do we know whether this more complex architecture proposed by the authors is superior to simpler and/or smaller models?

---

confidence: 4
contribution: 4
initial_rating: 6
presentation: 4
questions: 1. How does MLP-KAN perform in the presence of noisy or adversarial inputs compared to other models? Are there any robustness benchmarks included in the evaluation?

2. What optimization algorithms and strategies were employed to train MLP-KAN effectively? Were any specific techniques used to balance the training of multiple experts?

3. Were all models, including baselines, trained under the same conditions to ensure fair comparisons? What datasets splits, augmentation techniques, and training epochs were used?

4. How sensitive is MLP-KAN to changes in hyperparameters other than the number of experts and Top-K values? For example, how do variations in learning rates, network depth, or activation functions affect performance?

5. Were there any challenges related to training stability when combining MLPs and KANs within the MoE framework? How were these challenges addressed?

6. How does the inclusion of MLP-KAN affect the standard attention mechanisms within transformers? Are there any changes to how attention weights are applied?

7. How effective is MLP-KAN in transfer learning where it is fine-tuned on different tasks after initial source pre-training?

8. What is the computational cost of MLP-KAN compared to other architecture with MLPs (e.g., transfomer) or KANs alone? How does the addition of multiple experts affect training and inference times?

9. How interpretable are the latent features generated by MLP-KAN? Are there any visualizations demonstrating the semantic captured by the model (e.g., t-SNE visualizations)?

10. What future research directions do the authors suggest to address the current limitations or to further enhance the capabilities of MLP-KAN?
soundness: 4
strengths: 1. The manuscript is well-written, presenting clear motivations and providing step-by-step derivations of the proposed method.

2. Combining MLPs and KANs within an MoE framework is interesting. Moreover, integrating this block into a transformer architecture develops a robust backbone that effectively extracts and integrates features across various data modalities.

3. The ablation studies demonstrate that the proposed method can scale easily by increasing the number of experts, which enhances performance without introducing too much computational burdens.

4. The proposed method is extensively evaluated on multiple CV and NLP datasets, demonstrating its versatility for diverse AI applications and practical values.
summary: This manuscript introduces MLP-KAN, a unified block that combines representation and function learning within a single framework. By using MLPs for representation learning and KANs for function learning in a mixture-of-experts setup, MLP-KAN adapts to various tasks and data modalities. When integrated into a transformer-based architecture, MLP-KAN demonstrates both versatility and robustness across multiple data domains. The proposed method is evaluated on four datasets, CIFAR-10, CIFAR-100, mini-ImageNet, and SST2, showing strong performance in both image classification and natural language processing tasks.

############################ Post Rebuttal ############################

All of my concerns have been addressed during rebuttal. I am happy to raise my score from 6 to 8.

############################ Post Rebuttal ############################
weaknesses: 1. Although the proposed technique is shown to be generalizable to different tasks, its effectiveness in other types of tasks or with different types of data (e.g., time-series, reinforcement learning) remains unexplored.

2. Using multiple experts in an MoE architecture, especially with higher Top-K values, can significantly increase computational resource requirements. I suggest that the authors conduct ablation studies on runtime complexity and compare their proposed method with the standard transformer architecture.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: * Is the number of tasks connected to optimal k for topk?
* Is the number of experts for MLP and parameters the same across experiments with MLP-KAN?
* Are all experiments for Tables 2 and 3 trained together as a multi-task scenario? Or are experiments for Tables 2 and 3 separated?

Suggestion:

* It would be great to add more details about experiments and summaries in section 4.1.  
* I think it improves the justifiability of the paper if they provide an ablation on the router gating to show how it assigns tokens to each expert.
soundness: 2
strengths: * The paper is well-written for the most part.
* The motivation of the paper is valid and exciting. 
* The idea is simple but yet effective.
summary: This paper proposes MLP-KAN, a mixture of experts method, to combine MLP and KAN. The authors claim their method can address the shortcomings of both KAN and MLP in one structure and solve the scalability problem of MLPs. Furthermore, they show the superior performance of their method across different tasks and datasets compared to other methods and investigate the effectiveness of various components of their method.
weaknesses: * The discussion part of the paper heavily relies on the description of Multi-expert and router gating, which is not the paper's contribution. More discussion or experiments are needed to show why combining MLP and KAN should improve the performance. 
* One of the paper's main points is scalability, but experiments about scalability, computation time, and memory are missing. 

Minor Weakness:

* The numbers in all Tables don't have confidence intervals, so it is hard to grasp the significance of the differences.  The authors should include confidence intervals or standard deviations from multiple runs.
* Details about competitors are missing in the experimental setup.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: This paper contributes to advancing the KAN model, though it requires some clarifications on theory and experiments. Given these clarifications, if provided in an author's response, I would consider increasing the score. 

For the theory, there are a few steps that need clarification and further clarification on novelty. 

1. KAN has advantages in model scaling under certain constraint conditions but not for all learning tasks. The KAN model obeys the Error Scaling formula $\\| f - (KAN) \\|\_{C^m} \le C G^{-k-1+m}$ and the scaling law $l \\propto N^{-\\alpha}$, but the author did not clarify the relationship between grid size $G$ and the input space dimension $n$. As a function approximation method, usually, the number of grid points $G$ is directly proportional to $G \propto I^n$ ($I+1$ is the number of intervals on each dimension), which makes $\\| f - (KAN) \\|_{C^m} \le C G^{(-k-1+m)/n}$ and triggers a serious curse of dimensionality (COD) problem (see KAN [1], Fig. 3.1, $f(x_1, \cdots, x\_{100}) = \text{exp}(\frac{1}{100} \sum\_{i=1}^{100} \sin^2{\frac{\pi x_i}{2}})$). To address this problem, KAN authors hypothesize that the objective high-dimensional function is smooth and has sparse compositional structure to reduce the number of grid nodes $G \ll I^n$. The authors did not provide the scaling law of MLP-KAN in the paper. Therefore, I don't know whether MLP-KAN overcomes the inherent limitations of KAN.

2. This is particularly called into question due to the integration of KAN and ViT, since KAN and ViT usually exhibit different behavior on datasets of varying sizes. At present, KAN is suitable for low-dimensional function learning, and the dataset is generally small, with only a few thousand samples. However, ViTs are particularly powerful on large datasets (e.g., ImageNet) and tend to underperform relative to convolutional models on small datasets. Empirically, KAN and Transformer have potential conflicts.

For the experiments, the following should be addressed.

1. It would have been better also to show the performance of learning the **non-smooth** or **high-dimensional functions**. The Feynman Equations may be too simple for conventional function approximation methods. You can try $f(x)=\frac{1}{x} \\ \sin{\frac{1}{x}}$ and $f(x_1, \cdots, x_{100}) = \sum_{i=1}^{99} \sin{(x_i + x\_{100-i})}$; testing these functions would indicate if MLP-KAN overcomes certain limitations of KAN.

2. In our testing, we found that KAN's training process differs from neural network models and is considerably slower. Comparing wall-clock training time could reveal any potential efficiency advantages.

3. The central contribution focuses on enhancing Vision Transformer performance on CIFAR. It would be beneficial to compare with the Vision Transformer baseline on larger datasets like ImageNet-1K, which would add value.


---

[1] Liu, Z., Wang, Y., Vaidya, S., Ruehle, F., Halverson, J., Soljačić, M., ... & Tegmark, M. (2024). Kan: Kolmogorov-arnold networks. arXiv preprint arXiv:2404.19756.
soundness: 4
strengths: 1. The paper proposes KAN-MLP, a novel approach to enhance Vision Transformers (ViT) for function learning and CIFAR classification tasks.

2. It introduces the explainable KAN architecture into the ViT model, which could improve interoperability.

3. The paper compares KAN, MLP, and MLP-KAN in function learning, showing potential advantages of MLP-KAN over other architectures.

4. An ablation study on CIFAR demonstrates that the proposed model improves on the plain ViT model, highlighting the effectiveness of the enhancements.
summary: This paper proposes a method, KAN-MLP, for improving the performance of ViT on function learning and CIFAR classification tasks. In order to achieve that, they replaced the traditional MLP layers within the transformer architecture. In this method, experts are dynamically selected based on the input through a gating mechanism, ensuring efficient routing of tokens to the most relevant experts. Their main contribution is applying an explainable KAN architecture to the Vision Transformer model. Comparing KAN, MLP, and MLP-KAN on function learning tasks, they show that MLP-KAN performs better than the other architectures in certain cases. Additionally, they also perform an ablation study on the CIFAR dataset to demonstrate that their extended model outperforms the naive ViT.
weaknesses: 1. The scaling law for MLP-KAN is missing, making it difficult to assess if MLP-KAN overcomes KAN’s limitations.

2. The method may still suffer from the curse of dimensionality (COD), and the paper does not address how MLP-KAN performs for learning non-smooth and high-dimensional functions.

3. There is a potential conflict between KAN’s suitability for low-dimensional tasks and ViT’s insuitability for small datasets. The experiments are limited to smooth functions and CIFAR datasets. Testing on larger datasets like ImageNet or MSCOCO would provide a more comprehensive view of the model’s performance relative to the Vision Transformer baseline.

Meta-review: The authors claim that KAN networks and MLPs are effective for solving different problems, i.e., MLPs are good for representation learning, while KANs are good for function learning, and they introduce MLP-KAN, a unified block that combines representation and function learning within a single framework. It received a mixture of comments, with one being strong reject. There was a long discussion with this reviewer and authors, and the reviewer didn't change his mind. This paper, after revisions, still has limited evidence to support the claim. I recommended a rejection.

Decision: Reject