KAN versus MLP on Irregular or Noisy Functions

Authors: Chen Zeng, Jiahui Wang, HaoranShen, Qiao Wang
Primary area: foundation or frontier models, including LLMs
Keywords: Kolmogorov-Arnold networks, Multi-layer Perceptrons, KAN, MLP, Irregularization, Noise

Abstract:
In this paper, we compare the performance of Kolmogorov-Arnold Networks (KAN) and Multi-Layer Perceptron (MLP) networks on irregular or noisy functions. We control the number of parameters and the size of the training samples to ensure a fair comparison. For clarity, we categorize the functions into six types: regular functions, continuous functions with local non-differentiable points, functions with jump discontinuities, functions with singularities, functions with coherent oscillations, and noisy functions. Our experimental results indicate that KAN does not always perform best. For some types of functions, MLP outperforms or performs comparably to KAN. Furthermore, increasing the size of training samples can improve performance to some extent. When noise is added to functions, the irregular features are often obscured by the noise, making it challenging for both MLP and KAN to extract these features effectively. We hope these experiments provide valuable insights for future neural network research and encourage further investigations to overcome these challenges.

Decision: reject
Reviewer scores: 1, 3, 3, 3

Reviews and discussion:
<thread>
creation date:
1730068500931

thread_type:
official_review

<thread_object>
creator: reviewer_fmgf

<content>
summary:
In this empirical study, the authors compare the performance of two ad-hoc versions of KAN and MLPs, both with identical parameter counts, in learning ten single-dimensional real functions, with and without added noise.

strengths:
The work has considered different classes of functions with some common irregularities. The empirical comparison is sound, and extensive plots provided let's the reader to compare the performance of tested KAN and MLPs in each case.

weaknesses:
The comparison primarily centers on the resulting test accuracy curves; however, it lacks the necessary theoretical justification and fundamental analysis to substantiate the findings.

While the authors have structured the text well, the plots are somewhat cluttered and could be presented more effectively for better clarity. 

Overall, the work appears basic and does not demonstrate the level of novelty typically expected from submissions to ICLR.

questions:
If authors can provide some theoretical insights backing the observed empirical results in all or some of the function classes tested, it would make the work more promising and considerable for this conference.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730190245176

thread_type:
official_review

<thread_object>
creator: reviewer_ncqp

<content>
summary:
The paper conducts a comparative analysis of experiments between MLP and KANs, discussing the outcomes. It challenges the assumption that KANs consistently outperform MLP in modeling mathematical equations, particularly with irregular functions. The experiments involve applying MLP and KAN to various functions—regular, non-differentiable, discontinuous, singular, and coherent oscillation, with and without noise. These functions are single input and single output. Variations include different training sample sizes, iteration counts, and optimizers. The findings demonstrate that KANs do not always surpass MLP.

While this paper serves as a great exploration to KANs and does establish that KANs are not invariably superior to MLP, it falls short by only providing experimental evidence without introducing new theoretical insights or network structures, thus lacking substantial academic contribution.

strengths:
The structure of the paper is clear and well-organized.
The experimental results are clearly presented.
The experiments validate that KANs are not consistently superior to MLP.

weaknesses:
The experiments could be designed more targeted. For instance, in the experiments for non-differentiability, both functions feature only a single non-differentiable point. A comparison between functions with single versus multiple non-differentiable points would be more insightful, given the focus on the impact of these points.

The discussion lacks depth. Given the simplicity of both the functions and network structures used, there is potential for a more detailed examination of how parameters are trained and the reasons behind specific outcomes.

The discussion section does not yield any intriguing or unexpected conclusions, nor does it propose any novel theories or structures.

questions:
Given that the structures of both MLP and KANs are well-known, a deeper analysis of their capabilities and limitations in the related work section would be beneficial. More thorough research could uncover more significant findings. For instance, some limitations of KANs identified in the paper are not due to the Kolmogorov-Arnold Theorem but rather due to B-spline, a critical component in KANs that is not discussed in the paper at all.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730691371722

thread_type:
official_review

<thread_object>
creator: reviewer_pbqy

<content>
summary:
Authors compare the performance of Kolmogorov-Arnold Networks (KAN) and Multi-Layer Perceptron (MLP) networks on irregular or noisy functions. The author experimentally demonstrated that KAN does not always outperform MLP.

strengths:
- The author compared KAN and MLP on various irregular and noisy functions and experimentally demonstrated in which cases KAN is worse than MLP.

weaknesses:
- The author merely compared KAN and MLP experimentally but did not analyze why KAN or MLP performs poorly in certain situations.

- The author experimentally demonstrated that KAN is sometimes inferior to MLP. It would be better to propose a new, improved KAN model to address this.

- There are no experiments on high-dimensional functions. In one dimension, both KAN and MLP are likely to approximate well to some extent, but more experiments are needed to explore how they perform in high-dimensional spaces with irregular points.

- If the experiments are conducted only on univariate functions, many models besides MLP can be compared with KAN. It would be beneficial to include other models commonly used in machine learning in the experiments.

questions:
I do not have a complete understanding of KAN, but I think KAN appears to be a generalization of projection pursuit regression. Is this correct?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730696628909

thread_type:
official_review

<thread_object>
creator: reviewer_idcx

<content>
summary:
This work empirically compares Kolmogorov-Arnold Networks with Multi-Layer Perceptron on
learning irregular or noisy functions. The experiment results show that KAN do not always perform
the best.

strengths:
Experiment codes are provided for reproducibility.

Do provide some insight on what KAN may be good at modeling.

weaknesses:
The finding is purely empirical.

The paper does not clearly state the experiment setting in the main text.

The experiment does not provide conclusive results.

The experiment only tries to fit relatively simple functions. The result may not be relevant to real-world problems.

questions:
It is possible to include more challenging problems for comparison? It is well established that MLP can model fairly complicated functions.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper compares the performance of Kolmogorov-Arnold Networks and Multi-Layer Perceptrons in approximating various types of functions, including irregular and noisy ones. Experiments demonstrate that while KANs often outperform MLPs for certain regular or singular functions, MLPs show better performance for functions with localized irregularities or jumps. 

The paper is motivated by an interesting question but lacks sufficient evidence and deep discussion. It also offers no new interpretations, theoretical implications, or methodological suggestions for improving the problem. The reviewers agree that much can still be done with this paper.