Title: value: KAN versus MLP on Irregular or Noisy Functions
Authors: value: Chen Zeng
Jiahui Wang
HaoranShen
Qiao Wang
Venue: ICLR 2025
Keywords: value: Kolmogorov-Arnold networks
Multi-layer Perceptrons
KAN
MLP
Irregularization
Noise

Abstract: value: In this paper, we compare the performance of Kolmogorov-Arnold Networks (KAN) and Multi-Layer Perceptron (MLP) networks on irregular or noisy functions. We control the number of parameters and the size of the training samples to ensure a fair comparison. For clarity, we categorize the functions into six types: regular functions, continuous functions with local non-differentiable points, functions with jump discontinuities, functions with singularities, functions with coherent oscillations, and noisy functions. Our experimental results indicate that KAN does not always perform best. For some types of functions, MLP outperforms or performs comparably to KAN. Furthermore, increasing the size of training samples can improve performance to some extent. When noise is added to functions, the irregular features are often obscured by the noise, making it challenging for both MLP and KAN to extract these features effectively. We hope these experiments provide valuable insights for future neural network research and encourage further investigations to overcome these challenges.


Review discussion:
The work has considered different classes of functions with some common irregularities. The empirical comparison is sound, and extensive plots provided let's the reader to compare the performance of tested KAN and MLPs in each case.
The comparison primarily centers on the resulting test accuracy curves;
however, it lacks the necessary theoretical justification and fundamental analysis to substantiate the findings.
While the authors have structured the text well, the plots are somewhat cluttered and could be presented more effectively for better clarity.
Overall, the work appears basic and does not demonstrate the level of novelty typically expected from submissions to ICLR.
If authors can provide some theoretical insights backing the observed empirical results in all or some of the function classes tested, it would make the work more promising and considerable for this conference.
Yes, Other reasons (please specify below)
The authors cite a paper with "our work" in the first page, fourth paragraph of Introduction. This seems to be an instance of violation of anonymity of the authors in the double-blind review process.
Thank you for your valuable time and insightful suggestions. We have carefully considered and thoroughly discussed your suggestions. Below, we provide our detailed responses to your comments.
> Q1: It lacks the necessary theoretical justification and fundamental analysis to substantiate the findings.
A1: This paper is experimental rather than theoretical, aiming to compare the performance of KAN and MLP in fitting functions with poor regularity through experiments.
> Q2: The authors cite a paper with "our work" in the first page, fourth paragraph of Introduction. This seems to be an instance of violation of anonymity of the authors in the double-blind review process.
A2: Thank you for your careful reading. We deeply apologize for this, it was indeed our oversight. We have corrected this error in the new version.
The structure of the paper is clear and well-organized.
The experimental results are clearly presented.
The experiments validate that KANs are not consistently superior to MLP.
The experiments could be designed more targeted.
For instance, in the experiments for non-differentiability, both functions feature only a single non-differentiable point. A comparison between functions with single versus multiple non-differentiable points would be more insightful, given the focus on the impact of these points.
The discussion lacks depth. Given the simplicity of both the functions and network structures used, there is potential for a more detailed examination of how parameters are trained and the reasons behind specific outcomes.
The discussion section does not yield any intriguing or unexpected conclusions, nor does it propose any novel theories or structures.
Given that the structures of both MLP and KANs are well-known, a deeper analysis of their capabilities and limitations in the related work section would be beneficial. More thorough research could uncover more significant findings.
For instance, some limitations of KANs identified in the paper are not due to the Kolmogorov-Arnold Theorem but rather due to B-spline, a critical component in KANs that is not discussed in the paper at all.
No ethics review needed.
Thank you for your valuable time and insightful suggestions. We have carefully considered and thoroughly discussed your suggestions. Below, we provide our detailed responses to your comments.
> Q1: A comparison between functions with single versus multiple non-differentiable points would be more insightful, given the focus on the impact of these points.
A1: Thank you for your suggestion. The aim of our paper is to demonstrate the potential issues KAN networks may encounter when handling functions with poor regularity. A detailed exploration of the impact of irregular points on KAN would require significant space and is better suited for a separate paper.
Therefore, we chose to use simpler functions to illustrate the phenomenon.
> Q2: The discussion lacks depth. Given the simplicity of both the functions and network structures used, there is potential for a more detailed examination of how parameters are trained and the reasons behind specific outcomes. And the discussion section does not yield any intriguing or unexpected conclusions, nor does it propose any novel theories or structures.
A2: This paper is experimental rather than theoretical, aiming to compare the performance of KAN and MLP in fitting functions with poor regularity through experiments.
> Q3: Some limitations of KANs identified in the paper are not due to the Kolmogorov-Arnold Theorem but rather due to B-spline, a critical component in KANs that is not discussed in the paper at all.
A3: Thank you for your suggestion. This is a very interesting idea.
However, to validate this claim, we might need to introduce "KAN" model using other basis functions for discussion, which is beyond the scope of our paper.
- The author compared KAN and MLP on various irregular and noisy functions and experimentally demonstrated in which cases KAN is worse than MLP.
- The author merely compared KAN and MLP experimentally but did not analyze why KAN or MLP performs poorly in certain situations.
- The author experimentally demonstrated that KAN is sometimes inferior to MLP. It would be better to propose a new, improved KAN model to address this.
- There are no experiments on high-dimensional functions. In one dimension, both KAN and MLP are likely to approximate well to some extent, but more experiments are needed to explore how they perform in high-dimensional spaces with irregular points.
- If the experiments are conducted only on univariate functions, many models besides MLP can be compared with KAN. It would be beneficial to include other models commonly used in machine learning in the experiments.
I do not have a complete understanding of KAN, but I think KAN appears to be a generalization of projection pursuit regression. Is this correct?
No ethics review needed.
Thank you for your valuable time and insightful suggestions. We have carefully considered and thoroughly discussed your suggestions. Below, we provide our detailed responses to your comments.
> Q1: The author merely compared KAN and MLP experimentally but did not analyze why KAN or MLP performs poorly in certain situations. And it would be better to propose a new, improved KAN model to address this.
A1: Thank you for your suggestion. This paper is experimental rather than theoretical, aiming to compare the performance of KAN and MLP in fitting functions with poor regularity through experiments.
> Q2: There are no experiments on high-dimensional functions.
A2: We aim to focus on poor regularity rather than the types of functions. And as demonstrated in our paper, even with simple functions, KAN shows certain problems.
> Q3: It would be beneficial to include other models commonly used in machine learning in the experiments.
A3: The primary neural networks under consideration are MLP and KAN, and when KAN was initially proposed, it was primarily compared to MLP.
Therefore, we believe it is unnecessary to include additional models in our study.
> Q4: I do not have a complete understanding of KAN, but I think KAN appears to be a generalization of projection pursuit regression. Is this correct?
A4: KAN and projection pursuit regression (PPR) do share some similarities. Both rely on projecting data into a lower-dimensional space and applying nonlinear functions to these projections for regression. PPR works by finding the best linear projections to reveal the structure of the data, and then fitting nonlinear functions to these projections.
In contrast, KAN networks are based on the Kolmogorov-Arnold theorem, representing complex multidimensional functions as a combination of several one-dimensional functions.
Therefore, KAN can be seen as a generalization and extension of PPR.
Experiment codes are provided for reproducibility.
Do provide some insight on what KAN may be good at modeling.
The finding is purely empirical.
The paper does not clearly state the experiment setting in the main text.
The experiment does not provide conclusive results.
The experiment only tries to fit relatively simple functions. The result may not be relevant to real-world problems.
It is possible to include more challenging problems for comparison? It is well established that MLP can model fairly complicated functions.
No ethics review needed.
Thank you for your valuable time and insightful suggestions. We have carefully considered and thoroughly discussed your suggestions. Below, we provide our detailed responses to your comments.
> Q1: The finding is purely empirical.
A1: This paper is experimental rather than theoretical, aiming to compare the performance of KAN and MLP in fitting functions with poor regularity through experiments.
> Q2: The paper does not clearly state the experiment setting in the main text.
A2: We did not include the experimental setting in the main text but provided them in the figure captions and tables.
> Q3: The experiment does not provide conclusive results.
A3: Our paper does not aim to prove that KAN is inherently better or worse than MLP.
Instead, it seeks to highlight potential issues KAN may encounter when fitting functions with poor regularity.
Besides, as demonstrated in the paper, KAN's performance varies across different types of functions.
Therefore, we opted to provide conclusions separately for each type of function.
> Q4: The experiment only tries to fit relatively simple functions. The result may not be relevant to real-world problems. Is it possible to include more challenging problems for comparison? It is well established that MLP can model fairly complicated functions.
A4: We aim to focus on poor regularity rather than the types of functions or tasks. As demonstrated in our paper, even with simple functions, KAN shows certain problems.

Meta-review:
role: AC
data: metareview: value: The paper compares the performance of Kolmogorov-Arnold Networks and Multi-Layer Perceptrons in approximating various types of functions, including irregular and noisy ones. Experiments demonstrate that while KANs often outperform MLPs for certain regular or singular functions, MLPs show better performance for functions with localized irregularities or jumps. 

The paper is motivated by an interesting question but lacks sufficient evidence and deep discussion. It also offers no new interpretations, theoretical implications, or methodological suggestions for improving the problem. The reviewers agree that much can still be done with this paper.
additional_comments_on_reviewer_discussion: value: Despite differences in detail, the reviewers were generally in agreement. The paper is limited to basic experimental results, and the experiments alone leave much room for more complex and realistic settings. There is also room for deeper consideration of theory, methodology, and discussion.
role: PC

Decision: Reject