UKAN: UNBOUNDED KOLMOGOROV-ARNOLD NETWORKS

Authors: Alireza Moradzadeh, Lukasz Wawrzyniak, Miles Macklin, Saee Gopal Paliwal
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: KAN, Acceleration, Unbounded KAN, Grid Free, Function Approximation

Abstract:
We present Unbounded Kolmogorov-Arnold Networks (UKANs), a novel algorithm that eliminates the need for bounded grids in traditional Kolmogorov-Arnold Networks (KANs). The key innovation is a coefficient generator (CG) model that dynamically produces B-spline coefficients, operating on an infinite symmetric grid. UKANs integrate multilayer-perceptrons with KANs, using positional encoding of grid groups as input to the CG model. This approach enables function approximation on unbounded domains without data normalization. Additionally, to reduce UKAN and KAN computational cost, we introduce a GPU-accelerated library that reduces B-spline evaluation complexity by a factor of $\mathcal{O}(\text{grid size})$ compared to existing libraries, enabling efficient large-scale learning. Our experiments on regression, classification, and generative tasks demonstrate UKANs' effectiveness, while benchmarks confirm superior memory and computational efficiency compared to existing methods. This work advances function approximation techniques, offering a flexible solution for complex, large-scale learning problems.

Decision: reject
Reviewer scores: 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730480341163

thread_type:
official_review

<thread_object>
creator: reviewer_ykyw

<content>
summary:
The paper presents a method to efficiently compute a node in KAN (Kolmogorov-Arnold Networks) using B-splines allowing unbounded coefficients that are generated using an MLP; KANs differ from MLPs by allowing non-linear operations on edges instead of nodes. They provide a GPU implementation that speeds up earlier methods by a factor of “grid size” (number of grid points for discretization). They conduct a number of experiments that show significant speed up of their method over prior methods. Experiments on several datasets including N-body problem and MNIST show improved performance over standard KAN and MLP architectures.

strengths:
Provides a method to significantly speed up inference using KANs. Since KAN nodes are more expressive, this has the potential to be widely applicable.

weaknesses:
The presentation can be improved. For someone not familiar with KANs, the notions of grid size, and other parameters should be explained clearly. The speed up and accuracy tradeoffs could be highlighted early on to give a sense of the impact. State what delta and g are in eqn 2.

The method used for speeding up is somewhat straight forward. 

It would have helped to include some of the popular LLM style tasks in your experiments clearly showing the speed vs accuracy including standard MLP based implementation of LLMs.

questions:
How exactly does your method compare with standard MLP based architectures -- do you ensure the total compute remains the same? 
How would your method compare on LLMs for language prediction tasks? Do you think it will provide a speed up over LLMs?
For the setup in figure 4, where you compare UKAN, KAN, MLP how do the latency/speed of each technique compare?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730505017186

thread_type:
official_review

<thread_object>
creator: reviewer_pik8

<content>
summary:
The paper introduces Unbounded Kolmogorov-Arnold Networks (UKANs), a novel approach to function approximation that addresses the limitations of traditional Kolmogorov-Arnold Networks (KANs), specifically the need for bounded grids and inefficiencies in computation and memory usage. UKANs utilize a coefficient generator (CG) model, which dynamically generates B-spline coefficients over an infinite grid, integrating multilayer perceptrons (MLPs) with KANs and leveraging positional encoding for efficient large-scale learning. The authors present a GPU-accelerated implementation that significantly reduces the computational cost of B-spline evaluations. Experimental results across regression, classification, and generative tasks show the effectiveness of UKANs, demonstrating superior computational efficiency and competitive accuracy compared to existing methods. The work advances function approximation techniques, offering a scalable and flexible solution for complex tasks.

strengths:
The authors provide a significant contribution in making function approximation more scalable and efficient. The integration of a coefficient generator (CG) model that dynamically produces B-spline coefficients enables UKANs to handle unbounded domains, a major advancement over existing KAN architectures. The use of GPU acceleration to reduce computational and memory costs is another strong aspect, as it makes UKANs practical for large-scale applications that were previously out of reach for KANs.

weaknesses:
- As shown in https://www.arxiv.org/abs/2407.16674, KAN can be considered a more interpretable model, particularly effective when applied to symbolic formulas. So improving the performance on downstream tasks may not be that important.
- Additionally, as shown in Tables 3 and 4, the performance improvements reported for UKAN compared to KAN are not substantial. These improvements could simply be due to the increased number of parameters or some level of randomness in the training process. The authors should consider evaluating UKAN on a broader range of datasets to strengthen the claims about its effectiveness.
- A few minor suggestions: making Figure 3 smaller and Figure 4 larger would improve readability. 
- Also, I noticed a small typo in the caption of Figure 3, where the KAN paper is cited twice, and the use of ‘[32, 32]’ appears unnecessarily. (I’m not sure why the manuscript does not have line numbers.)

questions:
- Have you thought about evaluating UKAN on more diverse datasets to provide a stronger and more convincing comparison, particularly in scenarios where interpretability is crucial?
- What are your thoughts on the potential interpretability trade-offs between KAN and UKAN, given that KANs are known to be more interpretable for symbolic tasks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730625118367

thread_type:
official_review

<thread_object>
creator: reviewer_frp4

<content>
summary:
This paper provides a GPU implementation of KAN that utilizes local matrix representations of B-spline functions. In addition, the paper proposes using MLPs to generate B-spline coefficients by embedding the grid-group index and feature index. Experiments are conducted to compare the performance of competing methods in terms of computational efficiency and the accuracy of regression, classification, and generative tasks.

strengths:
The proposed wrapKAN reduces computation time compared to the original torchKAN. The proposed UKAN offers performance that is either better or comparable in regression, classification, and generative tasks.

weaknesses:
The primary contribution of this paper is the introduction of unbounded grid. However, the advantages of the unbounded grid are called into question. Why is grid updating or data normalization in KAN not considered preferable? The experimental results indicate that the improvements of UKAN over KAN are limited.

questions:
If the grids are unbounded, it appears that an infinite number of coefficients would be required for B-spline curves. Is this the case? If not, how does it different from grid updating?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper extends KANs networks to unbounded grids and proposes a fast(er) library. 

In my opinion the weaknesses of this paper outweighs its strengths:
- The performance improvements reported for UKAN compared to KAN do not seem to be significant.
- The presentation can be significantly improved to showcase the contributions:
  - More details on the KAN architecture (instead of referring to another paper) in order to emphasize the differences between KAN and UKAN
  - More details on the B-splines (the constants $k$, $d_{in}$,... are important to appreciate the improvements) 
  - More details on how you managed to get a faster implementation than the baselines