Title: From Kolmogorov to Cauchy: Shallow XNet Surpasses KANs
Venue: NeurIPS 2025
Accepted: yes
Keywords: Universal Approximation Theory

Rational Activation Functions

Scientific Machine Learning

Function Approximation

Symbolic Reasoning
Abstract: We study a shallow variant of XNet, a neural architecture whose activation 
functions are derived from the Cauchy integral formula. While prior work 
focused on deep variants, we show that even a single-layer XNet exhibits 
near-exponential approximation rates—exceeding the polynomial bounds of 
MLPs and spline-based networks such as Kolmogorov–Arnold Networks (KANs).

Empirically, XNet reduces approximation error by over 600× on discontinuous 
functions, achieves up to 20,000× lower residuals in physics-informed PDEs, 
and improves policy accuracy and sample efficiency in PPO-based reinforcement 
learning—while maintaining comparable or better computational efficiency 
than KAN baselines.

These results demonstrate that expressive approximation can stem from 
principled activation design rather than depth alone, offering a compact, 
theoretically grounded alternative for function approximation, scientific 
computing, and control.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nFinding a replacement for MLPs at the base of neural networks is an interesting problem to either increase computation efficiency or interpretability for simple networks, which was recently spearheaded by KANs. The authors introduce XNet, a neural network type based on the Cauchy basis functions, which has a theoretically superior approximation result for smooth functions. Experiments are provided to support this claim.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- The problem of finding a provably better fundamental base for neural networks is an interesting and important question.\n- The construction of XNet is novel given the previous work mentioned.\n- The experimental results are good, although there are some concerns about fair comparison (see Weaknesses).\n## Weaknesses\n- It would be helpful to have a figure showing what Cauchy basis functions are. I believe this would make understanding the idea behind the paper more easily. As an example, the KAN paper has figures showing what the spline functions are and how the elementary operations over the spline functions operate in a layer.\n- Misleading claim: The $p$ constant in the convergence rate for the Cauchy approximation result is the smoothness of the target function class. For most of the main text up to Section 3, this is never explicitly mentioned and it is only mentioned that the rate of $O(N^{-p})$ holds for arbitrary $p$. This is only true for infinitely differentiable target functions. I don't believe this is true for most applications we care about. This in turn makes it confusing why XNet is performing well in Section 4.1.1 because the target function is non-differentiable. Can the authors explain why XNet is performing well despite no theoretical guarantee for this particular case?\n- Lack of clarity in writing: When introducing a new function like the Cauchy basis function, it is important to formally define what mathematical object it is. For example in Sec 3.2, it is not mentioned what the domain and co-domain of $\\phi$ is.\n- Missing baselines:\n\t- In Section 4.1.1, another baseline should be a comparable ReLU network. For non-differentiable targets, ReLU should be a natural fit and a more appropriate baseline.\n- In Sec 4.1.2, the fairness in comparison between different configurations of the network width across different models is questionable. I am not familiar with the computation complexity of KAN and XNet, so I don't know if the much larger width in XNet (500 vs 100 in MLP and 6 in KAN) is fair. This is also true for the configuration in Table 8. Can you give an explanation for why you chose these hyperparams?\n\nQUESTIONS:\n- In XNet, do you need to tune the $p$ constant or is that automatically adapted to the highest smoothness order \"detectable\" from the target function? Is there any evidence if you claim the latter?\n- In Table 5, is Time here training/inference time?\n\nLIMITATIONS:\n- The limitation about the assumption that the target function is infinitely differentiable (as mentioned in my review) is not discussed at all.\n- Limitations about scaling XNet to larger networks should also be mentioned (is the memory usage or computation time a bottleneck).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces XNet, a shallow neural network architecture leveraging novel activation functions derived from the Cauchy integral formula. The authors claim that a single-layer XNet achieves approximation rates that are near-exponential, surpassing the limitations observed in traditional MLPs and spline KANs. The empirical evaluation demonstrates XNet's superior performance on small-scale problems like discontinuous function approximation, PDE solving and PPO tasks from reinforcement learning. The central claim is that network expressivity can be achieved through a principled design of activation functions rather than relying on increased architectural depth.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- Experimental Analysis of \"adaptive\" activation functions  design for shallow neural networks that provides a competitive advantage over learned spline based activation functions (KANs)\n- Strong theoretical grounding for convergence rates, that are better than existing frameworks for shallow neural networks\n- Empirical Validation Across Diverse Domains: The paper provides compelling empirical evidence across three distinct domains: function approximation, PDE solving, and reinforcement learning. The breadth of evaluation provides an initial confidence in XNet's versatility and effectiveness.\n- In all of the provided experiments, XNet outperforms KANs. Additionally the runtime compared to MLPs are similar (on a CPU) whereas in these scenarios KANs are much slower.\n- Experiments demonstrate that the activation function enables robustness to noisy and discontinuous problem settings.\n\n### Weaknesses\n\n- While the empirical results demonstrate broad applicability, the claims in the introduction are too broad and should be toned down. For example the authors claim \"PDE solving: over 10,000× lower error than deep MLPs in PINN settings\". While this is \"true\", it is only demonstrated for 2 PDEs with low complexity. Additionally prior works like KANs which operate in the parameter efficient regime, tend to not scale well when tested on larger problems [1][2][3] (to cite a few examples where KANs were evaluated)\n\n[1] https://arxiv.org/abs/2407.16674\n[2] https://openreview.net/forum?id=yPE7S57uei\n[3] https://arxiv.org/pdf/2503.10632\n\n### Minor Stylistic Nits\n\n- In Table 4 it would be helpful to highlight the best RMSE for each row\n- In Figure 2, making the scales same will make it easier to compare the errors.\n\nQUESTIONS:\n- Table 4. How is the input shape to XNet constant at 2? Should that be 4?\n- Section 4.1.4: If we increase the number of basis in XNet, does it demonstrate over fitting? Similarly if the width of the KANs are tuned do they still show over fitting to noise?\n- PINNs are generally not an effective mechanism to solve simple PDE cases that XNet was tested on. Since the proposed XNets claim to demonstrate expressiveness from activation functions rather than depth, can XNets be evaluated on PDEs like navier stokes where we need deep MLPs to get good accuracy?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel approach to learned function approximations, in which shallow, wide networks with rational activation functions are used. In function approximation, PINNs learning and RL tasks, these networks are shown to be more accurate than MLPs and KANs with similar parameter counts.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe approach seems to be both theoretically sounds and empirically well-validated. The overall presentation is clear and effective.\n\nWeaknesses:\n\n1) The theoretical results rely heavily on existing theorems but don't provide citations. For example, even after reading the supplementary material it was not at all clear to me how the bound in Theorem 3.1, which involves high-order polynomials in the denominator, is connected to theorem 3.2, in which the denominator has order only 2. To understand this I had to first find the explanation in Li et al., 2024. This should be explicitly clarified.\n\n2) The results are impressive but fairly limited in scope. Since an alternative is being proposed to MLPs and KANs, which are applicable to a wide variety of tasks, I would ideally like to see some additional tests beyond the small number presented here. In particular, more challenging tasks should be considered for the PINNs example than the heat and Poisson equations. It would also be good to see performance comparisons on real-world datasets.\n\n3) Is it really surprising that ReLU MLPs cannot learn smooth functions or those with discontinuities? Aren't there stronger baselines in the literature for this task?\n\n4) How much better are Xnets than MLPs with smooth activation functions, such as SIRENS, ELUs or SiLUs? The KANs paper compares to MLPs with multiple activation functions, but this has not been done here.\n\nQUESTIONS:\nThe Cauchy activation function's gradient tends towards zero for very large positive or negative values. Could this lead to problems with learning?\n\nIn the experiments carried out here, the inputs and targets were treated as unstructured vectors as in an MLP. Is there some way of applying Xnets that would take advantage of known structure in the inputs/targets, as in convolutional, recurrent or attention-based models?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper shows that single layers XNets achieve near-exponential approximation rate. The authors prove and empirically demonstrate that this shallow architecture can achieve superior accuracy compared to both MLPs and KANs. The central thesis appears to be that principled activation function design can be a more efficient source of expressivity than architectural depth.\n\nBased on my research of XNet, it appears that this architecture does not take a large performance hit either, so the results of this paper seem promising.\n\nSTRENGTHS AND WEAKNESSES:\nI would say the strengths are as follows:\n1. The theoretical grounding seems quite solid. This is not just an empirical paper but seems to provide formal theory around convergence rates.\n2. The clarity of the central thesis is nice. Namely, \"expressivity from activation design can be better than expressivity from depth\".\n3. The empirical validation is diverse. We see a range of cases from function approximation to scientific computing to RL.\n\nI would argue there are a few limitations:\n1. There is a focus on the single-layer case. I understand this is probably because the theory is based on the single layer case, however, I would have liked to see empirical results on multi-layer networks. Maybe there could be a way that the theory generalises partially or the theory provides intuition.\n2. I didn't see much analysis of computational overhead. The Cauchy activation is obviously more expensive in terms of FLOPs but I didn't see discussion of how this scales with network size. If Cauchy is to be adopted widely then it would need to make sense from a computational perspective in architectures like the transformer. Based on my understanding, the activation would not change the scaling of an architecture like the transformer but would be nice to have some explanation here.\n3. There isn't much discussion of how the activation function interacts with other parts of training like optimisation or regularisation.\n4. I would have liked more experiments and ablations for the domains chosen.\n\nQUESTIONS:\nI have a few questions:\n1. How does XNet account for the hierarchical compositionality essential for complex perception and language tasks? How do we think it would perform in these cases? Is there anything limiting it there?\n2. What is the empirical performance of deep XNets, and how does the activation's computational overhead affect the practical trade-off between model size and training speed?\n3. Does the Cauchy activation introduce any bottlenecks for parallelization?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-22"}