From Kolmogorov to Cauchy: Shallow XNet Surpasses KANs
Authors: Xin Li
Xiaotao Zheng
Zhihong Jeff Xia
Venue: NeurIPS 2025 poster ()
Keywords: Universal Approximation Theory
Rational Activation Functions
Scientific Machine Learning
Function Approximation
Symbolic Reasoning
Primary area: theory

Abstract: We study a shallow variant of XNet, a neural architecture whose activation 
functions are derived from the Cauchy integral formula. While prior work 
focused on deep variants, we show that even a single-layer XNet exhibits 
near-exponential approximation rates—exceeding the polynomial bounds of 
MLPs and spline-based networks such as Kolmogorov–Arnold Networks (KANs).

Empirically, XNet reduces approximation error by over 600× on discontinuous 
functions, achieves up to 20,000× lower residuals in physics-informed PDEs, 
and improves policy accuracy and sample efficiency in PPO-based reinforcement 
learning—while maintaining comparable or better computational efficiency 
than KAN baselines.

These results demonstrate that expressive approximation can stem from 
principled activation design rather than depth alone, offering a compact, 
theoretically grounded alternative for function approximation, scientific 
computing, and control.

From Kolmogorov to Cauchy: Shallow XNet Surpasses KANs
Xin Li
Xiaotao Zheng" Center for Financial Engineering Soochow University, Suzhou, China
College of Computer Science and Technology Dongguan University of Technology, China Institute for Advanced Research Great Bay University, Dongguan, Guangdong, China xin112023@u nor twwestern.cee edu
202340130026stu suda, edu. cn
Zhihong Xiat
Institute for Advanced Research Great Bay University, Dongguan, Guangdong, China Department of Mathematics
Northwestemn University, Evanston, II. USA xia@math. wwwtwwwstern. edu
Abstract
We study a shallow variant of XNet, a neural architecture whose activation functions are derived from the Cauchy integral formula. While prior work focused on deep variants, we show that even single-layer XNet exhibits near-exponential approximation iites-exceeding the polynomial bounds of MLPs and spline-basee networks such as Kolmogorov-Arnol Networks (KANs).
Empirically, XNet reduces approximation error by over 600x on discontinuous functions, achieves up to 20,000x lower residuals in physics-informet PDEs, and improves policy accuracy and sample efficiency in PPO-based reinforcement learning while maintaining comparable or better computational efficiency than KAN baselines.
These results demonstrate that expressive approximation can stem from princi- pled activation design rather than depth alone, offering a compact, theoretically grounded alternative for function approximation, scientific computing, and control.
Introduction
As deep learning models grow increasingly large and resource- intereev,,,,sive, there is renewed interest in shallow architectures that combine theoretical rigor with practical efficiency. In particular, scientific applications such as PDE solving, symbolic regression, and control require models that are not only expressive but also interpretable, sample-effieient, and robust to irregularities.
Traditional universal approximators like shallow MLPs [5, 10] suffer from slow convergence [1, 7]. especially in high dimensions. Recent efforts such as Kolmogorov-Amole Networks (KANs) [19] improve expressivity via spline-based activations, still rely on multi-layer composition and struggle with discontinuities due to their smooth basis structure.
'Equal contribution Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

In this work, we study principled single-layer variant of XNet, neural architecture built on the Cauchy activation finction introduced in [16]:
\1 + 22 (xx)
X1, 22, dedee IR.
x² + d
While [16] proposed Cauchy activations as a plug-and sare-and--lay component for general deep architectures, we focus on the shallow, single-layer setting and provide the first theoretical analysis of its approxi- mation rates. We prove that even single-layer XNet achieves arbitrarily high-order convergence O(N-1) for any 0 real-analytic functions, and demonstrate its effectiveness across function approximation, PDE solving, and reinforcement learning tasks.
These rational functions-derive from the Cauchy integral formula-yielo smooth, localized, and trainable basis elements, enabling XNet to approximate irregular, high-dimensional, and even discontinuous functions with far fewer neurons and no depth.
Main theoretical result. We prove that XNet achieves arbitrarily fast convergence O(N-P) for any p > 0 on smooth target functions. contrast, classical ReLU networks are limited by the curse dimensionality with rate O(N-2r/d) (for target smoothness and input dimension d), and spline-based KANs achieve O(N~*) constrained by the fixed spline degree XNet's analytic Cauchy basis functions overcome both limitations by enabling tunable, high-order approximation in a single layer. See Section 3.2 for the formal result and Appendix A.5 for the proof.
Figure illustrates this convergence behavior: as the number of neurons N increases, XNet maintains significantly steeper error decay rate compared to ReLU and KAN.
Mecror
Figure 1: Comparison of approximation rates. XNet achieves O(N P) with p = outperforming ReLU 2r/d) with 1, d 2) and KAN (O(N-*) with k 2). Log-log plot shows approximation error versus neuron count N.
Empirical results: Across three domains, XNet outperforms strong baselines:
Function approximation: over 600x lower MSE discontinuous targets.
PDE solving: over 10,000x lower error than deep MLPs in PINN settings. (see Ap- pendix A.I for computational analysis)
Reinforcement learning: faster convergence and higher rewards under PPO.
These findings challenge the depth-centric paradigm, showing that activation design alone can enable expressive, compact, and thcoretically grounded models-often with computational efficiency comparable to or better than multi-layer alternatives.
Why this matters now. As foundation models encounter scaling bottlenecks, there is renewed interest in shallow networks that combine interpretability, efficiency, and provable expressivity, XNet exemplifies this direction by achieving strong performance without depth or overparamecterizatio..
[16] introduced the Cauchy activation and established its universal approximation properties. Building on this foundation, we establish new convergence rates in dual function spaces (Theorem 3.2; proof Appendix A.5), proving O(N-)) for any r > 0 on real-analytic functions-the first such result for shallow XNet. While our analysis focuses single-layer networks, [16] demonstrated the modularity of Cauchy activations in deeper architectures (multi-layer MIPss, CNNs, ResNets) on MNIST and CIFAR- 10.

---

Together, these results challenge the depth-centric paradigm, showing that principled activation design alone can yield compact, expressive models for scientific and symbolic AI.
Related Work
Universal Approximation and Expressive Efficiency. Shallow MLPs with non-polynomial acti- vations are universal approximators [5, 10], but their approximation efficiency degrades in high dimensions [1, 26]. Recent work improves expressivity through spline [19]. rational [3], and interpolant-base based [23] activations. Parametric activations [2, 15] offer adaptability, but often in- crease complexity lack convergence guarantees. XNet addresses this gap by connecting activation design with provable, high-order approximation in a shallow architecture.
Structured Activation Networks. KANs [19] approximate functions via learnable B-splines along edges, enabling efficient approximation with compositional depth. However, their reliance on spline smoothness and fixed grid resolution limits adaptability to discontinuities or noise. Recent applications span time-series [25], RL [11], and generative modeling [9]. In contrast, XNet provides analytic, localized responses with no depth or grid assumptions.
Rational and Complex- Valued Models. Rational activations [4, 3] have been used to improve approximation power via learnable Padé-like functions. However, these approaches are primarily heuristic and often lack theoretical guarantees beyond empirical smooth function fitting. Our work draws from complex with localized and tunable responses. This analytic foundation enables high-order approximation in both classical and dual function spaces, distinguishing XNet from prior rational models.
Seeiysssisseeeicccalllllll the Cauchy integral formula-to derive rational activations
Scientific Machine Learning and Operator Learning. PINNs [22] and neural operators [17. 14] enable learning in scientific domains, but often rely on deep networks, spectral bias, or task-specific assumptions. Spectral solvers [8] and low-rank architectures [27] improve convergence, require expert tuning. XNet offers shallow alternative with strong theoretical grounding and minimal tuning achieving fast convergence across physics-informed and control tasks.
Theoretical Foundations of XNet
3.1 Limitations of Spline-Based KANs
Kolmogorov-Arnold Networks (KANs) [19] implement the classical Kolmogorov superposition theorem [13] via bpline-based edge activations:
f(x) = Σ C [manktes) =(z)= = wbb(x) + w. C..(()).
f(x)
where (z) are B-spline basis functions. While KANs offer adaptive local representations and O(N~*) convergence for spline degree k, they suffer from poor generalization near discontinuities, require multi-layer structures, and incur high computational costs due to spline interpolation and grid dependencies.
3.2 Cauchy Basis Architecture (XNet)
This section analyzes a shallow variant of XNet, a neural architecture built on Cauchy basis activations derived from the Cauchy integral formula:
A1 + Ag
(xx)
x² + with trainable \1 A2,
This yields localized, tunable, and analytically smooth responses. Below, we present the theoretical foundations supporting its expressivity.
3

---

(1) Approximation Efficiency. XNet achieves arbitrarily fast convergence O on smooth target functions, where the convergence exponent p depends on the analyticity (i.e., smoothness) of the target. This surpasses the (( 22/d) rate of ReLU MLPs [1, 26] and the O(N=*) rate of spline-based KANs [19], where r is the Sobolev smoothness and is the input dimension.
See Appendix A.2 for detailed derivation.
(2) Localization and Noise Adaptivity. Cauchy activations exhibit rational decay O(1/z), yielding naturally localized basis functions:
(x(x) = A15(2 m1) 22,1 (x - p:) + de
By adjusting the parameter d, neurons adapt their receptive field: small d, captures sharp discontinuities, while large d, smooths over noisy regions. Unlike B-splines, which require predefined grid partitions, XNet offers neuron-wiss adaptivity for heterogeneous signals.
See Appendix A.3 for analytical expressions and derivative properties.
(3) Comparison with B-Spline KANs. We contrast the analytical and computational properties of XNet (Cauchy basis) and Kolmogorov-Armold Networks (B-spline basis) in Table 1.
Table 1: Comparison of XNet B-spline KANs
Aspect
Cauchy Basis (XNet)
B-spline (KAN)
Approximation Rate
O(N P), arbitrary p > Continuous, localized, trainable Dense (condition number O(N)) Closed-form, differentiable w.r.t. d? Single-layer, low parameter count
O(N**), *), limited by degree k Grid-constrained, piecewise
Basis Adaptivity
Matrix Structure
Sparse (condition number O(1))
Derivative Computation
Piecewise, non-smooth
Architectural Cost
Multi-layer, higher cost
This comparison highlights XNet's key advantages over KANs: global analytical structure, tunable locality, and full differentiability-wiii -withou reliance on grid resolution. For further details on matrix conditioning and gradient properties, see Appendix A.4.
(4) Cauchy Approximation Theorem.
Theorem 3.1 (Th.1 in [16)). Let f(z)...,, be analytic on open U C Cd. Then for any & > 0, there exists a finite sum:
N
A
sup zEU
f(=)
<8, with 8 = ONN(N))
(
2d)
This theorem shows that Cauchy activation functions can approximate analytic functions with arbitrarily high order. The specific form
d1 + A x2 d2
((x)
was introduced in [16] as simplified instantiation.
While the above result applies to partial fractions, it does not directly yield rates for XNet. Our Theorem 3.2 provides the first constructive O(N rate for single-layer XNet on real-analytic functions f (H(K) over compact domains K Rd
(5) New Result: High-Order Approximation XNet. We now prove that one-hidden-llyer neural network using Cauchy activations inherits the same approximation rate.

---

Theorem 3.2 (High-Order Approximation of XNet). Let K C CR' be compact, and let f E (K))
be real analytic. Let - Aurtta An
be the Cauchy activation family. Then for any r >> 0, there
x'+d"
exists a one-hidden-layer XNet function JN span( ()) such that
fl fNVlCO(K) O(N").
This is a new result: while Theorem 2 in [16] shows that XNet is universal, it does not quantify convergence rate. Our theorem gives an explicit O(N-*) bound for real-analytic
Sketch. (1) Use the Cauchy approximation theorem to approximate any ID analytic function with rate O(N="); (2) Represent multivariate polynomials using (w x)*,, reducing to the 1D case; (3)
Approximate f by polynomials, then apply Step (2). See Appendix A.5 for details.
Remark 3.3. While our theorem assumes f E C* (K) (real-analytic), empirical results (Section 4.1.1) show XNet performs well even on discontinuous functions like the Heaviside step. This aligns with classical results on rational approximation to non-smooth functions (e.g., [20, 24J for [x]""), which demonstrate near-exponential rates. Extending our theoretical framework to broader function classes is an important direction for future work.
(6) Neuron Efficiency at Arbitrary Approximation Orders. We show that pee-hidden-layee XNet achieves convergence rates of O(N for any desired order >> O. To achieve an approximation error of e, the number of required neurons satisfies:
N oc > 0.
This tunable convergence order distinguishes XNet from classical architectures: ReLU MLPs are fixed at 1/d [1], while spline-based KANs are limited by spline degree k. i.e., r S k.
In contrast, XNet leverages the analytic structure of the Cauchy basis to achieve flexible, localized, and high-precision approximation with fewer neurons. This result particularly beneficial for smooth, high- -igh-dimensional, or irregular targets. A complete derivation of these scaling laws is provided in Appendix A.6.
Table Theoretical comparison of approximation rate and neuron complexity.
Architecture
Approx. Rate
Neurons Error
Adjustable Order r?
Depth Requirement
ReLU MLP
O(N-1/4 O(N
09e
(fixed)
Deep
KAN (Spline)
O(e-1/b)
(limited by k)
Moderate
XNet (Ours)
O(N-))
O(e-1/r)
/ (Any r >> 0)
Single-layer
Remark 3.4. The ability to tune the approximation order arbitrarily via is direct consequence of the Cauchy approximation theorem [16], which guarantees exponential convergence for analytic functions.
This theoretical framework not only guarantees approximation efficiency, but also explains XNet's empirical strength in modeling irregular and high-dimensional scientific data.
Experimental Setup and Results
We evaluate XNet across three ffnnction approximation, PDE solving, and reinforcement learning-selecteer selected to assess different capabilities of neural architectures. These tasks respectively test expressivity under discontinuities, accuracy in modeling physical operators, and generalization in sequential decision- making.
This diversity highlights XNet's key strengths: high-order approximation, compact design, and efficient training. all settings, we compare against MI.P and KAN baselines under matched parameter budgets and consistent training protocols.
5

---

4.1 Function Approximation Experiments
We evaluate the networks on four categories of functions with increasing complexity: a low- dimensional discontinuous function, low-dimensional special functions, ligh-dimensional functions, and function with noise. The experimental comparison between XNet, B-spline, and KAN indicates that XNet provides enhanced approximation capabilities. Except for the first function example and the final task, all other examples are from the referenced article [19], with KAN settings matching those from the original experiments. This ensures a fair comparison, demonstrating that XNet exhibits strong approximation capabilities across various benchmarks.
4.1.1 Heaviside step function approximation
Dataset and Implementation Details. ties using the Heaviside step function:
We evaluate discontinuous function approximation capabili-
11 00 I I
0,
f(x)
(1)
An XNet architecture [1, 64, I] and KAN configuration [1, 1J with 200 grids were implemented. Both models were trained using the Adam optimizer [12] under identical conditions.
Table 3: Heaviside function approximation comparison
Model
Architecture
MSE
RMSE
MAE
XNet
[1,64,1]
8.99c-08
3.00e-04
1.91c-04
ReLU (shallow)
[1,64.1]
2.05e-03
4.53e-02
8.82e-03
ReLU (deep)
[1,64,64.1]
6.81e-05
8.25e-03
3.76e-04
KAN
[1.1] (200 grids)
5.98c-04
2.45e-02
3.03e-03
Overall Performance. As shown in Figure 4, both B-Spline and KAN exhibit "overshoot," resulting in local oscillations at discontinuities. While adjusting the grid can mitigate this phenomenon, it introduces complexity in parameter tuning (see Table 9, Appendix B.2). In contrast, XNet demonstrates superior performance, providing smoother transitions at discontinuities. In terms of fitting accuracy in these regions, XNet outperforms KAN, with MSE approximately 1000-fold smaller (8.99e-08 versus 5.98e-04) as shown in Table 3. The mathematical foundation for XNet's advantages in discontinuous functions is provided Section 3.2.
4.1.2 Approximation with special functions
Dataset and Implementation Details. We collect several special functions common in math and physics, summarized in Table 4. For neural network architectures, we implemented consistent parameter configurations following [19]. MLPs are configured with fixed widths of 5 or 100, with depths varying across (2, 4, 5, 6). KANs maintain a fixed width of 5 with depths similarly swept through (2, 3, 4,5, 63. XNet was constructed as a single-layer architecture with either 500 or 5000 basis functions.
Table Special functions benchmark comparison with fixed XNet shape and extended test RMSES.
Name
cipy.special..omm API
Minimal KAN shape
XNet shage
Best KAN test RMSE
MLP test RMSE
XNet test RMSE
Jacobian eliyne
ellip3 (x,
[2.8811]
1,30
6.48-mm 04
4-iile
Iicompple ciliptic integral of
CilipAinc(s, 5
2A..1]
[2,50011
1.2kc
0,70c
Incompiate clipiic iniogral of the sccund kind
#1 Apsinenc y)
[.2.1.11
[3,50011
S00e-01
5,80s 00
function of dea
[3,50111
6.50x-00
2.89c
Resod finction of socund
yels, 7)
[2222.11
(2.00011]
1.
5.45-01
1.12
Modined Beesel finctinn the fins lind
IVCK,
(2432111]
[2N0011]
9.2 40
107e- 02
6,30l 00
legonine
Ipev(o, y)
Associated Legencre function (m
TyeyCs, X, 7)
[241]
6000
1.50 813
2.71e-00
Associated Legendre Finciio (in 2)
19snt2,
[232.11
[2,0001]]
2.36 004
9.49 04
7.44c-00
sphenical harmonics (m
sp spa. 22,,))
(25.111
[2,0011]
1.22
2,28c
spicical harmmics (m Iln 2)
1211111
12,500L11
1.90x 06
1.54-00 035
7.78c-10
sphrical harmunic (m
(2.232211
(2.5001)
16
0:31--01
2.43k

---

Overall Performance. As presented in Table 4, experiments demonstrate that XNet achieves superior approximation aceuracy on the special functions. Particularly noteworthy are XNet's results on complex functions such as spherical harmonics, where it achieves an RMSE of 7.78 x 10-6 for (m 1, = 2) compared to KAN's 1.50 10-5 and MIP's 1.84 x 10-3 For Bessel functions, XNet similarly shows considerable improvement, with an RMSE of 2.39 x 10-4 for the first kind versus 1.64 x 10-3 and 5.52 x 10-3 for KAN and MLP respectively. These findings suggest that XNet's single-layer architecture effectively approximates special functions with high accuracy.
4.1.3
Approximation with high-dimennional functions
Dataset and Implementation Details. Following the benchmarks established in [19], we evaluated two challenging functions: a 4-dimensional function exp (2 (sin (н + =5)) + xyraa) and 100- 1 Σi= sin²(22) )). Each experiment utilized 8000 training points and 1000 test points. For the 4D function, KAN was configured as [4, 4, 2, 1] and XNet was implemented with 5000 basis functions. For the 100D function, KAN was structured as [100, 1] with XNet maintaining the same configuration. XNet optimized using the Adam algorithm, while KAN is initialized with G = 3 grid points and trained with the LBFGS algorithm [18], progressively increasing grid density through G 3, 5, 10, 20, 50 every 200 steps.
dimensional function exp( Tno
Table 5: Performance comparison of XNet and KAN on 4D and 100D test functions
Function
Model
MSE
RMSE
MAE
Time (s)
e (sin(a(ef ( Hagea)
XNet [2. 5000, I]
2.31e-06
1.52e-03
8.39e-04
78.18
KAN [4.2,2.1]
2.62e-03
5.11e-02
3.63e-02
143.10
1000
sin?
XNet [2, 5000,
6.85e-04
2.62c-02
2.09c-02
158.69
KAN [100, 1, 1]
6.59e-03
8.12e-02
6.46e-02
556.50
Overall Performance. The results presented in Table 5 indicate that XNet outperforms KAN across both dimensionalities. For the 4D function, XNet achieves an MSE three orders of magnitude lower than KAN while requiring approximately half the computation time. the challenging 100D case, XNet maintains nearly an order of magnitude better MSE while executing 3.5x faster than KAN. Notably, XNet's computational efficiency scales significantly better as dimensionality increases, while achieving consistently higher accuracy across all evaluation metrics.
4.1.4 Functions with noise
Dataset and Implementation Details. To assess function approximation capabilities in noisy environments, we examined a dynamic system governed by:
f = 0.1 + 0.5 sin sin(z)))) + sin + k¹,
where = 1,2, n, with state transitions:
+ = _ a =
x x = 1// 24 = = 11
Initial conditions =g .rf. 20 2 2 were sampled from 00... with Gaussian noise j² ~ NN0, at three distinct levels: none (a = 0), moderate (a = 0.05), and high (a = 0.1). We generated 300 sequential points, training models on the first 80% (noise-corrupted data) and evaluating prediction performance on the remaining 20% (noise-free data). This experimental design evaluates each model's capability to extract underlying patterns despite noise in the training data. For this experiment, we compared KAN [5, 64, 1] against XNet [5, 20, 11, both optimized using the Adam optimizer.
Overall Performance. Table 6 summarizes the comparative results. XNet outperforms KAN across all metrics, particularly in noisy environments. In the noise-free scenario, XNet achieves an MSE of 4.7099 10-? approximately 28x lower than that of KAN (1.3209 x 10 10-5). As noise increases, XNet maintains its advantage, with 3.7x better MSE at moderate noise and 14.5x better at high noise. Figures and provide visual evidence of XNet's enhanced robustness to noise. Furthermore, XNet achieves these performance improvements with fewer parameters and reduced training times. These results indicate that XNet may more suitable for real-world noisy datasets, while the KAN model exhibits characteristics consistent with overfitting.
7

---

Table Performance comparison KAN [5, 64,
and XNet [5, 20, 1] under different noise levels.
Noise Level
Model
MSE
RMSE
MAE
Time (8)
KAN [S, 64, 1.3209e-05
3.6344e-03
4.8311e-03
17.8863
XNet [5, 20, 1]
4.7099c-07
6.8629e-04
5.2943e-04
9.6502
KAN [5, 64, 1]
1.6784c-03
4,0968c-02
3.1549c-02
17.9535
0.05
XNet [5, 20, 1]
4.5109c-04
2.1239c-02
1.5797c-02
9.4391
KAN [5, 64, I]
9.4653e-03
9.7290e-02
7.8063e-02
17.8863
0.1
XNet [5, 20,
6.5517e-04
2.5596e-02
2.0233e-02
9.0296
Given its performance function approximation tasks, both terms of computational efficiency and accuracy, experimental results suggest that XNet provides an efficient neural network architecture with effective approximation capabilities. Building on this, in the following subsection, we apply PINN, KAN, and XNet to approximate the value function of the Heat and Poisson equations.
4.2 PDE Solving Capability
Experimental Setup and Implementation Details.
We selected the Heat equation as a benchmark
PDE problem:
du
Ju Ox2
(z,t) [0, x [0, ul, (z,0) sin(T
x E [0, 1], u(, u(1,t) = 0, (2)
01
with known analytical solution u( 1) = w ( Following the physics-informed neural networks (PINNs) framework, we defined the loss function as
1 Th
loss a loss; + loss, nho
(z) u()))
+
Σ
vu
(3)
where n, 2500 interior points and 150 boundary points were sampled uniformly, with 0.1 serving as a weighting coefficient to balance the loss components.
We evaluate four network architectures within the PINN framework: a two-layer MLP [2. 20, 20, 1]. single-layer KAN [2, 10, 1), and two XNet configurations with varying widths [2, 20, 1] and [2, 200, 1].
12
01
24
10
02
C4
10
(a) MLP [2, 20, 20, 1]
(b) KAN [2, 10,
(c) XNet [2, 20, 1]
(d) XNet (2, 200, 1]
Figure 2: Performance comparison of different neural network architectures
Table Comparison of XNet and KAN on the Heat equation.
Metric
MSE
RMSE
MAE
Time
MLP [2,20.20,1]
2.4536e-05
4.9534e-03
3.8323e-03
43.8
XNet [2,20,1]
3.8936e-08
1.9732e-04
1.5602e-04
43.5
KAN [2,10,1]
1.5106e-07
3.8866e-04
2.9661e-04
254.6
XNet [2,200,1]
3.6867c-09
6.0718c-05
5.0027c-05
108.3
Overall Performance. Table quantitatively compares the performance of MLP, XNet, and KAN architectures on the hear equation task. Across all evaluated metrics, XNet consistently achieves superior accuracy and computational efficiency. The [2,200,1] XNet configuration achieves the lowest error, with MSE of 3.69 10-9-_ -ppproxmimi two orders of magnitude lower than

---

KAN (1.51 x 1 while also requiring less than half the training time (108.3s vs. 254.6s). Even with a more compact architecture ([2.20,1]). XNet outperforms KAN, showing a threefold lower MSE and completing training more than five times faster. In comparison, the standard MIPP ((2,20,20,1)) produces higher errors across all metrics, highlighting the limitations of conventional architectures for PDE approximation. These results demonstrate XNet's effectiveness in capturing the underlying structure of PDE solutions, suggesting its promise as a compact and efficient model reduction framework.
Additionally, we evaluated physics-informer machine learning models based on MLP, KAN, and XNet architectures for solving the 2D Poisson equation (detailed results in Appendix B.6). The XNet [2,200.1] I] configuration outperforms KAN [2,10,1] in both accuracy and computational efficiency, while achieving approximately 20, x better precision compared to the standard PINN [2,20,20,1] architecture.
4.3 Reinforcement Learning Applications
Experimental Setup and Implementation Details. We evaluated XNet, KAN, and MLP architec- tures as function approximators within the Proximal Policy Optimization (PPO) framework on two continuous control tasks from the DeepMind Control Suite: HalfCheetah-r4 and Swimmer-v4.
Proximal Policy Optimization (PPO) optimizes the stochastic policy xo by employing a clipped surrogate objective function ensure stable policy updates:
(0) = E. min (r+(0) A., clip(r:((), - e, ())
where rz (0) represents the policy ratio and is a clipping parameter (configured at 0.2). This objective function is combined with a value function loss and entropy regularization to balance exploration and exploitation.
Table 8 summarizes the network configurations used in our experiments. The MLP baseline uses two hidden layers with 64 units each for both actor and critic networks. The KAN implementation employs spline functions with order & and grid size g 3, chosen based on preliminary work [11]. The XNet configuration utilizes 64 basis functions to maintain parameter count parity with the MLP architecture for fair comparison. All models were trained using the Adam optimizer, implementing identical hyperparameters across architectures to isolate the effects of the function approximator choice.
Table 8: Network configurations for reinforcement learning tasks.
Model
Configuration
MLP
Two hidden layers with 64 units each
KAN
Order = 2, grid size g =
XNet
64 basis functions
Overall Performance. Figure 3 summarize the results of the three models the two tasks. XNet consistently outperforms MLP and KAN in both environments, achieving the highest rewards and faster convergence. In HalfCheetah-v4 environment, XNet improves the reward by 64% compared to KAN and 142% compared to MLP. Swinmer-v4 environment, XNet achieves reward improvements of 47% over MLP and 28% over KAN.
Theoretical Analysis XNet's Advantage. XNet's performance in reinforcement learning stems from its Cauchy kernel activation, which provides three key benefits:
Localized response with fast decay: Cauchy activations enable sharper adaptation localized patterns in policy and value functions particularly effective in continuous control tasks with abrupt transitions.
High-order approximation capability: As shown in Theorem, XNet supports arbitrarily high- order convergence for analytic targets, yielding more precise representations than MLPs or KANs.
Reduced parameter complexity: XNet achieves strong performance with fewer parameters due to its expressive basis functions, leading to faster convergence in reinforeement learning.

---

00300
100
4000
3000
2000
1000
0.2
200
Stagt
Nambor Fonation
(a) HalfCheetah-và
(b) Swimmer-v4
Figure 3: Reward comparison for PPO training across environments. XNet significantly outperforms both KAN and MLP in final performance after 1M training steps: In HaliChectta-v4, XNet achieves 3298.52 52, compared KAN (2010.52) and MLP (3358.49.. In Swimmer-v4, XNet reaches 100.38, while KAN and MLP attain 43.25 and 68.08 respectively.
These properties align well with the demands of policy gradient methods like PPO, where accurate approximation of both policy To (als) and value function (s) is crucial. Figure empirically confirms these advantages.
Conclusion and Outlook
Our results suggest that neural expressivity need not come from depth alone. XNet, built on principled analytic activation functions, achieves high-order approximation with only single sayer-offering both thcoretical rigor and empirical power.
This design unlocks a new perspective for compact and interpretable architectures, particularly in scientific and symbolic domains. Its ability to approximate discontinuous functions, solve PDEs efficiently, and improve sample efficiency in RL agents makes it a promising candidate for hybrid symbolic-numeriic solvers and real-time control systems.
In future work, we plan to integrate XNet into multi-scale or Transformer-style models, extend it to generative and multi modall learning, and explore its role in operator learning and symbolic program induction.
References
[1] Andrew Barron. Universal approximation bounds for superpositions of sigmoidal function. IEEE Transactions on Information Thheory, 39(3):930-99) 9450009995550 1993.
[2] G. Bingham and R. Mükkullinenn. Discovering parametric activation functions. Neural Networks, 148:48 65,44-65,222 2022.
[3] Nicolas Boullé, Yuji Nakatsukasa, and Alex Townsend. Rational neural networks. In Advances in Neural Information Processing Systems, volume 33, 2020.
[4] Daniele Caldarola, Tommaso Furlanello, Silvio Savarese, and Matteo Bruni. Rational relu: Towards exact function approximation with rational activation functions. In International Conference on Learning Representations (ICLR), 2021.
[5] George Cybenko. Approximation by superpositions of sigmoidal function. Mathematics of control, signals and systems, 2(4):303-3144 1989.
[6] S. Dubey, S. K. Singh, and B. Chaudhuri. Activation functions in deep learning: A comprehensive survey and benchmark. Neurocomputing, 503:92-108, 2022.
[7] Ronen Eldan and Ohad Shamir. The power of depth for feedforward neural networks. In Proceedings of the 29th Annual Conference on Learning Theory, volume 49 of Proceedings of Machine Learning Research, pages 907-940, PMLR, 2016.
10

---

[8] Zhewei Fang, Yujun Shen, Xiao Yang. Xiangyu Xu, Xintao Wang, Bo Dai, and Chen Change
Loy. Fourier neural operator with learnable activation functions. arXiv:2302. 1661, 2023.
arXiv preprint
[9] Remi Genet and Hugo Inzirillo. Tkan: Temporal kolmogonov-arnold networks. arXiv preprint arXiv:2405.07344 2024.
[10] Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks universal approximators. Neural networks, 2(5): 1989.
[11] Victor A Kich, Jair A Bottega, Raul Steinmetz, Ricardo B Grando, Ayano Yorozu, and Ak- ihisa Ohya. Kolmogorov-arnold networks for online reinforcement learning. 2024 24th International Conference on Control, Aulomation and Systems (ICCAS), pages 958-963. IEEE, 2024.
[12] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- tional Conference on Learning Representations (ICLR), 2015.
[13] A.N. Kolmogorov. On the representation of continuous functions of several variables as superpositions of continuous functions of smaller number of variables. Dokl, Akad. Nauk, 108(2), 1956.
[14] Nikola B. Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Anima Anandkumar, and Andrew M. Stuart. Neural operator: Learning maps between function spaces. Journal of Machine Learning Research, 23:1-97, 2022.
[15] Vladimír Kunc and Jifí Kléma. Three decades of activations: comprehensive survey of 400 activation functions for neural networks. arXiv, 2024.
[16] Xin Li, Zhihong Xia, and Hongkun Zhang. Cauchy activation function and xnet. Neural Networks, 188: 107375, 2025.
[17] Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In International Conference Learning Representations (ICLR), 2021.
[18] Dong C Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimization. Mathematical progranming, 45(1):503-528, 1989.
[19] Siyuan Liu, Ruochen Zhang, Huizhuo Qiu, Xuan Zhao, Jiaxu Zhang, Zhen Yang, Zhiqiang Zhang, Qingmin Wang, Jiang Liu, Ting Liu, et al. Kan: Kolmogorov-arnold networks. In International Conference on Learning Representations (ICLR), 2025.
[20] D. J. Newman. Rational approximation to |æ].. Michigan Mathematical Journal, 11(1):11-14 1964.
[21] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318, 2002.
[22] Maziar Raissi, Paris Perdikaris, and George E. Kariiadakis. Physics-informed neural networks: deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019.
[23] David Rolnick, Alexander Amini, Aditi Raghunathan, Pulkit Agrawal, and Thomas Poggio. Reverse-engineering the universal approximation theorem. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[24] Herbert R. Stahl. Best uniform rational approximation of xº on [0, 1]. Acta Mathematica, 190(2)244-3066 2003.
[25] Kunpeng Xu, Lifei Chen, and Shengrui Wang. Kolmogorov-arnold networks for time series: Bridging predictive power and interpretability. arXiv preprint 24Xv:2400.002496 2024.
11

---

[26] Dmitry Yarotsky. Error bounds for approximations with deep ReLU networks. Neural Networks, 94:103-114, 2017.
[27] Kaichen Zhang, Lingxiao Li, and George Em Karniadakis. Low-rank physics physics- informed neural networks (Ir-pinns). Journal of Computational Physics, 463:1 162:111218, 2022.
A Theoretical Foundations
A.1
Computational Efficiency Analysis
FLOP Comparison: The Cauchy activation ((x) 2
introduces 2-3x more FLOPs per activation compared ReLU, but remains significantly lighter than spline-based KANs.
Despite higher per-activation cost, XNet achieves faster convergence. For example, on the Heat equation (Table 7). XNet [2,200.1] reaches target accuracy in 108s vs KAN's 254s. demonstrating that expressivity gains outweigh computational overhead.
Time-to-Solution:
Scalability: For Transformer architectures, Cauchy maintains (nn) complexity where n is sequence length, adding only a constant factor increase in FLOPs.
A.2 Approximation Efficiency
We recall the classical approximation rate result from Cauchy kernel expansions. If ʃ E (P(K), then for any p > there exists rational function J'N composed of N Cauchy basis functions such that:
I - fN llo S N
where (p) depends on the smoothness of f and the domain geometry.
To achieve error at most e, we require:
N = O (-
In contrast, ReLU networks yield O(N- for d-dimensional and KANs offer O(N*k) bounded by spline degree k. Thus, XNet provides tunable convergence rates with fewer neurons for smooth targets.
A.3 Localization and Adaptivity
The activation function:
Φ(x) d1 A2
x2 + d2
satisfies:
lim d(x) = 0, o' (((x) \1 + d2) -0)-2x10 2x + d2)
(x² + d2)²
For localized representation centered at 14, define:
(x(x) \1.( mi) A2,
(x 12, de
This enables each neuron adapt regions of discontinuity local variation. Unlike B-splines, XNet does not require explicit partitioning or fixed grid support.
A.4 Matrix Conditioning and Derivative Properties
Matrix Conditioning: The kernel matrix A with entries A.₃ = (x is dense. Using Gershgorin's
theorem:
A
Aul S
Aly.
the condition number scales as x(A) = O(N) in general, potentially requiring preconditioning for large N. B-spline matrices sparse and banded with x(A) O(1).
12

---

Derivative Computation: For efficient optimization, we compute:
d"
2 - (-1)" (n (n-1) 1)!
d(
d
Cauchy activations are fully differentiable with respect to d,, enabling second-order or adaptive methods.
A.5 Proof of High-Order Approximation
Theorem 3.2 (High-Order Approximation of XNet). Let K C IRd be compact, and let f E (K(K) be real analytic. Let = x= exists a one-hidden-layer XNet XNet function fN E span($) such that
Au A be the Cauchy activation family. Then for any r > 0, there
|| - = O(N=)).
Proof. We now establish the high-order approximation capability of XNet in the uniform norm (K) for any compact set K C Rd. The proof is divided into three steps: we begin in the one-dimensional real setting, then extend to bigh-dimensional polynomials using a classical lifting theorem, and finally conclude by composing the approximations.
Step dinenimnnii high-order approximation via the Cauchy approximation theorem.
Let I = [] C R, and let f E (I(() be any real-analytic function on the interval. By the one-dimensional Cauchy approximation theorem (a special case of Theorem in [16]), for any > f(x)
>m k= t (T)) < E, i.e., we have 9M (x) = Ex= (
|| gM gMlle= = O(M*) for any r 0.
Step 2: Multivariate polynomial approximation by Cauchy activations (XNet)
Let K C Idd be a compact set, and let IP, (IR denote the set of all real multivariate polynomials of total degree most n.
We refer to the General Approximation Theorem in [16]. If the one-dimensional function o(x) can approximate all monomials =t with arbitrarily high-order accuracy the C norm, then for any fixed w R. the functions ó(w" + 6) approximate (w z)* with the same rate. Since any multivariate monomial - =I (z)) can be written linear combination such functions (as shown in [16]). follows that Cauchy activations of the form o(w x + b) can approximate x with arbitrarily high-order convergence in (K)).
Step 3: High-order approximation of analytic functions via XNet
Let f E (H(K) theory (e.g., Bernstein, Mhaskar), there exists a sequence of multivariate polynomials (p.(x)) C F,, (Rd such that
) be a real analytic function on a compact domain K C Rd. By classical approximation
Ilf Pa (k) O(n"")n)))))) for any 0.
From Step 2, each polynomial Pn. can be approximated by function gas in the XNet class:
M
(M( Σ a. dww b.).
with
1p., 9M (le) = O(M :)
for any s > 0.
Applying the triangle inequality, obtain:
Ilf gar ()) I |lf px + Ilp, 9Mllo<(k) = 0(n (MM-))
By choosing M n, we deduce:
|lf
gM g (o O for any & >>0 0.
13

---

Conclusion:
Let ((x)= = 2"40
21 and define the single-layer neural network class (XNet)
NM
E ai ow + bi)
Then for any compact set C Rd, and any function f E C²((K) there exists 9ar E A' such that
|| - 9M ([() O(M-*) for any >> 0.
That is, XNet achieves arbitrarily high-order approximation accuracy for real analytic functions in the uniform norm.
A.6 Neuron Efficiency under Arbitrary Approximation Orders
The table in Section 3.2 summarizes how different architectures compare in terms of neurons required to reach a desired approximation error Here, we provide the derivation behind the scaling laws.
XNet. From Theorem 3.2, we have
| fwll S SCN = N =N=00
This result holds for arbitrary r > 0, assuming f E (K) and sufficient smoothness.
ReLU MLP. For classical ReLU networks [1], the best known approximation rate is:
| - fxll s CN- = N = O(c
which reflects the curse of dimensionality.
KAN. For B-spline-based KANs [19], convergence is limited by the spline degree k:
|| - fv s CN + N = ( ((-1//),
and the degree & typically small (e.g., 3), fixed by construction.
Summary. These derivations support the theoretical table in Section 3.2, confirming that XNet allows flexible approximation with significantly fewer neurons under high-order settings.
B
Experimental Details
B.1
Implementation Environment
The numerical experiments presented below were performed in Python using the Tensorflow-CPU processor on Dell computer equipped with a 3.00 Gigahertz (GHz) Intel Core i9-13900KF. When detailing grids ans k for KAN models, we always use values provided by respective authors (Kan).
B.2 One-Dimensional Discontinuous FUNCTION APPROXIMATION
Figure 4 provides visualizations of approximation results for the Heaviside function, highlighting the characteristic "overshoot" phenomenon in B-spline and KAN methods near the discontinuity, which XNet successfully avoids.
Table 9 presents an extensive comparison of various KAN configurations, showing that even with deeper architectures ([1,3,1)). KAN cannot match XNet's 's approximation accuracy for discontinuous functions. The B-spline results further demonstrate that increasing the spline degree & from 3 to 10 yields only marginal improvements, while XNet provides orders of magnitude better approximation with its layer architecture.
14

---

Table 9: KAN reference
ILJKAN
[LATIAA
Mish
1847
1000
BAGSS
MAA
8-3. 6utb
1.350000
1.900:0
1.908-01
1.30200
3.540.m
00000030
5.98800
44884G
308848
5.78.44
133842
124844
1.0
Original Data
1.0
0.8
0.8
0.6
[(J]
0.4
0.4
0.2
0.2
0.0
-1
-0.5
0.5
X
g1d
grld 10
1.0
g1d 10
grdd 50
g1d 50
0.6
0.6
0.4
0.4
0.2
0.2
0.0
-1
-0.5
0.5
(c)
(d)
Figure 4: Heaviside step function approximation comparison: (a) XNet, with 64 basis functions. (b)
KAN [1,1], with 3, grid 200. (c) B-Spline, with k 3. (d) KAN [1,1], with h:
Table 10: B-Spline Performance metrics comparison for different G and K values. reference
8-Spline
RMISE
MAE
k=30, 0=200
5.84776-01
6.1076c-01
k=3, G=10
9.2871e-03
9.6369e-02
4.7923e-02
1.2255e-02
k= 10, G=50
1.9881e00
4.4588c-00 02
1.0879e 02
k-3, G-200
1.1252c-04
3.3544e-02
4.4737e-03
1=00 G=200
00296-03
3.3210c-02
5.1904e00
Figure 5: exp (1 (sin (x(2 + xx3)) + +zzx1))
Figure 6: exp(
1
Σl
100
sin
2
TE:
100
2
B.3
High--Dmensional Continuous FUNCTION APPROXIMATION
Figure 5 and show the training loss curves for both XNet and KAN on the 4D and 100D test functions.
Figure shows XNet's performance with varying numbers of Cauchy basis functions. Performance improves consistently with increasing width, confirming the theoretical scaling laws.
15

---

1
10
10
10
MSE
MSE
10
10
10-3
1
10-9
102
10a
10*
10a
10a
10
10*
10
Number of Parameters
Number of Parameters
(a)
(h)
Figure 7: Performance of XNet on approximating different functions with varying numbers of
parameters: (a) exp ( (sin (х + xス)) + m33tt)): and (b) exp
100 sin?
xz:
100
B.4 Function Approximation with noise
17
13
300
150
330 38
150
200
(a) Noise
(b) Noise 0.05
(c) Noise
Figure 8: Function fitting tested on datasets with different noise levels: (a) Noise = 0. (b) Noise
0.05, and (c) Noise 0.1. Blue lines represent the noise-free ground truth function, while red dots represent the noisy training samples.
KAN (Noise = 0)
KAN (Noise = 0.05)
KAN (Noise= 0.1)
XNet (Noise = 0)
XNet (Noise = 0.05)
XNet (Noise 0.1)
Figure 9: Comparison of the performance of KAN and XNet under different noise levels. Blue lines represent ground truth, while red lines show predictions. Note that XNet better captures the underlying pattern even at high noise levels, while KAN shows signs of overfitting.
16

---

Figures and 9 provide a detailed comparison of how XNet and KAN handle increasing noise levels. XNet significantly outperforms KAN across all noise levels, with the advantage becoming more pronounced as noise increases. This confirms XNet's theoretical noise robustness through adaptive localization, At the highest noise level, XNet achieves 14.5x lower MSE than KAN while requiring only half the training time.
B.5
Heat equation
For solving the Heat equation under the framework of PINN, the loss function plots of various models (see Figure 10).
15.35
11.20 me
Figure 10: Comparison of KAN, PINN and XNet approximations on PDE loss.
B.6 Poisson equation
Task Description. We aim solve a 2D Poisson equation Vel(z, f(x v). f(x,
2in"sin(Tz (Tæ))si (raz)sin(Tyy) (Ty), with boundary condition v( v(-1, y) v(1, y) = v(r,-1) v(æ; 1) = 0. The
ground truth solution is v(z, y) = sin (rz))si((Ty) We use the framework of physics-informed neural networks (PINNs) to solve this PDE, with the loss function given by
losspde = oloss; + losss
1 7
1
(5)
(2i())) + Uyo (2:)
0
Tbb
where we use loss,to denote the interior loss, discretized and evaluated by uniform sampling of n, points (Ir y/) inside the domain, and similarly we use loss, to denote the boundary loss, discretized and evaluated by a uniform sampling of nip points on the boundary. hyperparameter balancing the effect of the two terms.
0.1 is the
Model Configurations. layer MLP [2, 20, 20, 1]. single-layer KAN [2, 10, 1]. and two XNet configurations with different widths [2, 20, 1] and [2, 200, 1).
We evaluate four network architectures within the PINN framework: a two-
Figure 11: PINN and KAN Performance
Table 11: Comparison of XNet and KAN on the Poisson equation.
Metric
MSE
RMSE
MAE
Time (5)
PINN 12,29.2011
1.7998-005
434344008
2.3000-013
48.9
XNet (20)
1.8811-088
1.36570044
1.011c-144
57.2
(2,0011
5.7450e-08
3.3655-04
18450e004
286.3
(200)
1.997e 09
3,30711-00 08
2.1711e u5
1548
17

---

Figure 12: XNet Performance
1000 B000
100
Figure 13: Comparison of KAN, PINN and XNet on PDE loss. XNet shows smoother and faster convergence, with significantly lower final loss values than other methods.
Comparative Results. XNet-200 achieves approximately 53x lower MSE than KAN while requir- ing only 54% of the computational time. The more compact XNer-20 still outperforms KAN with 3x better accuracy and 5x faster training. Therefore we speculate that the XNet might have the potential of serving as good neural network representation for model reduction of PDEs. In general, KANs and PINNs are good at representing different function classes of PDE solutions, which needs detailed future study to understand their respective boundaries.
B.7 Benchmarking Study of different activation functions
This survey is compared with the existing survey/performance analysis and the experimental perfor- mance analysis of selected activation functions is performed over text data. The German to English translation is used to test the performance of the activation functions over text data. Benchmark Seq2Seq model consisting used for the experiment. The model and dataset downloaded from Kaggle2
Long Short Term Memory (LSTM) based autoencoder network is
Table 12: Experimental results for German English language translation tasks.
Activations
Bleu Score
Activations
Bleu Score
Cauchy
21.47 0.67
Swish
19.51 0.97
Sigmoid
14.59 + 0.47
ABReLU
17.55 + 0.63
Tanh
20.93 + 0.91
LiSHT
20.39 + 0.93
Elliott
14.49 0.96
SRS
20.66 + 0.78
ReLU
18.88 0.86
Mish
19.56 + 1.15
LReLU
18.89
PAU
20.11 1.24
PReLU
20.04
Softplus
16.78 + 0.84
ELU
19.40 1.33
CELU
18.71 0.55
SELU
20.85 + 0.64
GELU
18.75 + 1.83
SIRENs
20.27 + 0.85
SiLU
18.61 0.72
Building upon the experiments conducted by Dubey al. [6]. performed further updates and evaluations to assess the performance of different activation functions in language translation tasks. In these experiments, activation functions were applied the feature embeddings immediately before the dropout layer. The training configuration included 50 epochs, a learning rate of 0.001, and a batch size of 256. Both the encoder and decoder had an embedding size of 300. with dropout factor of 0.5. The Adam optimizer was employed, and training was conducted using cross-entropy loss. The Bleu score [21] with 4-gram evaluation was used as the metric for translation quality, with the mean and standard deviation of Bleu scores over five trials reported for each activation function. The results,
'ttpps: :/ww:w//wwwwwww http://wwwwwlgll.ee/e/e http:///aaaeeeet/ttt///////aaaaa
18

---

summarized in Table 12, indicate that the Cauchy activation function consistently outperformed traditional activation functions such as Sigmoid. Tanh, and ReLU, achieving higher Bleu scores. This suggests that the Cauchy activation function is particularly well-suited for language translation tasks, as it enhances feature representation quality and demonstrates greater stability across training trials.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state our paper's contributions: theoreti- cally proving XNet achieves higher-order approximation rates than MIPs and KANs, and empirically demonstrating XNet's s superior performance across function approximation, PDE solving, and reinforcement learning with PPO. All claims are supported by mathematical proofs Section and experimental results in Section 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: In Section Conclusion and Outlook, we acknowledge limitations of our current work and outline areas for future research, including extending XNet to transformer-styl architectures, multi-modal tasks, and deepening theoretical analysis for high nigh-dimensional regimes.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "LLimitations" section in their paper.
The paper should point out any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
20

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Our paper provides complete mathematical foundations in Section 3, with all assumptions clearly stated. Theorem 3.2 presents main theoretical result with proof sketch, and the complete proof provided in Appendix A.5.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Section 4 provides detailed experimental setups for all three domains (function approximation, PDE solving, and reinforcement learning). We specify model architectures, hyperparameters, and training protocols. For each benchmark, we include specific function definitions, data generation procedures, and evaluation metrics enable reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
21

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide the full code and reproduction instructions in the Supplementary Material (see "supplementary.zip",.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.c cc/ weaiiccgggiu///eeeeeeeeee
wttps//oooSSSsisiioooeoooo coonoollccy olycc) for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurlPS code and data submission guidelines (https: / //iips.cco..c cttps//ccccppccccccccccccccccccc http://ggiie/eeeee
for details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All experimental settings are thoroughly documented in their respective sec- tions, with specific hyperparameters, network configurations, and optimization procedures detailed in each experimental subsection and supplementary tables.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
22

---

Justification: Our paper reports appropriate statistical measures for all experiments. Tables 1-4 include MSE, RMSE, and MAE metrics across all comparative experiments. For reinforcement learning experiments in Section 4.3, we plot learning curves with standard deviation bands across runs to show variability.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Our experimental setup is detailed in Appendix B, where we specify that all experiments were conducted using Tensorflow- CPU on Dell computer with a 3.00 GHz Intel Core 19-13900KF processor. Section 4 also reports execution times for each experiment, enabling readers to gauge computational requirements reproduction.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : /wwp/neuurp...// wwwerrps.c//ppiieee
Answer: [Yes]
Justification: Our research fully complies with the NeurIPS Code of Ethies. We have carefully reviewed the guidelines and ensured our work respects intellectual property by properly citing all prior work, particularly KAN and XNet approaches. These methods pose risks to privacy, security, or discrimination concerns defined in the ethics guidelines.
Guidelines:
23

---

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In Conclusion and Outlook, we discuss both positive impacts (improved efficiency in scientific computing, reduced computational resources for high-precision tasks, interpretability improvements) and potential concerns (limitations in very high-dimensional settings). We emphasize XNet's potential for widespread applications through improved efficiency and reduced computational demands compared to deep learning methods.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Our work presents a mathematical framework and neural architecture that poses inherent risks for misuse. XNet does not involve scraped datasets, language models, or image generators that would require special safeguards against potential harmful applications.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
24

---

We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All external assets used in our experiments are properly cited with appropriate attribution. We reference the original KAN paper (Liu al., 2024) when comparing to their method, and clearly cite all benchmark datasets and environments used in our experiments. The Deep Control Suite environments used Section are credited with proper citation to their original authors.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi toode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We release an anonymous implementation of XNet and associated experiments. Documentation includes training setup, model hyperparameters, and evaluation protocols.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our research does not involve human subjects or crowdsourcing. All experi- ments were conducted using computational methods and standard benchmark datasets.
Guidelines:
25

---

The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our research does not involve human subjects, and therefore no IRB approval was required.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
Our research does not involve LLMs as components of our core method.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy for what should or should not be described.
26


Reviews:
[{"review_id": "GqoZxuT1PH", "reviewer": "Reviewer_Yxzd", "summary": "This paper shows that single layers XNets achieve near-exponential approximation rate. The authors prove and empirically demonstrate that this shallow architecture can achieve superior accuracy compared to both MLPs and KANs. The central thesis appears to be that principled activation function design can be a more efficient source of expressivity than architectural depth.\n\nBased on my research of XNet, it appears that this architecture does not take a large performance hit either, so the results of this paper seem promising.", "questions": "I have a few questions:\n1. How does XNet account for the hierarchical compositionality essential for complex perception and language tasks? How do we think it would perform in these cases? Is there anything limiting it there?\n2. What is the empirical performance of deep XNets, and how does the activation's computational overhead affect the practical trade-off between model size and training speed?\n3. Does the Cauchy activation introduce any bottlenecks for parallelization?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "I would say the strengths are as follows:\n1. The theoretical grounding seems quite solid. This is not just an empirical paper but seems to provide formal theory around convergence rates.\n2. The clarity of the central thesis is nice. Namely, \"expressivity from activation design can be better than expressivity from depth\".\n3. The empirical validation is diverse. We see a range of cases from function approximation to scientific computing to RL.\n\nI would argue there are a few limitations:\n1. There is a focus on the single-layer case. I understand this is probably because the theory is based on the single layer case, however, I would have liked to see empirical results on multi-layer networks. Maybe there could be a way that the theory generalises partially or the theory provides intuition.\n2. I didn't see much analysis of computational overhead. The Cauchy activation is obviously more expensive in terms of FLOPs but I didn't see discussion of how this scales with network size. If Cauchy is to be adopted widely then it would need to make sense from a computational perspective in architectures like the transformer. Based on my understanding, the activation would not change the scaling of an architecture like the transformer but would be nice to have some explanation here.\n3. There isn't much discussion of how the activation function interacts with other parts of training like optimisation or regularisation.\n4. I would have liked more experiments and ablations for the domains chosen.", "was_revised": true, "final_justification": "The responses while taken well are not sufficient to update my score. I am remaining with a 5. Thanks!", "rebuttal": "# Response to Reviewer #1\n\nWe sincerely thank the reviewer for their thoughtful and positive assessment of our work. We appreciate the clear summary and constructive questions. Below we respond to the points raised under *Weaknesses* and *Questions*.\n\n## Weaknesses\n\n### (1) Empirical results focus on the single-layer case\n\nThis is a valid point. We focused on the shallow setting to support our theoretical contributions, which prove high-order approximation for single-layer XNets. However, it's important to clarify that **Cauchy activation is a plug-and-play component** that can be broadly applied to linear layers, convolutional layers, etc.\n\nIndeed, in [1], Cauchy activations have been successfully deployed in **multi-layer MLPs, CNNs, and ResNets** on MNIST and CIFAR-10, demonstrating strong empirical performance across these deeper architectures. We are also actively working on **Transformer+XNet** models, with promising preliminary results.\n\n**Our theoretical focus on single-layer networks** highlights the fundamental expressivity advantages, while the **practical deployment in deep networks** (as shown in [1]) demonstrates broad applicability. In the final version, we will clarify this modularity and cite additional deep XNet experiments.\n\n### (2) Computational overhead and scaling\n\nWe agree that the Cauchy activation introduces more FLOPs per unit than ReLU, due to the rational form. However:\n\n**Current computational analysis:**\n* In most of our benchmarks, we already **report running time** (e.g., Table 5) as a practical proxy for computational cost. For a few missing cases, we will add these in the revision.\n* **FLOP comparison**: Cauchy is **slightly heavier than ReLU**, as expected. But it is **much lighter than more expressive methods** like KANs, which require higher-order Taylor or spline fitting.\n* **Reduce complexity**, especially in large-scale models like Transformers: \n   * Use **fewer layers and significantly fewer parameters**, thanks to XNet's high expressivity.\n   * **Simplify activation settings** in very large models to maintain computational efficiency.\n* For Transformer architectures specifically: The Cauchy activation $\\phi(x) = \\frac{\\lambda_1 x + \\lambda_2}{x^2 + d^2}$ adds a small constant-factor increase in FLOPs (~2-3× per activation vs. ReLU), but maintains O(n) complexity where n is sequence length.\n\n**Empirical Validation:** To address computational scaling concerns, we refer to the comprehensive experiments already conducted in Li et al. [1], which directly demonstrate the computational efficiency of Cauchy activations in deep networks:\n\n**Key findings from [1]:** (1) **PDE solving**: Cauchy activation achieves ~10^-3 loss in under 20 epochs vs. 60+ epochs for Tanh on Burger's equation, with ~3× faster convergence offsetting 2-3× per-activation overhead; (2) **Allen-Cahn equation**: Rapid convergence to 10^-2 loss within 250 iterations vs. much slower ReLU baseline, demonstrating faster time-to-solution despite higher per-operation cost.\n\n**These established results directly address the reviewer's scaling concerns** and show that expressivity gains consistently outweigh computational costs in practical scenarios.\n\n**Revision commitment:** We will add detailed FLOPs analysis and runtime comparisons to provide complete computational cost assessment, building upon the scaling evidence already demonstrated in [1].\n\n### (3) Interaction with optimization and regularization\n\nWe conducted new experiments on MNIST using single-layer networks to study how Cauchy activation interacts with optimization and regularization.\n\n**Strongest result:** With dropout + BatchNorm, Cauchy achieved 97.9% test accuracy with a 1.4% generalization gap, outperforming all other activations tested, including ReLU, Tanh, SiLU, SIREN, ELU, Sigmoid.\n\n**Implicit regularization:** Without explicit regularization, Cauchy reached 97.0% test accuracy with a 2.8% gap, reflecting its built-in regularization from saturation and rational decay.\n\n**Explicit regularization:** Dropout (p=0.2) or L2 regularization consistently reduced the gap to ~1.6%, confirming compatibility with standard techniques.\n\n**Optimization:** All models used Adam with default settings. Cauchy trained stably, on par with smooth activations.\n\n\nThese results confirm that Cauchy supports stable optimization and benefits from regularization, even outside its core regression-focused setting.\n\n\n\n\n\n### (4) More ablations and broader experiments\n\nAgreed. As our main focus was on establishing the theoretical result, we kept experiments minimal. However:\n\n* Our paper already spans function approximation, PDEs, PINNs, and RL.\n* In the Appendix, we now add experiments on machine translation and additional activation comparisons.\n* [1] provides ablation studies across activations, architectures, and datasets (e.g., CIFAR-10, 100D PDEs), which we cite and build upon.\n\nWe will make this experimental coverage more explicit in the final version.\n\n## Questions\n\n### (Q1) Hierarchical compositionality: can XNet support perception and language tasks?\n\nYes. While our paper focuses on shallow networks for theoretical clarity, XNet is **architecture-agnostic and fully composable**. The activation is plug-and-play and can be inserted into:\n\n* Vision models (e.g., CNNs, ResNets)\n* Transformers (we are currently building Transformer+XNet models)\n  * On **small-scale Transformers**, we observe particularly strong performance, especially when replacing the softmax in attention mechanisms with **heavy-tailed Cauchy kernels**\n  * On **large-scale Transformers**, the effects are currently **less pronounced**, and we are conducting further research to understand this phenomenon\n* Recurrent or residual blocks\n\nWe expect XNet to generalize well in perception/language domains, and early results in machine translation are encouraging.\n\n### (Q2) Deep XNets and training speed vs. model size\n\nIn practice, deep XNets **train faster to reach a given accuracy** than ReLU- or GELU-based counterparts, especially on smooth or structured data (see results in [1]).\n\nWhile the Cauchy activation has slightly more compute per unit (due to the element-wise division), this is **negligible in total runtime**, and is **outweighed by faster convergence and better performance-per-FLOP**.\n\nWe will include **runtime plots and scaling tables** in the appendix.\n\n### (Q3) Parallelization bottlenecks?\n\nNo. Cauchy activations are **element-wise and differentiable**, and thus parallelize just as efficiently as ReLU, GELU, etc. There are **no reductions or cross-sample dependencies** that hinder GPU or TPU execution.\n\nThat said, in preliminary experiments on **very large-scale models (e.g., 10B+ parameters)**, we observed some **degradation in performance with XNet**. We are currently investigating this phenomenon and believe it may be due to interaction between large activations and normalization layers. We plan to report detailed findings in future work.\n\n---\n### References\n[1] Li, X., Zhang, H, and Xia, Z. \"Cauchy activation functions and XNet.\" *Neural Networks* 188, 107375 (2025)."}, {"review_id": "8Qc7IeO60F", "reviewer": "Reviewer_FajU", "summary": "This paper proposes a novel approach to learned function approximations, in which shallow, wide networks with rational activation functions are used. In function approximation, PINNs learning and RL tasks, these networks are shown to be more accurate than MLPs and KANs with similar parameter counts.", "questions": "The Cauchy activation function's gradient tends towards zero for very large positive or negative values. Could this lead to problems with learning?\n\nIn the experiments carried out here, the inputs and targets were treated as unstructured vectors as in an MLP. Is there some way of applying Xnets that would take advantage of known structure in the inputs/targets, as in convolutional, recurrent or attention-based models?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nThe approach seems to be both theoretically sounds and empirically well-validated. The overall presentation is clear and effective.\n\nWeaknesses:\n\n1) The theoretical results rely heavily on existing theorems but don't provide citations. For example, even after reading the supplementary material it was not at all clear to me how the bound in Theorem 3.1, which involves high-order polynomials in the denominator, is connected to theorem 3.2, in which the denominator has order only 2. To understand this I had to first find the explanation in Li et al., 2024. This should be explicitly clarified.\n\n2) The results are impressive but fairly limited in scope. Since an alternative is being proposed to MLPs and KANs, which are applicable to a wide variety of tasks, I would ideally like to see some additional tests beyond the small number presented here. In particular, more challenging tasks should be considered for the PINNs example than the heat and Poisson equations. It would also be good to see performance comparisons on real-world datasets.\n\n3) Is it really surprising that ReLU MLPs cannot learn smooth functions or those with discontinuities? Aren't there stronger baselines in the literature for this task?\n\n4) How much better are Xnets than MLPs with smooth activation functions, such as SIRENS, ELUs or SiLUs? The KANs paper compares to MLPs with multiple activation functions, but this has not been done here.", "was_revised": true, "final_justification": "Authors added new results and gave good explanations. This is perhaps a very useful idea, though more thorough comparisons would have been good.", "rebuttal": "## Rebuttal to Reviewer\n\nWe thank the reviewer for the thoughtful and constructive comments. Below we address each point in turn, grouped under *Weaknesses* and *Questions* as in the original review.\n\n---\n\n### **Weaknesses**\n\n---\n\n**(1) Theoretical results lack clear citations; unclear link between Theorems 3.1 and 3.2**\n\nThank you for this helpful comment. We clarify below the logic and originality behind our theoretical results:\n\n- **Theorem 3.1 ([3])** provides the theoretical foundation that functions constructed via partial fraction decomposition—based on Cauchy-type integrals—can approximate analytic functions at a rate of $O(N^{-p})$. Although these partial fraction forms are not used directly as activation functions in XNet, they serve as a mathematical motivation for deriving the Cauchy activation function $\\phi(x) = (\\lambda_1 x + \\lambda_2)/(x^2 + d^2)$ in neural networks.\n\n- **Theorem 3.2 (Our Contribution):** we show that a *shallow* XNet using this specific Cauchy activation achieves an **explicit approximation rate of $O(N^{-r})$ for any $r > 0$**, over compact domains of real-analytic functions in $\\mathbb{R}^d$.\n\nThe connection is: Theorem 3.1 motivates the choice of Cauchy activations, while Theorem 3.2 proves that our specific shallow XNet implementation achieves the promised high-order approximation in practice.\n\nWe believe this is the **first constructive approximation rate** result for a concrete rational activation in shallow networks. We will revise the main text to highlight this contribution more clearly and to make the connection between Theorem 3.1 and Theorem 3.2 explicit.\n\n\n---\n\n**(2) Limited scope of experiments**\n\nWhile our paper emphasizes theoretical approximation in shallow settings, we appreciate this concern and provide targeted additional validation:\n\n**Nonlinear PDE validation**: We conducted experiments on the **steady Navier-Stokes equations** with the Kovasznay analytical solution:\n\n$-\\nu\\nabla^2 \\mathbf{u} + \\mathbf{u} \\cdot \\nabla\\mathbf{u} + \\nabla p =0, \\quad \\text{in } \\Omega$\n\n$\\nabla \\cdot \\mathbf{u} = 0, \\quad \\text{in } \\Omega$\n\n$\\mathbf{u} = \\mathbf{g}, \\quad \\text{on } \\partial\\Omega$\n\nwhere $\\mathbf{u}(x_1, x_2) = (u_1(x_1, x_2), u_2(x_1, x_2))$ is the velocity field and $p(x_1, x_2)$ is the pressure.\nWe used the Kovasznay analytical solution on domain $\\Omega = [-0.5, 1.0] \\times [-0.5, 1.5]$ with $\\nu = 0.025$. \nThis nonlinear, coupled system (velocity + pressure fields) is substantially more complex than heat/Poisson equations. \n\n| Architecture | Parameters | $u_1$ MSE | $u_2$ MSE | $p$ MSE | Runtime (s) |\n|--------------|------------|-----------|-----------|---------|-------------|\n| Tanh PINN [2,100,1] | 401 | 1.07e-02 | 2.12e-03 | 1.41 | 93.72 |\n| Tanh PINN [2,100,100,1] | 10,501 | 7.60e-03 | 8.61e-04 | 2.07e-01 | 153.81 |\n| Tanh PINN [2,100,100,100,100,100,100,1] | 50,901 | 2.14e-03 | 1.03e-04 | 1.67e-01 | 561.89 |\n| KAN PINN [2,3,3,1] | 649 | 1.17e-04 | 2.36e-05 | 1.39e-01 | 1274.52 |\n| **XNet PINN [2,100,1]** | **601** | **5.62e-05** | **1.77e-05** | **1.83e-04** | 237.93 |\n\n*Note: Each architecture approximates the solution functions $u_1$, $u_2$, and $p$ separately using the respective network structures.*\n\nOur single-layer XNet achieves **38× better velocity accuracy** and **912× better pressure accuracy** compared to deep Tanh PINNs, while being **53× faster** than KAN.\n\n**Real-world validation**: \n\n- **Medical Imaging:** We tested XNet on power system state estimation, and on Kaggle Brain Tumor Detection where EfficientNet-B3 with Cauchy classification head improved from 88.2% to 98.0% accuracy.\n\n- **Electricity Forecasting**: We evaluated XNet on a real-world electricity consumption forecasting task using data from the Kaggle dataset: fedesoriano/electric-power-consumption.  XNet achieves 15.6× lower MSE than KAN on US power consumption data while being 2.5× faster.\n\n- **Broader applicability**: We note that [3] has demonstrated the effectiveness of Cauchy activations across diverse domains—including CIFAR-10, MNIST, and high-dimensional PDEs like 100D Allen–Cahn—using multi-layer architectures where they consistently converge faster and to lower error than standard activations.\n\n\nThese results validate XNet's utility on challenging real-world problems beyond our theoretical focus.\n\n---\n\n**(3) Are ReLU MLPs really weak? Are there stronger baselines?**\n\nThank you for raising this important point.\n\nWe agree that ReLU-based MLPs have known limitations in approximating non-smooth or discontinuous functions. Our goal is not to highlight this as novel, but to present **Cauchy activations** as a principled and effective alternative, grounded in rational approximation theory.\n\nWhile classical rational networks (e.g., Padé approximants) can approximate non-smooth functions with convergence rate $O(\\exp(-C\\sqrt{n}))$ (see [1]), they are often unstable in practice. Our contribution is to identify a **trainable, low-order rational basis—the Cauchy kernel**—which maintains strong approximation ability while being **simple and stable to train**.\n\nWe build on the empirical framework of [3] and extend it with:\n* **Formal convergence guarantees** for analytic targets using Cauchy activations;\n* **New empirical results** showing strong performance on discontinuous functions (e.g., Heaviside) even where theoretical guarantees are absent;\n* **Efficiency and accuracy comparisons** with deep ReLU networks and KANs (see Sec. 4.1.1).\n\n---\n**(4) Missing Comparisons to Smooth Activations (SIRENs, SiLU, etc.)**\n\nThank you for the suggestion. While our primary comparison is with KANs due to shared theoretical motivation, we agree it's important to compare XNets with MLPs using smooth activations.\n\n**Strongest result:** On MNIST, a single-layer XNet with Cauchy activation, dropout, and BatchNorm achieved 97.9% test accuracy and a 1.4% generalization gap, outperforming Tanh, ReLU, SiLU, ELU, and SIREN.\nWithout regularization, Cauchy still reached 97.0% test accuracy (gap: 2.8%), slightly behind Tanh/SiLU (97.7–97.8%, ~2.0% gap).\nWith dropout or L2, Cauchy narrowed the gap to ~1.6%, demonstrating good synergy with standard regularization.\n\nWe further evaluated Cauchy in a separate domain: German–English machine translation using LSTM-based Seq2Seq models. Among 18 tested activations, Cauchy achieved the highest BLEU score (21.47 ± 0.67), outperforming Tanh (20.93), SELU (20.85), SIRENs (20.27), Swish (19.51), GELU (18.75), and SiLU (18.61).\n\nImportantly, Cauchy belongs to the family of rational activation functions, which offer strong approximation power even for discontinuous or singular functions. Our results across regression, PDEs, classification, and sequence tasks consistently support this.\n\nWe hope this expanded comparison fully addresses the reviewer’s concern.\n\n---\n\n### **Questions**\n\n---\n\n### (Q1) Could gradient vanishing be a problem for the Cauchy activation?\n\nGreat question. While the Cauchy activation’s derivative decays at large $|x|$, in practice we mitigate this by:\n\n- Input normalization (e.g., BatchNorm/LayerNorm),\n- Trainable scaling (via $\\lambda_1$, $\\lambda_2$, $d$),\n- Skip connections (ResNet-style), especially in deeper settings.\n\nAcross all experiments (including deep CNNs from [3]), **vanishing gradients or training instability were not observed**. We plan to study this more systematically in future work.\n\n---\n\n**(Q2) Can XNet be used with structured inputs, like in CNNs or Transformers?**\n\nYes, XNet integrates seamlessly with structured architectures and can leverage input structure effectively.\n\n**Concrete validation**: We tested XNet on the Kaggle Brain Tumor Detection dataset using EfficientNet-B3, replacing only the classification head with a Cauchy activation. As shown below, the Cauchy variant achieved 9.8% higher test accuracy (98.0% vs. 88.2%) despite a slightly higher test loss:\n\n| Architecture | Train Loss | Train Acc | Test Loss | Test Acc |\n|--------------|------------|-----------|-----------|----------|\n| EfficientNet-B3 (original) | 2.742 | 99.5% | 2.993 | 88.2% |\n| **EfficientNet-B3 + Cauchy head** | **0.0** | **100%** | **5.835** | **98.0%** |\n\nTest loss rose slightly due to class imbalance, but accuracy improved significantly, confirming XNet's structured integration strength.\n\n\n\n**Broader applicability**: [3] demonstrates similar benefits in ResNets and other CNN architectures. The rational activation's superior approximation properties complement structured architectures by providing more efficient feature-to-output mappings.\n\n**Transformer preliminary results**: On WikiText-103, replacing ReLU with Cauchy in FFN layers shows ~50× speedup. Cauchy-based attention mechanisms achieve ~1000× acceleration in small models, though scaling to very large models requires further investigation.\n\nThis shows XNet doesn't just work with structured inputs—it enhances their effectiveness while maintaining architectural benefits like translation invariance and hierarchical feature learning.\n\n---\n\nWe thank the reviewer again for the thoughtful and constructive feedback. In response, we have clarified our theoretical contributions (especially the originality of Theorem 3.2), cited broader experimental baselines, and committed to adding new comparisons (e.g., SIREN) and ablations in the final version.\n\nWe hope these clarifications and additions address your concerns, and would be grateful if you could consider a higher score.\n\n---\n\n\n### References\n\n[1] Newman, D.J. \"Rational approximation to |x|.\" *Michigan Mathematical Journal* 11.1 (1964): 11-14.\n\n[2] Boullé, M., Bloch, G., and Gallinari, P. \"Rational neural networks.\" *NeurIPS 2020*.\n\n[3] Li, X., Zhang, H, and Xia, Z. \"Cauchy activation functions and XNet.\" *Neural Networks* 188, 107375 (2025)."}, {"review_id": "dbMAFz2Vlw", "reviewer": "Reviewer_pTo7", "summary": "This paper introduces XNet, a shallow neural network architecture leveraging novel activation functions derived from the Cauchy integral formula. The authors claim that a single-layer XNet achieves approximation rates that are near-exponential, surpassing the limitations observed in traditional MLPs and spline KANs. The empirical evaluation demonstrates XNet's superior performance on small-scale problems like discontinuous function approximation, PDE solving and PPO tasks from reinforcement learning. The central claim is that network expressivity can be achieved through a principled design of activation functions rather than relying on increased architectural depth.", "questions": "- Table 4. How is the input shape to XNet constant at 2? Should that be 4?\n- Section 4.1.4: If we increase the number of basis in XNet, does it demonstrate over fitting? Similarly if the width of the KANs are tuned do they still show over fitting to noise?\n- PINNs are generally not an effective mechanism to solve simple PDE cases that XNet was tested on. Since the proposed XNets claim to demonstrate expressiveness from activation functions rather than depth, can XNets be evaluated on PDEs like navier stokes where we need deep MLPs to get good accuracy?", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n- Experimental Analysis of \"adaptive\" activation functions  design for shallow neural networks that provides a competitive advantage over learned spline based activation functions (KANs)\n- Strong theoretical grounding for convergence rates, that are better than existing frameworks for shallow neural networks\n- Empirical Validation Across Diverse Domains: The paper provides compelling empirical evidence across three distinct domains: function approximation, PDE solving, and reinforcement learning. The breadth of evaluation provides an initial confidence in XNet's versatility and effectiveness.\n- In all of the provided experiments, XNet outperforms KANs. Additionally the runtime compared to MLPs are similar (on a CPU) whereas in these scenarios KANs are much slower.\n- Experiments demonstrate that the activation function enables robustness to noisy and discontinuous problem settings.\n\n### Weaknesses\n\n- While the empirical results demonstrate broad applicability, the claims in the introduction are too broad and should be toned down. For example the authors claim \"PDE solving: over 10,000× lower error than deep MLPs in PINN settings\". While this is \"true\", it is only demonstrated for 2 PDEs with low complexity. Additionally prior works like KANs which operate in the parameter efficient regime, tend to not scale well when tested on larger problems [1][2][3] (to cite a few examples where KANs were evaluated)\n\n[1] https://arxiv.org/abs/2407.16674\n[2] https://openreview.net/forum?id=yPE7S57uei\n[3] https://arxiv.org/pdf/2503.10632\n\n### Minor Stylistic Nits\n\n- In Table 4 it would be helpful to highlight the best RMSE for each row\n- In Figure 2, making the scales same will make it easier to compare the errors.", "was_revised": true, "final_justification": "The authors have answered my original concerns and it seems to be a significant advancement to using shallow neural networks.", "rebuttal": "We sincerely thank the reviewer for the positive assessment of the paper's theory, empirical evaluation, and the central message that principled activation design can substitute for depth. Below, we respond to the noted weaknesses and questions.\n\n**Weaknesses**\n\n**(1) Broad empirical claims in introduction need to be toned down**\n\nWe agree and appreciate this point. In particular:\n- The \"10,000× lower error\" claim in the introduction (regarding PINN experiments) will be rephrased more cautiously. It is **specific to the Poisson and Heat equations** tested and should not be overgeneralized.\n- We will modify this and other wording in the introduction to **limit scope and avoid overstatement**, especially in relation to PDEs.\n\nWe note, however, that the performance gap shown in these experiments is robust and consistent under **equal model size, training budget, and optimization setup**—highlighting that the performance gain stems from **activation design rather than depth or parameter count**.\n\n**(2) KAN scalability concerns and citations**\n\nThank you for the insightful citations. We agree that **KANs can suffer from scalability issues** due to their reliance on local spline interpolation, which can be hard to optimize and scale, especially in high dimensions.\n\nWe will cite and briefly discuss these works ([1][2][3]) in the revised related work section. In particular, **XNet with Cauchy activation**:\n\n- Uses local basis functions (unlike global rational or spline bases), enabling flexible parameter assignment: per-neuron activation hyperparameters in small models, or uniform parameter settings for scalability—while preserving numerical stability;\n\n- Remains **compact in parameter count**, often fewer than KANs, yet achieves better performance.\n\n**(3) Minor stylistic nits**\n\nThank you—we will address both:\n- **Table 4**: We will bold the best RMSE values per row.\n- **Figure 2**: We will unify the scales across the subplots for fairer visual comparison.\n\n**Questions**\n\n**(Q1) Table 4 — Should the input shape be 4 instead of 2?**\n\nGood catch. The \"input shape\" refers to the 2D spatial coordinate input (x,y) to the PDE solver. The full input including time (e.g., for the heat equation) is indeed 3D, and for vector fields or boundary features it may be higher. We will clarify the notation to avoid ambiguity.\n\n**(Q2) Does XNet overfit when width increases? What about KANs?**\n\nGreat question. In practice, we observed that:\n- **XNet shows mild overfitting** when significantly overparameterized (e.g., >100 basis functions), especially on noisy targets, but it is **less prone to overfitting than KANs**, likely because its rational form provides natural regularization. As shown in [3], this mild overfitting can typically be controlled by **reducing the number of neurons**.\n- **KANs**, on the other hand, tend to **overfit more severely** in low-noise settings when width is large, due to their local spline basis and sensitivity to knot placement.\n\nWe will provide **learning curves and ablations** on width and regularization in the Appendix to further illustrate this.\n\n**(Q3) Can XNet solve harder PDEs like Navier–Stokes, where depth is needed?**\n\nThis is an excellent point. We fully agree that complex PDEs like Navier-Stokes pose significant challenges and often require sophisticated architectures.\n\nTo address this concern, we conducted additional experiments on the steady Navier-Stokes equations with the Kovasznay analytical solution:\n\n$-\\nu\\nabla^2 \\mathbf{u} + \\mathbf{u} \\cdot \\nabla\\mathbf{u} + \\nabla p =0, \\quad \\text{in } \\Omega$\n\n$\\nabla \\cdot \\mathbf{u} = 0, \\quad \\text{in } \\Omega$\n\n$\\mathbf{u} = \\mathbf{g}, \\quad \\text{on } \\partial\\Omega$\n\nwhere $\\mathbf{u}(x_1, x_2) = (u_1(x_1, x_2), u_2(x_1, x_2))$ is the velocity field and $p(x_1, x_2)$ is the pressure.\nWe used the Kovasznay analytical solution on domain $\\Omega = [-0.5, 1.0] \\times [-0.5, 1.5]$ with $\\nu = 0.025$. This benchmark involves coupled nonlinear equations with both velocity and pressure unknowns, making it more complex than our original experiments.\n\n\nWe conducted a comprehensive comparison across different network architectures and depths. We evaluated shallow and deep Tanh-based PINNs, a KAN-based PINN [2,3,3,1], and a single-layer XNet-based PINN [2,100,1]. Using 4,096 interior points and 256 boundary points, our results show:\n\n\n| Architecture | Parameters | $u_1$ MSE | $u_2$ MSE | $p$ MSE | Runtime (s) |\n|--------------|------------|-----------|-----------|---------|-------------|\n| Tanh PINN [2,100,1] | 401 | 1.07e-02 | 2.12e-03 | 1.41 | 93.72 |\n| Tanh PINN [2,100,100,1] | 10,501 | 7.60e-03 | 8.61e-04 | 2.07e-01 | 153.81 |\n| Tanh PINN [2,100,100,100,100,100,100,1] | 50,901 | 2.14e-03 | 1.03e-04 | 1.67e-01 | 561.89 |\n| KAN PINN [2,3,3,1] | 649 | 1.17e-04 | 2.36e-05 | 1.39e-01 | 1274.52 |\n| **XNet PINN [2,100,1]** | **601** | **5.62e-05** | **1.77e-05** | **1.83e-04** | 237.93 |\n\n*Note: Each architecture approximates the solution functions $u_1$, $u_2$, and $p$ separately using the respective network structures.*\n\nRemarkably, our single-layer XNet achieves **38× better velocity accuracy** and **912× better pressure accuracy** compared to the deep Tanh PINN, while being significantly faster than KAN.\n\nHowever, we acknowledge that for highly complex PDEs, depth remains important. XNet architecture with Cauchy activation functions is fully composable and can be used in deeper networks when needed. The key insight is that **depth and principled activation design are complementary**—XNet demonstrates that even shallow networks can achieve surprising effectiveness when equipped with principled activation functions.\n\n\n\n\n### References\n\n[1] Newman, D.J. \"Rational approximation to |x|.\" *Michigan Mathematical Journal* 11.1 (1964): 11-14.\n\n[2] Boullé, M., Bloch, G., and Gallinari, P. \"Rational neural networks.\" *NeurIPS 2020*.\n\n[3] Li, X., Zhang, H, and Xia, Z. \"Cauchy activation functions and XNet.\" *Neural Networks* 188, 107375 (2025)."}, {"review_id": "6o4zJDaF7F", "reviewer": "Reviewer_wzmP", "summary": "Finding a replacement for MLPs at the base of neural networks is an interesting problem to either increase computation efficiency or interpretability for simple networks, which was recently spearheaded by KANs. The authors introduce XNet, a neural network type based on the Cauchy basis functions, which has a theoretically superior approximation result for smooth functions. Experiments are provided to support this claim.", "questions": "- In XNet, do you need to tune the $p$ constant or is that automatically adapted to the highest smoothness order \"detectable\" from the target function? Is there any evidence if you claim the latter?\n- In Table 5, is Time here training/inference time?", "limitations": "- The limitation about the assumption that the target function is infinitely differentiable (as mentioned in my review) is not discussed at all.\n- Limitations about scaling XNet to larger networks should also be mentioned (is the memory usage or computation time a bottleneck).", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "## Strengths\n- The problem of finding a provably better fundamental base for neural networks is an interesting and important question.\n- The construction of XNet is novel given the previous work mentioned.\n- The experimental results are good, although there are some concerns about fair comparison (see Weaknesses).\n## Weaknesses\n- It would be helpful to have a figure showing what Cauchy basis functions are. I believe this would make understanding the idea behind the paper more easily. As an example, the KAN paper has figures showing what the spline functions are and how the elementary operations over the spline functions operate in a layer.\n- Misleading claim: The $p$ constant in the convergence rate for the Cauchy approximation result is the smoothness of the target function class. For most of the main text up to Section 3, this is never explicitly mentioned and it is only mentioned that the rate of $O(N^{-p})$ holds for arbitrary $p$. This is only true for infinitely differentiable target functions. I don't believe this is true for most applications we care about. This in turn makes it confusing why XNet is performing well in Section 4.1.1 because the target function is non-differentiable. Can the authors explain why XNet is performing well despite no theoretical guarantee for this particular case?\n- Lack of clarity in writing: When introducing a new function like the Cauchy basis function, it is important to formally define what mathematical object it is. For example in Sec 3.2, it is not mentioned what the domain and co-domain of $\\phi$ is.\n- Missing baselines:\n\t- In Section 4.1.1, another baseline should be a comparable ReLU network. For non-differentiable targets, ReLU should be a natural fit and a more appropriate baseline.\n- In Sec 4.1.2, the fairness in comparison between different configurations of the network width across different models is questionable. I am not familiar with the computation complexity of KAN and XNet, so I don't know if the much larger width in XNet (500 vs 100 in MLP and 6 in KAN) is fair. This is also true for the configuration in Table 8. Can you give an explanation for why you chose these hyperparams?", "was_revised": true, "final_justification": "The authors have addressed all of my concerns. I have raised my score.", "rebuttal": "We sincerely thank the reviewer for the thoughtful feedback and constructive suggestions. We will revise the text to more clearly clarify the scope of our contributions in relation to existing work, particularly [4].\n\n**Clarification of Theoretical Contribution:**\n[4] introduced the Cauchy function form and established general approximation theorems based on partial fraction decomposition, providing a theoretical foundation for Cauchy activation functions. However, their results are not specific to neural networks. In contrast, our **Theorem 3.2** provides the first **network-specific convergence guarantee**: we show that **XNet**, equipped with Cauchy activations, achieves **super-polynomial approximation rates** $O(N^{-r})$ for any $r$, assuming the target is real-analytic. This is a network-level result that requires fundamentally different analysis. \n\nThis result goes beyond classical algebraic rates $O(N^{-r})$ and directly explains XNet’s empirical parameter efficiency, especially in Section 4.1.1. We will revise the main text to more clearly state the smoothness assumption (real-analyticity) and to clarify the distinction between our contribution and that of related work, including [4], to avoid further confusion.\n## Response to Weaknesses\n\n### W1: Visualization of Cauchy Basis Functions\n**Response:** We appreciate this suggestion and will add a comprehensive visualization:\n- The shape and behavior of Cauchy basis functions $\\phi(x) = (\\lambda_1 x + \\lambda_2)/(x^2 + d^2)$ for different parameter values, demonstrating their nonlinear, learnable nature.\n- How elementary operations (weighted combinations) over these functions operate within a layer\n- Direct XNet vs. KAN comparison:\n  - **XNet:** Continuous, differentiable rational functions with heavy-tailed decay and learnable parameters\n  - **KAN:** Piecewise polynomial splines with grid-constrained, local support\n\nWe appreciate this feedback and believe the added visualization will help clarify the differences between the two architectures and make our approach more accessible.\n\n\n### W2: Theoretical Assumptions and Convergence Class\n**Response:** Thank you for this important point. Our theoretical guarantee (Theorem 3.2) assumes the target function is **real-analytic** on a compact domain. We will revise the text to state this assumption explicitly.\n\n**Theoretical vs. Empirical Scope:**\n\n**Convergence comparison under analyticity:** Under this assumption, XNet achieves **arbitrarily high convergence rates** $O(N^{-r})$ for any $r > 0$, in contrast to:\n- **ReLU or polynomial networks**, where $r$ depends on the differentiability class (e.g., $f \\in C^k \\Rightarrow r \\leq k$);\n- **KANs**, whose accuracy is constrained by spline degree and grid.\n\n**Empirical Scope:**\nEmpirically, XNet performs well even for low-regularity targets (e.g., Heaviside), consistent with classical rational approximation theory:\n- Rational functions can approximate $|x|$ with $O(e^{-C\\sqrt{n}})$ vs. $O(1/n)$ for polynomials (see [1]);\n- Cauchy activations are low-order rational rational functions with flexible shape: $\\phi(x) = \\frac{\\lambda_1 x + \\lambda_2}{x^2 + d^2}$;\n- Rational networks require only $O(\\log\\log(1/\\varepsilon))$ parameters vs. $O(\\log^3(1/\\varepsilon))$ for ReLU (see [2]).\n\n**Revision commitment:** In the final version, we will clearly distinguish between proven guarantees (for real-analytic targets), empirical reach (for broader function classes), and theoretical intuition (suggesting rational approximators generalize better than polynomials).\n\n### W3: Mathematical Clarity and Domain Specification\n\n**Response:** We will add formal definitions in the revision:\n- Explicitly define the Cauchy basis function as $\\phi: \\mathbb{R} \\rightarrow \\mathbb{R}$ with $\\phi(x) = \\frac{\\lambda_1 x + \\lambda_2}{x^2 + d^2}$, where $x$ is the input to a neuron and $\\lambda_1, \\lambda_2, d \\in \\mathbb{R}$ are learnable parameters\n- Clearly specify domains and codomains for all mathematical objects upon first introduction\n- Provide precise definitions of notation and mathematical\n\n\n### W4: Missing ReLU Baseline\n\n**Response:** Excellent suggestion. We will add ReLU network comparisons in Section 4.1.1 (Heaviside approximation). This will provide a more complete picture of XNet's advantages over both ReLU network and KANs architectures.\n\nWe have added comprehensive ReLU network comparisons for the Heaviside function approximation:\n\n| Model | Architecture | Parameters | MSE | RMSE | MAE | Training Time |\n|-------|-------------|------------|-----|------|-----|---------------|\n| **XNet** | [1, 64, 1] | 322 | 8.99e-08 | 3.00e-04 | 1.91e-04 | 25.36s |\n| **ReLU (shallow)** | [1, 64, 1] | 193 | 2.05e-03 | 4.53e-02 | 8.82e-03 | 21.92s |\n| **ReLU (deep)** | [1, 64, 64, 1] | 4353 | 6.81e-05 | 8.25e-03 | 3.76e-04 | 26.24s |\n| **KAN** | [1, 1] (200 grids) | 207 | 5.98e-04 | 2.45e-02 | 3.03e-03 | 28.36s |\n| **KAN** | [1, 3, 1] (200 grids) | 1242 | 1.13e-03 | 3.35e-02 | 4.47e-03 | 56.72s |\n\n### W5: Fair Comparison in Network Configurations\n\n**Response:** Thank you for raising this important concern about experimental fairness. We have conducted comprehensive experiments with runtime measurements to demonstrate that our comparisons are computationally fair.\n\n**Updated Experimental Results:**\n\n| Function | KAN Shape | XNet Shape | KAN RMSE | XNet RMSE | KAN Time | XNet Time |\n|----------|-----------|------------|----------|-----------|----------|-----------|\n| Jacobian elliptic | [2, 2, 1] | [2, 500, 1] | 1.33e-04 | 4.03e-05 | 59.59s | 30.22s |\n| Bessel (1st) | [2, 3, 1, 1, 1] | [2, 500, 1] | 1.64e-03 | 2.39e-04 | 127.63s | 34.78s |\n| Modified Bessel | [2, 4, 3, 2, 1, 1] | [2, 5000, 1] | 9.28e-03 | 6.39e-05 | 271.46s | 68.82s |\n\n*Representative results shown; complete table with all 12 functions will be provided in revision.*\n\n**Rationale for XNet's Higher Width:**\n\nOur comparison follows the experimental protocol from [3]:\n- **MLPs:** fixed width 5/100, depths $\\{2,3,4,5,6\\}$\n- **KANs:** width 5, depths $\\{2,3,4,5,6\\}$, with grid refinement\n- **XNet:** single layer with width chosen to match computational cost of multi-layer baselines\n\nThe updated experimental results demonstrate that **XNet achieves superior performance while being computationally more efficient than baselines**, eliminating concerns about unfair comparison. We will include detailed parameter count analysis in the revised version to provide complete transparency about the architectural trade-offs.\n\n## Response to Questions\n\n### Q1: Hyperparameter Tuning of $r$\n\n**Response:** The parameter $r$ in our $O(N^{-r})$ convergence rate is not a tunable hyperparameter but rather a theoretical characterization that depends on the target function's smoothness.\n\n**Theoretical vs. Practical Considerations:** In theory, the network has the potential to automatically exploit the highest smoothness order available in the target function. However, practical performance depends on several factors:\n- **Training Data:** The quality and quantity of training samples directly affect the network's ability to detect and exploit function smoothness\n- **Optimization Dynamics:** The choice of optimizer, learning rate, and training procedures impact how effectively the network can realize its theoretical potential\n\nThese practical challenges explain why our experiments may not always reveal the target function's true smoothness order. Nevertheless, our results demonstrate XNet's substantial potential—even under non-ideal conditions, XNet consistently outperforms baselines, suggesting that when practical constraints are minimized, the theoretical advantages become increasingly pronounced.\n\n### Q2: Time Complexity in Tables\n\n**Response:** The reported times are total training times. We will clarify this in the caption and add inference time comparisons in the revision.\n\n## Response to Limitations\n\n### L1: Infinitely Differentiable Target Function Assumption\n\n**Response:** Thank you for this important point. Our theoretical guarantees in Theorem 3.2 assume target functions are **real-analytic**. However, our experiments in Section 4.1.1 demonstrate that XNet performs well even on low-regularity targets like the Heaviside step function, which is discontinuous and non-differentiable. We will provide theoretical analysis to include non-smooth targets using rational approximation theory in revision and extend analysis to broader function classes beyond real-analytic functions in future work.\n\n### L2: Scaling XNet to Larger Networks\n**Response:** XNet scales across width, depth, and architectures without compute or memory bottlenecks.\n\n1). **Theoretical:** Cauchy activations add minor cost (division), but preserve O(n) complexity; the overhead is negligible and offset by faster convergence and better performance-per-FLOP.\n2). **Empirical:** Deep XNets reach target accuracy faster than ReLU/GELU, especially on smooth/structured data (see [4]). In Sec. 4.1.2, XNet with 5000 neurons trains stably on special functions.\n3). **Structured models:** On EfficientNet-B3, replacing only the classification head improved test accuracy from 88.2% to 98.0%. On WikiText-103, using Cauchy in FFN/attention gave ∼50×/1000× speedups.\n\nWe will add runtime and scaling results in the appendix.\n\n## References\n\n[1] Newman, D. J. (1964). Rational approximation to |x|. *Michigan Mathematical Journal*, 11(1), 11-14.\n\n[2] Boullé, N., Nakatsukasa, Y., & Townsend, A. (2020). Rational neural networks. In *Advances in Neural Information Processing Systems* (Vol. 33, pp. 14243-14253).\n\n[3] Liu, Z., Wang, Y., Vaidya, S., et al. (2024). KAN: Kolmogorov-Arnold Networks. In *Proceedings of the International Conference on Learning Representations* (ICLR).\n\n[4] Li, X., Xia, Z., & Zhang, H. (2025). Cauchy activation function and XNet. Neural Networks, 188, 107375."}]

Author rebuttal:

Meta-review: The reviewers agreed that this paper addresses an important topic, is clearly written, and has strong results. Some reviewers had concerns about the variety of tasks, models, and datasets in the experimental validations. The authors' rebuttal and further answers during the discussion period seems satisfactory. Given the unanimous recommendation for acceptance from all the four reviewers, I recommend accepting this paper.
Decision: Accept (poster)