Schauder Bases for $C[0, 1]$ Using ReLU, Softplus and Two Sigmoidal Functions
Authors: Anand Ganesh
Babhrubahan Bose
Anand Rajagopalan
Venue: Accepted by TMLR ()

Abstract: We construct four Schauder bases for the space $C[0,1]$, one using ReLU functions, another using Softplus functions, and two more using sigmoidal versions of the ReLU and Softplus functions. This establishes the existence of a basis using these functions for the first time, and improves on the universal approximation property associated with them. We also show an $O(\frac{1}{n})$ approximation bound based on our ReLU basis, and a negative result on constructing multivariate functions using finite combinations of ReLU functions.

Published in Transactions on Machine Learning Research (November /005))
Schauder Bases for C(o, 1] Using ReLU, Softplus and Two Sigmoidal Functions
Anand Ganesh
anandg@rias.ress
National Institute of Advanced Studies, Manipal Academy of Higher Education
Babhrubahan Bose
babberubahabb Wrisc. in
Depl. of Mathemalics
Indian Institute of Science, Bengahuru
Anand Rajagopalan Cruise
maaddrr@gman com
Reviewed on OpenReview: https:// openreview. net/ forum? id= 17790ubbb
Abstract
We construct four Schauder bases for the space C/O, 1], using ReLU functions, another using Softplus functions, and two more using sigmoidal versions of the ReLU and Softplus functions. This establishes the existence of a basis using these functions for the first time, and improves on the universal approximation property associated with them. We also show an ( approximation bound based on our ReLU basis, and negative result on constructing multivariate functions using finite combinations of ReLU functions.
1 Introduction
Functions in spaces such as C'O,, 1] with the supremum norm, L2 1], and (RR can be approximated, or in Kolmogorov's sense represented, using finite linear combinations the form:
C b:)
(1)
Expressions like equation 1 resemble single hidden-layer neural networks. The component functions o + b,), also known as plane waves, ridge functions, or sigmoids, have been widely applied in fields such as finance, data analysis, statistics, and medical imaging [Ismalloy] 2021
Although originally linked to Kolmogorov's representation theorem (Kolmogorov 1957 Sprecher 1965 this
formulation has found broader utility in function approxination, neatly delineated by Sprecher (Demb and Sprecher 2021 This shift is seen in Cybenko's universal approximation theorem (Cybenko) 1989) where is assumed to be fixed sigmoidal function. The Kolmogorov-Spredhee representational approach uses function compasition, like two layer network, and the function a depends on the specific target function f. In contrast, Cybenko's approximation perspective is like single layer network with no function composition, where a is fixed and independent of f.
Our work extends beyond these two perspectives by constructing infinite series representations using fixed activation functions. Specifically, we present four Schauder bases for C(O, 1]: one based on the widely used ReLU function (Theorem another based on its smooth variant, the Softplus function (Theorem and two more based sigmoidal versions of these functions (Theorem 2 Theorem 6).

---

Published in Transactions on Machine Learning Research (November/ ) 2022)
There is a lot of interest in the neural network community on understanding the expressive power and approximation capabilities of ReLU based networks. Our basis results can be applied towards these questions. For instance, Theorem 7 shows an order O(2)) approximation bound based on the first n basis functions. This O bound based on the supnorm on C 1] is stronger than the ( L bound in (Barron 1993) since the L norm on C(0, 1] dominated by the supnorm. Further note that the L error bound represents sort of average expected error while result reprosents worst case error offering better protection against outliers. Our result is also improvement, in terms of number of layers, on the construction of (Daubechies et. al. (022) for 0, 1}, as they require a multilayer network compared to our single layer basis or network, to obtain a ( L approximation bound for univariate functions. In limited way for univariate functions, our approximation result is also an improvement on Savarese et al. (2019) where they look at infinite width ReLU networks for functions s + R. Whereas their O 1,² error bound depends on target-specific basis functions, we achieve the same error bound with a fixed basis.
Finally we show a negative result in Theorem that finite linear combinations of ReLU functions cannot represent multivariate functions in general, thus suggesting the need for deep neural networks to support multivariate functions. We hope to generalize this result to countable linear combinations, but this step remains open at this time.
In the context of neural networks, having a basis offers certain potential benefits compared to density results like those in Cybenko| (1989). For example, a basis allows for structured initialization in neural network training. Consider an infinite expansion of the form:
f(x) = D6:
(.),(2)) where o,(x)) = (wxx x by)
This expansion is to be interpreted in the sense of Schauder basis, as discussed in the next section. A finite truncation yields:
M
f(x) ~ fM(() = Σ (1)
Here, far(x) can be seen as neural network of width M. To improve the approximation, one could consider wider network fx( with N > M, retaining both the functions o, and coefficients a, for i S M, and only learning the new coefficients for i > M. This reuse of parameters from narrower model is theoretically justified only when an explicit basis is available. In contrast, density results offer no such structure and require retraining from scratch. Another possible advantage of using a basis is that it ensures unique expansions, and thus a unique global minimum during training.
In Banach space literature, functions similar to ReLU have appeared in basis constructions, such as the Schauder hat and restricted hat functions (Semadeni] 1982, p.28). However, the restricted hat, which is essentially ReLU, is only used to describe the boundary behavior of Schauder hat function, and not to construct a full basis. The ReLU and Softplus functions have not been previously employed for full basis constructions, possibly because their development was motivated by later applications in neural networks.
Within ridge function literature there is work on a universal sigmoidal function independent of the target function ʃ. The existence of such a universal function for target functions in C(RR), with a prescribed approximation error, and a prescribed number of neurons is guaranteed by a theorem of Maiorov and Pinkus [Ismailov] 2021 p.158), and an algorithmic construction is provided by Ismailov] 2021 p.144). Our basis construction (Theorem [] seemingly provides such universal function for C 1), but this is misleading as the number of neurons will depend on the level of approximation desired. But again, the ability to approximate arbitrarily well with a fixed number of neurons depends on the use of wild and pathological functions, as Sprecher notes in Demb and Sprecher (2021), or the use of intricate algorithms as in Ismailov's smooth, almost monotone construction. In contrast, we use simple and standard functions like the ReLU our basis constructions, and do not require the use of function composition.
From a theoretical standpoint, Cybenko (1989) notes that completeness results typically fall into two broad categories: those related to Weierstrass's 's theorem polynomial density and those based on Wiener's
2

---

Published in Transactions on Machine Learning Research (November/ /
translation-invann systems. In a way, our results incorporate aspects of both, much like Schauder's original basis from 1927.
Our construction, like Weierstrass's 's result, is situated within C(O, 1] and relies on discrete bump function the Schauder hat. At the same time, employs scaled and shifted versions of a single activation function, akin the translation-ivariiant methods of Wiener. This use of scaling and shifting is common in L 1] literature, and in wavelet thoory by extension.
2 Preliminaries
A countable sequence { in Banach space X is basis for X if for all in X there exist unique scalars an (xx) such that
I ag (x)
(2)
Heil 2010 A normalized basis basis (2n) with ||,,||| 1 for all Schauder basis for is a basis for X where the a, are continuous linear functionals. There are other definitions of Schauder basis that involve uncountable index sets, but we restrict ourselves to the above countable version. As it turns out, any basis for Banach space is Schander basis. When X = C 1] use the standard supnorm topology. Since C(O, 1] is not a Hilbert space there is no notion of an inner product, and the linear functionals an (z) are the closest approximation to the usual coordinates (xm. 2) based on an inner product ...
where the above series converges in the norm
The ReLU function r(x) defined as follows:
0
if r
r(x)
(3)
x
if 2.
The parameterized Softplus function defined as follows
Dugas et al. 2000
Pa(x) In(1 + ta
(4)
A function a IR + R is called sigmoidal if
lim (22) 1
(5)
lim (xx) 0
(6)
Schauder's original basis functions for C(O, 1]
(Heil 2010 p. 142)) are defined by the ordered set S
{sn,k|n NU NU & {0, 1...,.... 1}} under dictionary ordering, where
k+
if x
2
2
8n, (x))
linear 2 otherwise.
k+
and k+
k+1
(7)
0
Please note that (x, = 1 in the first case rather than (x) 1 is standard. This change makes no difference to the basis propcrty, but helps with some downstream calculations.
Given ʃ C( let us demote its basis expansion by:
f = ao + +1181 + Σ an,k-pnkk
(8)
k=
3

---

Published in Transactions on Machine Learning Research (November/ /202m 2025)
Besides &n,k the Schauder basis expansion contains two additional terms aoX(o.1) and 0131. These involve two basis functions, namely the characteristic function X(0.1] and the linear function 81 (1 = I. We assume standard ordering for the Schauder basis, which is essentially a dictionary ordering of (n, k). The
ordering important for convergence since the basis is conditional. We will now look al a basis construction for C²OO, based on the above ReLU function r(z).
3 ReLU Basis
We now construct a Schauder basis for C(O, 1] using the ReLU function r(x) and two auxiliary functions. In particular, we prove the following theorem:
Theorem 1. The basis functions X(o1)(z),, s1(x) x, r(2"x k) and r(2" (k + 2)) form a Schauder basis for C0, 1]. In parlicular, borrowing on,k from the Schauder basis exponsion equalion 7 and selling an 0, have
20 2
1
f = 000 + +a181+ +
Σ (annk + an.k- 1))(== - k) - 2an,xr(2"x (A: + 211
(9)
where the coefficient functionals ooo, 01, (Ahn, + (n.k-11 and 22 are all bounded.
Please note that we have simplified notation by implicitly restricting our basis functions to [0, 1], and we will continue to do so. iee we will write f(x) for [())]o
We begin our proof with the following lemma:
Lemma Define tnk as follows:
(x) = r(2"xx - k:) 2r + ( + r(2"x (k: (k+1))
(10)
We claim that (x,)(x)= (x) for all E IR.
The proof of this lemma is entirely routine. The heart of Theorem is in the specific construction of fn and the proof of boundedness which follows the proof of Lemma [ below. Not all linear combinations similar to
fn lead to valid Schauder basis. For instance, consider r(2"x k) - r(2"xx - (k: + h)) r(-2"x +
(k + })) + r--2" 2"xx x + (k: + 1)) - 1. We can prove that 9v.2 = sn, (x), but unlike Ln,k, Io.R does not lead
to ReLU basis. The boundedness proof of Theorem does not work for 9n.k-
Proof. Let x =
k+8 2n
Then,
fn = r(2" k k r(2n
+
(k + ()) + r(2n to +
k))
2r(2"
(k: + 1)) 2
2n
2n
1
-
r(5 1 1).
2
We will now establish equality of tn and Sn,k in all the four cases listed in equation
For b+1 we have 8 =
1 5) + r(6
1n,k = r(0) 2r(6
r( 1
2r(0) + +r( -1
2
r((
= 2

---

Published in Transactions on Machine Learning Research (November /0025)
8 [0. j] Thus, For 2 2
4+1
tn = = = 2"xx k,
which is linear in x.
Similarly, for x E
1 kt1], we have E 122 1]. So, 2
1
t, = =
2r(6
2
+ r(6 1)
1
6 2(8
+
2
-1-6 8 (2" k)
k + 1 x",
which is also linear in a.
Next for 8 < O. Thus,
1
mn = r(6) 22 2 + r(6 1)
Finally, for k-1 2"
8 and
1
tn, r(6) 22
2
+ r(5
1
= - 2(8
(8 1)
2
0.
We have (x) (,,((x) (I) in all the cases considered, and thus (x) (n, for all x E IR
We will now prove Theorem on the boundedness of the coefficient functionals.
Proof.
00
ooX + 0151
m=0
k=0
2
00X[0,1] + 0181 +
an.tta
==0
k=0
00
2
an."(r(2"' 2r(2"x (k ) + r(2"x (k+1))}
aoX(o.1] + 0151
2
coX[o,1] + 0181 +
(2 x-
k)) - 22 - (k + 1) + an, - (k + 1))}
n=0
k=0
2n
coX(o,1] + 0151
(an,or(2"z) +
Σ
flankk + an,k an, k) 22n (k 2 (x)
2
(an + an k) 20 - (k + 1 1)1
00X + 0181 +
m=0
km0

---

Published in Transactions on Machine Learning Research (November/ /202m
where, for convenience, we have set an,-1 = O in the last step. Notice that equation does not involve any rearrangement of terms. The grouping of (0n, & + Ox,k-1) involves only associativityy, and no commutativity. In particular, the conditional convergence of the carlier scries and that of equation are equivalent.
Finally, given that the coefficient functionals an,k are bounded, the coefficients of the ReLU expansion, namely, ag, 01, On,k + 0n1 and n2a,, are all bounded as well. This establishes that the sequence of functions (2), $1 (z)
æ,, r(2" k) and r(2" x (k 2) form Schauder basis.
]
Remarks:
Let t(x) be defined as follows to be the discrete second derivative of r(z):
((x) - r(x) 2r(x 2 + r(r-((1))))
(11)
((x) is a triangular bump or hat function which the essential building block of the original basis of Schauder. Letting d(x:) denote the expression r(z)-r(x- 2), we can see that t(x) matches the expression d(z)- )).
Here d(x) can be interpreted as the first discrete derivative of r(z), and t(x) as the corresponding first discrete derivative of d(z), in effect the second discrete derivative of r(z).
The subscripted functions (=),(=),, (x)) and (x) are dyadically scaled and shifted versions of r(z)), d(x:) and (x)), for instance (x) = x(2"x= k). Correspondingly, dn,x(x) is the first discrete derivative of Tn. (x) and (2) is the first discrete derivative of dz,,(x),, or in effect the second discrete derivate of (x,)(z). It is well known that the Haar basis elements represent the first derivative of the Schauder basis elements. The current construction shows that the Schauder basis clements represent the first discrete derivative of dn and the second discrete derivative of the ReLU basis elements Tn,k-
da.x(a)) is continuous sigmoidal function, and thus dense in C(O, 1] per Cybenko (9899). Going beyond Cybenko's result, da,k can be used to assemble basis as shown in the following theorem:
Theorem 2. The functions dn))(r) with n 2 and I 2" 1 along with the auziliary functions X (x) and si (x) forms Schauder basis for C(O, 1] with exponsions of the following form for all f in C(O. 1]:
2
f = co0 + 0181 +
dn - ankdnak-
(12)
== k=0
where the coefficient functionals are all bounded.
Proof. We start with the basis expansion of f using the basis clements tnkk where ay and an,k are borrowed from the Schauder basis expansion equation 7
2 an,
f +0180 +
+
2
= auX + 0113 +
an, (dr,k - dn,
R=0
k=0
00 2"
-o + 0181
an - ankdn
==0
This concludes the proof since the coefficient functionals ag, an and ane are all known to be bounded. 7
We note general principle at play here with regards to first and second discrete derivatives. In particular, if fj E X form a Schauder basis for Banach space X, and if fj are the first discrete derivatives of gj and the second discrete derivatives of hi, then Jj and h, form Schauder bases for X as well. One may argue that the basis property of hy follows from the basis property of gj by induction, but an examination of the above proofs shows that some care is required to avoid double counting as seen the functional (an, + an,k-1).
As noted earlier, da, forms basis, but d(x) is not a universal sigmoidal function for C(0, 1]. Unlike the construction of Ismailov 2022 p.164) for C(R), the number of terms in series truncation increases with the desired level of approximation.

---

Published in Transactions on Machine Learning Research (November/ /202) 2025)
4 Softplus Basis
Schauder bases possess stability properties wherein the basis property holds even if each element is perturbed slightly. We use this stability property to perturb the ReLU basis and obtain a basis using Softplus functions. In particular, we will prove the following theorem:
The basis functions (x) (1) 2, Pa(n,a)22'x k) and Po(n. (2" (k + +)) form Schauder basis for 1]. In particular, given in ClO, 1], we have the following basis expansion with a(n,k) k) 4ln 2K 2
Theorem 3.
2
1
f = [0 +7181
+
1m (2 - k) (n(n (2" x (k 1
(13)
k=0
where the coefficient, funclionals 'oo, T1 7n.k and wn. are all bounded.
We start by recalling some stability properties Schauder bases. First, we have the notion a basis constant whose existence is asscrted in the following classical thcorem.
Theorem 4. Lindenstrouss and Teafriri, 1977) Let {un} be a Schauder basis of a Banach Space X. Then the projections Pa X + X defined by P.. Pn a,3;)) = Σiz 0,3; are bounded linear operators and sup,, || <0. The supremum K = sup, P/ll is called the basis constant of {xn).
We then have the following stability property:
Theorem 5. [Tindenstrauss and Tzafriri, 1977, Prop. 1.a.9) Let {xr) be a normalized Schander basis of Banach Space X with basis constant K. If (yn) is sequence of vectors in X such that E || 1/lnn 2K then {yn} is also Schauder busis of X. This properly holds for any fixed normalization constant >> 0 such that || =c n.
To prove that we construct basis with Softplus functions, will start with an intermediate basis whose basis elements & are defined as follows using the parameterized Softplus function:
(x))(x)) (2 k) 2p (2"x: (k+ + 2)) + pa(2" (k: 1))
(14)
a = a(n, k)) 4 4 2. 2K 22n
(15)
We then have the following lemma::
Lemma The functions (x),)(2), s1(x) x and (x,)(2) (n 2 for C(O,1)
0,0
form Schauder basis
Proof. Let us apply the stability property in Theorem 5 with X C(O. 1] to perturb the ReLU basis tn Let K denote the basis constant for the ReLU basis and note that the ReLU basis is a normalized basis with ||tn,xl| = as required by Theorom
We will perturb the ReLU basis elements ty using a parameterized Softplus function Pa(x) instead of the ReLU function r(x) to obtain our new basis elements 4n,k- As we show below, increasing a a suitable multiple of 2 will ensure that the individual perturbations get smaller, and that the total perturbation across all basis elements remains small.
A simple analysis of the parameterized Softplus function Pa (z) equation 4 shows that (pa(x) r(x)) attains its maximum at x = 0, and this maximum value is (0) ln2 This maximum value drops as when we increase the sharpness parameter a.
We further observe that for fixed, sup (a. - k) r(2"r k) sup (.)x)) - r(x) In2 That is, the supremum of the difference between Pa and does not change when the function parameters scaled and shifted. Since sup, |aa (z) - r(æ) equals llp. - -rl,,
that the norm of the perturbation llp. r|l

---

Published in Transactions on Machine Learning Research (November/ /
remains unchanged for given even when the function parameters for (z(z) and r(z) are scaled and shifted. This gives us the following bound on the perturbation error between individual basis elements l,k and Lx
||tn, gn = sup [r<(2= - k) 2r(2"xx2"" - (k + 2) + 7(2"2 (k + +)))
(pa(2"x - b) 22- (k + b)) + Pa (A: 1)))
1 1 2)) pa (I: + 2021
Ssup (ap"rr22'x - k) Pa (2" - kb) + 22r - (k +
+ 1r(22" - (k: + 1)) - p2 (k ((+1)))
sup [r(x) p.() 2/r(x) px)))) + |r(z) Pa (r)
- 4sup (r(x) p2))
4 a
The total perturbation error A across all basis elements can now be bounded as:
A
||tn, ku,kl
2
4ln
0=0
1
2K
2a+2 n
R=0 k=
2K D==
20+2
2K
Since the total perturbation A <
we conclude that lo,k is Schauder basis for C(o. 1]. This proves
Lemma 2
Notice that lr.6 form smooth basis, but the elements of this basis are not scaled and shifted versions of single mother function. The sharpness parameter destroys this scale-shift property of the basis. We also note that this construction cannot be based on sigmoidal functions like tanh since they lack a sharpness parameter to control the supnorm error, and to thus bound the total perturbation.
We will now prove Theorem establishing that Pa(x) forms a basis. We mimic the proof of Theorem though make use of the basis elements In.k defined above instead of the original Schauder basis elements
kn
8

---

Published in Transactions on Machine Learning Research (November/ 2025)
)
Proof. We will start with basis expansion using In.k denoting the coefficient functionals as B, and Sn.k:
0
2
f == -oX + B181 +
Qu.k
0=0
k=0
Ba,k Ba (pa 2pa(2" +k+ 1 2)) pa - (k+ + (1))
= = +1811
==0
k=0
2
(Bn,tPa xx -k) 28n - (k + 1 2) + Bn,kPa (k: + 11))
= = + +1881 +
Σ
n=
k=0
2n
=00[[o.1] + +1111 +
Σ
(Pm (2"
+ [( k + (2,,k-))a - k) P3 (k }) (*)
Σ
n=0
00
2
1 2 (Brnk + -m (nna-1)pa(22= -k) - 103n, (Is +
= 10 + +
+
12
k=0
where, for convenience, we have set 30,-1 = 0 in the last step. Like in the earlier proof, we note that equation x preserves conditional convergence. Finally, given that the coefficient functionals B, and 8n,k are bounded, the coefficients the Softplus expansion, namely, To = fo, ?1 = $1, Tn. = (Bn, + Bm and Vn.A = -23n,k are all bounded as well. This establishes that the sequence of functions [ 81 (x) = x, Pa(2"x x k) and (2" (Ie + 2) form a Schander basis, and concludes the proof of Thoorem 3
As a simple corollary, we construct a sigmoidal basis based on Softplus functions. The construction of un.k mimics the construction of the first discrete derivative dn,k in Theorem 2
Theorem 6. The functions (x) defined to be Pa(n, (2" k)) Pa(n.a (2 - (k + 2)), with a(n, k) 4 2- 2K 22n+22 , for n 2 0,0 < k < 2" 1, along with the auriliary functions X(0. and $1(x) = x form a sigmoidal Schauder basis for C(O, 1]
Observe that Un, are indeed sigmoidal functions since lim (x) equals 0, and limz-+oo un (æ) equals 1. It is also easy to see that un are smooth, monolonically increasing functions. We skip the proof of the basis expansion as it follows the same lines as Theorem 2 except we use In,k for the initial basis expansion instead of kn,k.
5 Applications
In machine learning theory, it is well known that in the univariate case, achive O(+) approximation error with N functions. We prove that our basis expansions achieve the same order of approximation with N functions based on a simple interpolation scheme. Sctting (I:) = r(2"x - k), have the following:
Theorem 7. Given Lipschilz function ClO,, with Lipschilz conslant
O, and basis expansion f = anXo + 01X1 + En= Ek=olan,Tn. + Bn,Kn,,kt)) }), can obtain O approrimations using the first N basis functions. In particular, given the first N basis functions referred to in general as b;, there exist Bi such that ||ʃ > 3,bell < K
N for N sufficiently large, and for some fized constant K independent of N.
Proof. The argument is a standard one for Tipschitz functions based on interpolation of equispaced points. Let us first set ag = f() and 01 = f(1 - f(0) where xo(x) = 1, the constant function and xi (z( x, the identify function. We see that the function g = (auXu + @1X1) has g(0) = 0 and g(1) = 0. Based on this simple device, for the rest of this proof, we will assume without loss of generality that f()) f(1) = 0 and do our interpolation based on the other basis functions without using Xo and X1-
We will derive 0 ) approximation error based on the Schauder basis (x), and use simple counting argument to prove that the same ) approximation error carries over to the ReLU basis as well.

---

Published in Transactions on Machine Learning Research (November/
) 22025)
Given N > O, let P= 2" be the largest power of 2 less than or equal to N. Clearly P N Notice that the first. P terms of the basis expansion include all the basis functions of the form 82,,8 for k E [0, 2P). The peaks
2k+1 for h - 2D 1. Let us now of these basis functions are the dyadic points 21 = 20+1. 22 = 20t1...... IA = 2++1 is clear that [r(xx) ) - f(xxx) V k since
3
build an interpolating function fP as fr(æ)= = Σ (x the support of each basis function $gr,k: essentially the hat, is limited to the interval [zy. 2 Ik + z/r]. Further, fo is piecewise linear in (Ix 2 Ft 2241) ). and for x E (3% 20-1, 2h 2=+1); ), and x + 3kk we have (x) S c, where e is the Lipschitz constant of f. This immediately gives us the approximation bound
we are looking for:
1fr(x) - f(x) = lfp(x) f(xx) f(x)) f(x)
1fp(z) - fp(2x) + f(xz) f(x)
S (f)(z) - fr((x)) + [f(x)) - f(x)
2c 2c
= 0 1
- - N.
p
as required for the Schauder basis kn,k. Now, we note that P Schauder basis functions 8n.k are subsumed by 2 P ReLU basis functions r(2"x k), r(2"x 2). In other words, the N-term approximation error for ReLU functions is bounded by 2 ( as required.
We end with a simple but intriguing proposition that suggests the power of depth in ReLU networks. The proposition is a negative result that our basis property does not generalize to dimensions 2 2, and that multilayer networks are required to represent multivariate functions in general. We do not have proof of this proposition yet, but we offer a preliminary result involving finite sums instead of countable sums that conveys the general idea, and we believe this idea can be generalized. This negative proposition ties in naturally with the Kolmogorov-Sprocher representation theorems which require function composition, in effect 2-layer nelworks. It is interesting future work to construct such 2-layer or multilayer network, perhaps based on the pyramidal functions of (Semadeni 1982).
Generalizing the theorem below from finite lincar combinations to countable lincar combinations will prove the actual proposition we are looking for, that one cannot have a Schauder basis for C(O, 1] based on ReLU functions.
Theorem 8. There erist functions in C10, 1jd that cannot be represented using a finite linear combination of ReLU functions of the form r(wT x + b).
Proof. We will exhibit a pyramidal function f: [0. 1J2 + R that cannot be expressed as a finite combination of ReLU functions. From this, we can obtain similar negative results for fa : [0, 1j - R with d >>> 2, by setting fa(#1, =2) := f(11, 22). To verify this, let us assume, for the sake of contradiction, that fa can be expressed as a finite sum of ReLU functions as fa(xi,,2,,... xaa) = Em arr (w? (11, .2,.,,,)) + b,). Fd O, we get x2) Em aur (w!" (x1, + +b;) where w/ denotes
In that case, setting 33 the vector consisting of the first two coordinates of w;. This violates the assumption that f cannot be represented as a finite linear combination of ReLU functions.
Now consider a pyramidal function f [0, 1|2 + R (as visualized in Figure Appendix A with peak value of 1 at (2. )) and zero on and outside the basc The base, denoted as B, is given by the set of points enclosed by the lines x 1. = 1. -= 1. y = I, or equivalently as the convex hull of {(E, $), (3. }), (2 2), ( ) We claim that this pyramidal function I cannot be constructed using finite number of ReLU functions. Suppose to the contrary, that f(x)= = Edn Zimi asri(z) where I E [0, 1]2 and ri(x) = =(wff x + b.). Without loss of generality, we assume that none of the w; or a; are zero, and the tuples (w, b,) are distinct.
For a given r; define the associated line 1, by fa E [0, 1]2 | w cofx+bb = 0}. Notice that for each of the lines 4; ra is zero (or off) on one side of the line, and affine (or on) on the other side. Note that the 4 along with the four boundaries of the unit square partition [0, 1]2 into convex polygonal regions (which we take to be closed), and we denote the set of these regions as R. Given two regions R, Q E R, we call them neighbors if
10

---

Published in Transactions on Machine Learning Research (November/ /
they share some part of an edge. A region R E R. is called a zero region if f(x) A z E R, and a non-zero region otherwise.
Our main observation is that a zero region cannot have another zero region as a neighbor. To see this, let us first define 2, [0, 1]2 + {0, as 2,(2) = whenever r, r2(2) I 0 and 2x(x) 1 whenever r;(2) > 0. 2((x) is a binary valued function which tracks whether r; is on or off, and thus r;(a)) = ==(x)( x x b.). For a given region R. E R and fixed i, (z) is the same for all
R. It is clear that crossing the line 4 changes the value of 2; (x)) from zero to one vice versa, and that 2j remains unchanged for + i.
Let us now consider the weight matrix w(x) E (,z(()) x and bias matrix b(x) - - (22( Within given region R R, w(xx) and b(x) remain constant since 2;(x) remains constant. Let us denote by wR, the constant weight matrix for R, and by bg, the constant bias matrix. Now note that WR must be zero for a zero region R. Indeed, if wn were not zero, we can find z E R such that w ba F O. Now, given neighboring regions Q, R E R with a common edge 4i, wi wn E oxwx(()) which implies that wR and wQ cannot both be zero since wi; 0g * O. This verifies the main observation above that a zero region cannot have another zero region as a neighbor.
Let us now consider one of the corner points P of the base of the pyramid B, and all the regions outside B which contain P a vertex. There must be at least two such regions since the regions are convex, and they subtend a total angle of 3 ? at P. But all these regions must be zero regions since is identically zero outside B, which forces two adjacent zero regions, contradicting our main observation.
Thus, the pyramidal function f cannot be represented as finite linear combination of the form f(x) = Σa (.r((r).
Note that the method of the proof can be used to generalize the above theorem to pyramidal functions whose base is a convex polygon strictly contained the unit square.
6 Conclusion
We constructed four Schauder bases for C(O,, 1], one using the ReLU function (Theorem D another using the Softplus function (Theorem 3/, and two more using sigmoidal versions of the above (Theorem 2 and Theorem The last basis consists of smooth, monotonically increasing sigmoidal functions. Finally we show an approximation bound using our ReLU basis (Theorem 7 and negative result on constructing multivariate functions with finite linear combinations of ReLU functions (Theorem
In terms of future work, we wonder scaled and shifted bases for C(O, 1] are possible using smooth functions. In particular, we pose the following question: does there exist a smooth function o : [0, 1] + R such that on,A (3) (2 - k) forms basis for C(O, 1]
Acknowledgementt Dedicated to thatha-patts Our sincere thanks to Prof. Nithin Nagaraj for supporting this work, to Prof. K B Sinha for early comments on problem formulation, and to the anonymous reviewers for the time spent, and helpful pointers connecting our mathematical results to machine learning theory.
References
Vugar F. Tsmailov. Ridge functions and applications in neural networks. American Mathematical Society, 2021.
AN Kolmogorov. The representation of continuous functions of several variables by superposition of contin- uous functions of one variable and addition. Doklady Akad. Nauk. SSSR, 1957.
David A Sprecher. On the structure of continuous functions of several variables. Transactions of the American Mathematical Society, 1965.
Robert Demb and David Sprecher. A note computing with kolmogorov superpositions without iterations. Newral Networks, 2021.
11

---

Published in Transactions on Machine Learning Research (November/ 2025)
G. Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems, 1989.
Andrew R Barron. Universal approximation bounds for superpositions of sigmoidal function. Transactions on Information theory, 1993.
IEEE
Ingrid Daubechies, Ronald DeVore, Simon Foucart, Boris Hanin, and Guergana Petrova. Nonlinear approx- imation and (deep) ReLU networks. Constructive Approzimation, 2022.
Pedro Savarese, Itay Evron, Daniel Soudry, and Nathan Srebro. How do infinite width bounded norm networks look in function space? In Conference on Learning Theory. PMLR, 2019.
Zbigniew Semadeni. Schauder bases in Banach spaces of continuous functions. Springer, 1982.
Christopher Heil. A basis theory primer: expanded edition. Springer Science Business Media, 2010.
Charles Dugas, Yoshua Bengio, Francois Belisle, Claude Nadean, and Rene Garcia. Incorporating second- order functional knowledge for better option pricing, Advonces in newral information processing systems, 2000.
Joram Lindenstrauss and Lior Tzafriri. Classical Banach spaces I: Sequence Spaces. Springer Verlag, 1977.
A Function Plots
This section contains function plots for casy visualization. We start with the ReLU function r(z), then look the first differences of these ReLU function da, (2), and then the second differences (n,(2)) which correspond to the Schauder hat functions (x, as well. The last row contains the main part of the counter example 9n.* (æ), then a perturbed Schaudor hat function built from softplus functions, and finally the pyramidal lunction used in Theorem g
40
35
35
35
30-
30
20
20
20
15
15
15
10
(a) /(4x 1)
(b) r(4x 2)
(c) r(-8x +
Figure ReLU function plots
12

---

Published in Transactions on Machine Learning Research (November 2025)
06
0.
0.6
044
00
0.
0.0
8
(a) r(4x
r(4x
r(4x r(4x
4x
2)
Figure First. Differences (
06
06
04
04
04
02
02
02
0.
0.0
00
8
(a) r(xx) 2r(4x 0.5) r(4x 1)
(b) r(4x 1) 2r(4x 1.5) r(4x
(c) /(4x
2r(4x 2.5) r(4x
Figure 3: Second Differences (xx(x) = (n,((1)
0.4
04
0.
02
(a)
(b)
11x 2) r14x 2.5) r(-4x + 2.5) r4xx 3)
Pa|4x
2p-4x palAx
Pyramidal Function
Figure 4: (a) describes the counter example In.k +055, (b) describes a perturbed Schauder built using Softplus functions Pa (z) with 10, and (c) shows the pyramidal function used in Theorem g
13


Reviews:
[{"review_id": "heedFeqKs8", "reviewer": "Reviewer_nkMh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**: I'm not an expert in this area and wasn't aware of the stability result used in the second part of the paper. This result gives a clean way to extend their result for ReLU to softplus. The main novelty of the softplus results is that unlike Schauder's original construction, the constructions in Theorems 3 and 6 are smooth.\n\n**Weaknesses**: I didn't see a convincing argument in the introduction section for why to study these questions. I can understand that maybe the ReLU result is more desirable than the Schauder hat given the practical relevance of the former, but the technical contribution here is quite limited as they just use the fact that piecewise linear functions can be written as linear combinations of ReLUs. Also while the softplus results are less trivial, these activations are less standard, and from a technical standpoint the techniques seem quite standard within approximation theory.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper constructs several different so-called *Schauder basis* functions for the space of continuous functions on the unit interval. A basis is Schauder if, when writing an arbitrary such function $f$ as an infinite linear combination of basis elements, the coefficients are continuous linear functionals of $f$. Constructions of such bases are already known, e.g., Schauder's original construction consists of a collection of piecewise-linear hat-shaped bumps.\n\nThis paper first uses the fact that such piecewise-linear functions can be written as linear combinations of ReLUs to construct a Schauder basis of shifted and scaled ReLUs. It similarly shows that the Schauder's hat functions can be written as differences of step functions (given by differences of ReLUs) to exhibit another Schauder basis consisting of scaled and shifted step functions; these steps have the additional nice property of being \"sigmoidal\", that is, they tend to 0 on the left and 1 on the right.\n\nIn the second half, the paper further uses standard stability properties of Schauder bases to show that softplus functions also form a Schauder basis because they can be obtained by perturbing ReLUs. More concretely, these stability results say that if the elements of a Schauder basis are perturbed by a small enough amount pointwise, they remain a Schauder basis. Using the same idea but perturbing the step functions above instead, they construct a sigmoidal basis out of softpluses.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "- P. 2: \"minimmum\"\n- Typo in Theorem 4: $\\sum_i$ should be $\\sum_n$\n- It would be helpful to provide plots of some of the relevant activation functions", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "iTYJABfuwW", "reviewer": "Reviewer_Bxph", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "These are strengths of the paper:\n\n- The paper shows clean theoretical results.\n\nThese are weaknesses of the paper:\n\n- The proofs are not highly novel technically. The proof for ReLU functions is basically immediate, by rewriting Schauder hat functions as linear combinations of ReLUs. The more challenging part is the stability argument to obtain the result for softplus. The extensions to the sigmoidal versions are also straight-forward. \n- The existence of Schauder bases using particular activations may be interesting from a theoretical standpoint, but is very removed from any kind of empirical phenomenon, because in practice, when training neural networks, the ReLU activations are learned. Hence, I would argue that the results shown are more appealing to analysts than to machine learning researchers.\n- No algorithm is proposed to learn/approximate the coefficients. Is this possible? Can the authors comment?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors show how to construct Schauder bases for C[0,1] using ReLU functions, and softplus functions. The proof of the existence for ReLU functions builds on a prior result that provides a Schauder basis of C[0,1] in terms of piece-wise linear functions. The proof for softplus functions follows from the result for ReLUs and a stability argument, The authors also translate these two results into the constructions of Schauder bases using the sigmoidal versions of ReLU and softplus, which are the functions that yield the ReLU and softplus upon discrete differentiation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Could the authors comment on extensions to Schauder bases of C([0,1]^n), since neural networks are most useful in high dimensions. Intuitively, it looks like the ReLU argument should go through using the n-dimensional Schauder function. \n- Can the authors comment on the approximation error incurred by the truncated expansion associated to the Schauder basis, especially in comparison to results in which the ReLU functions can be chosen adaptively, as in Barron’s work [1], and follow-ups such as [2], [3], [4], [5]. If a theoretical analysis is possible, I’m expecting the bound to be much worse. This is somewhat similar to the separation results between the F1 (adaptive) and F2 (fixed features, RKHS) shown in [6].\n\n[1] A. Barron. Universal approximation bounds for superpositions of a sigmoidal function. IEEE\nTransactions on Information Theory, 39:930 – 945, 1993.\n\n[2] L. Breiman. Hinging hyperplanes for regression, classification, and function approximation. IEEE\nTransactions on Information Theory, 39(3):999–1013, 1993.\n\n[3] J. M. Klusowski and A. R. Barron. Approximation by combinations of relu and squared relu\nridge functions with ℓ1 and ℓ0 controls. IEEE Transactions on Information Theory, 64(12):\n7649–7656, 2018.\n\n[4] G. Ongie, R. Willett, D. Soudry, and N. Srebro. A function space view of bounded norm infinite\nwidth relu nets: The multivariate case. In International Conference on Learning Representations\n(ICLR 2020), 2019.\n\n[5] C. Domingo-Enrich, Y. Mroueh. Tighter sparse approximation bounds for ReLU neural networks. In International Conference on Learning Representations\n(ICLR 2022), 2021.\n\n[6] F. Bach. Breaking the curse of dimensionality with convex neural networks, Journal of Machine Learning Research, 2017.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pZ0uqwUkmn", "reviewer": "Reviewer_Ewx5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n1. This paper is very well written with concise and self-contained writing, which makes it easy to follow and enjoyable to read.\n2. The results provide a new perspective in looking at ReLU, Softplus, and Sigmoid activation functions in neural networks.\n3. The existence of Schauder basis in the form of given functions is interesting, which, potentially, could have implications in approximation theory that helps provide better explanation in the approximation power of finite dimension neural networks (lightly hinted in Section 1).\n\nWeakness:\n\nMy main concern regarding this work is in terms of existence of audience in TMLR community. Although I personally enjoyed reading the work (and as such, I selected yes to the audience question), the subject itself is more of a mathematical piece than a topic directly relates to machine learning. I do not see any direct practical implications (algorithmic) of such derivation in machine learning community, as the results only wrap up to the existence and uniqueness of the coefficient $\\alpha$. The only loose connection to machine learning that I could identify is Strength 3 I stated above. Therefore, at the current stage, it is still hard to justify the relevancy of this work in the TMLR community.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper derives four versions of Schauder basis for $C[0,1]$ functions, one using ReLU, one using Softplus and two using sigmoid functions. The ReLU basis is obtained through the decomposition of the original Schauder basis into ReLU basis, and the Softplus basis is obtained through the stability property of Schauder basis and perturbation of the ReLU basis. Following the construction of these two basis, sigmoid versions of them are constructed by representing the basis as their first discrete derivatives.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper itself is well-written and easy to follow. The only changes that I would request is a discussion or remark on the potential implications of this work in the context of machine learning, highlighting the relevancy of this work in the TMLR community.", "broader_impact_concerns": "This is a foundational theoretical work, I believe broader impact does not apply here.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)