Minimum Width for Universal Approximation using Squashable Activation Functions
Authors: Jonghyun Shin
Namjun Kim
Geonho Hwang
Sejun Park
Venue: ICML 2025 poster (2025)
Keywords: universal approximation
minimum width
Primary area: deep_learning->theory

Abstract: The exact minimum width that allows for universal approximation of unbounded-depth networks is known only for ReLU and its variants. In this work, we study the minimum width of networks using general activation functions. Specifically, we focus on squashable functions that can approximate the identity function and binary step function by alternatively composing with affine transformations. We show that for networks using a squashable activation function to universally approximate $L^p$ functions from $[0,1]^{d_x}$ to $\mathbb R^{d_y}$, the minimum width is $\max\\{d_x,d_y,2\\}$ unless $d_x=d_y=1$; the same bound holds for $d_x=d_y=1$ if the activation function is monotone. We then provide sufficient conditions for squashability and show that all non-affine analytic functions and a class of piecewise functions are squashable, i.e., our minimum width result holds for those general classes of activation functions.

Minimum Width for Universal Approximation using Squashable Activation Functions
Jonghyun Shin |
Namjun Kim 2
Geonho Hwang² Sejun Park
Abstract
The minimum size of networks that can universally approxi- mate or memorize has also been studied. For example, clas- sical results show that the minimum depth for both universal approximation and memorization is exactly two (Pinkus, 1999; Baum, 1988). The minimum number of parameters depends on the depth of networks. For universal approx- imation using RALU networks, it is known that shallow wide architectures require more parameters than deep nar- row ones (Yarotsky, 2018), where similar results are also known for memorization (Park et al., 2021a; Vardi et al., 2022). While these results show the benefits of depth, they also imply the existence of the minimum width enabling universal approximation and memorization.
The exact minimum width that allows for univer- sal approximation of unbounded-deptt networks is known only for RELU and its variants. In this work, we study the minimum width of networks using general activation functions. Specifically, we focus on squashable functions that can ap- proximate the identity function and binary step function by alternatively composing with affine transformations. We show that for networks us- ing a squashable activation function to univer- sally approximate LE functions from [o, 1jd, to Rd,, the minimum width is max(d2, dy, 2} un- less dg = dg = 1; the same bound holds for d= = dy if the activation function mono- tone. We then provide sufficient conditions for squashability and show that all non-affine analytic functions and class of piecewise functions are squashable, i.e., our minimum width result holds for those general classes of activation functions.
There have been extensive research efforts characterize such minimum width. The minimum width for memoriza- tion is constantly bounded (i.e., independent of the input dimension) since any finite set of inputs can be mapped to distinct scalar values by projecting them (Park et al., 2021a). Intriguingly, the minimum width for universal ap- proximation depends on the input dimension de and the output dimension dy. Several works have shown that the minimum width lies between dy and dg + dy + where a 2 0 is some constant depending on the activation func- tion and target functions space; however, the exact minimum width is known only for approximating L" functions when the activation function is RELU or its variants (Park et al., 2021b; Cai, 2023; Kim al., 2024).
1. Introduction
Understanding what neural networks can or cannot do is a fundamental problem in deep learning theory. The clas- sical universal theorem states that two-layer networks can approximate any continuous function if an activation func- tion is polynoooll (Cybenko, 1989; Hornik et al., 1989; Leshno et al., 1993; Pinkus, 1999). Likewise, several studies on memorization show that neural networks can fit arbitrary finite training dataset (Baum, 1988; Huang and Babri, 1998). These results guarantee the existence of networks that can perform tasks in various practical applications such as com- puter vision (He et al., 2016), natural language processing (Vaswani, 2017: Brown et al., 2020), and science (Jumper et al., 2021).
1.1. Related works
The minimum width for universal approximation has been studied for two function spaces C(X,)) ))) and (P(X,))) C(X' Y)) denotes the space of continuous functions from X' to Y endowed with the supremum norm suprex llf( and T.P (,,,,, denotes the space of I.P functions from X' to y' endowed with the L"-norm [| 2 (Jx llfllldpp.. 1/p for 2 1. Recent studies on the minimum width (say Wmin) was initiated by Lu et al. (2017). They show that d₂ + S Wiin S d, + 4 for universally approximat- ing L (Rd,,, ,R) using RELU networks. Hanin and Sellke (2017) consider universally approximating C((O, 1]d=, R-2)) using RELU networks and prove de + Wmin S d, + dy. Johnson (2019) proves the lower bound Wmin 2 d. + 1 for
1/p
'Department of Mathematics Education, Korea Univer- sity 'Department of Artificial Intelligence, Korea University e *Department of Mathematical Seiences, GIST. Correspondence to: Sejun Park <sejun. @ar.k000r gmail.com>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Minimum Width for Universal Approximation using Squashable Activation Functions
Table I: A summary of known bounds on the minimum width for universal approximation.
Reference
Function class
Activation o
Upper /lwwer bounds
Lu et al. (2017)
( R)
RELU
d2 + 1 S Wmin S dz + 4
Hanin and Sellke (2017)
1 Rdy)
RELU
de + 1 S. Wmin S de + dy
Johnson (2019)
C( [[00,
uniformly conti.
d, +1++ S Wmin
conti. nonpoly.
Wmin S dz + dy +
Kidger and Lyons (2020)
0[10,1]4,11e"
nonaffine poly.
Wmin S dz + dy +
RELU
Wmin max d2 + 1, dyf
Park et al. (2021b)
I2((0, 1]d=, RR))
conti. nonpoly.
Wmin S max + 2, dy + +1
Cai (2023)
LP((0, 1 _ R
Leaky-RELU
Wmin = max dzy dy. 2}
Kim et al. (2024)
TP((0, 1 1|== R
RELU-LIKEE
Wmin = max ( dy. 2]
Ours (Theorem 2)
L"([0, 1|==, R
Squashable
Wmin max du dy. 2}
|| requires that is uniformly approximated by sequence of one-to-one functions.
i requires that is continuously differentiable at some point 2, with o' (z) + O.
t denotes RELU, leaky-RELU, ELU, SOFTPLUS, CELU, SELU, GELU, SILU, and MISH.
& includes all analytic functions and class of piecewise functions such as leaky-RELU (see Sections 3.1 and 3.3).
dr dy 2 3 is required for non-monotone activation functions.
activation functions that can be uniformly approximated by a sequence of one-to-one functions, which has been later extended to Lipschitz continuous and monotone functions when dz 2 dy (Rochau et al., 2024). Kidger and Lyons (2020) show that for C((0, 1jd= RR,),, Wmin S dz + dy + an activation function continuous, non-polynomial, and continuously differentiable at some point with non-zero derivative. For non-affine polynomial activation functions, they also show Wnin S d, + dy + 2. However, the upper bounds in these results are at least d2 + dy, which has large gap compared to the lower bound dg + 1. Such limitation arises from their universal approximator constructions that use dy neurons to preserve the dy-dimensional input and dy + neurons to compute the dy dimennsiio output.
versally approximate LP( 100,,11,,,,,, RR4). Specifically, they
derive the bound using the two results: (1) neural ODE
can approximate any LP functions (Li et al., 2022) and (2) Leaky-RELE RELU networks can approximate any neural ODEs
using width max [dz, dyy, 2} (Duan et al., 2022).
The coding scheme proposed by Park et al. (2021b) has also been used to characterize Wmin for different problem setups. For example, Cai (2023) show Wimin max (dz, dly, 2} for networks using both RELU and FLOOR, Kim et al. (2024) show Wiin = max ( (dz. dy, 2} for variants of RELU (see the footnote t in Table 1), and Rochau et al. (2024) show Wmin = max max(d,, dy. 2) for leaky-Rel.U. Nevertheless, ex- isting proof techniques for tight minimum width highly rely the property of RELU and/or its variants, and the mini- mum width for general activation functions was unknown.
The exact minimum width was first characterized by Park et al. (2021b). They show Wmin = max[d, + 1,, dy for RELU networks to universally approximate L R and for networks using both RELU and the binary step func- tion STEP(>)) 1 to universally approximate C( 1]d= Rd).. For (1O 1jd- R24), they also show Wmin I max(d,, (d. 2, + 1 for a class of continuous non-polynomial activa- tion functions. Specifically, Park et al. (2021b) construct two networks called the encoder and decoder to approxi- mate a target function '" de + Rdy; the encoder maps a dy-dimensional input vector I to some scalar value c con- taining the information of x and the decoder maps c to a neighborhood of the dy-dimensional target vector (z). Then, the concatenation of the encoder and decoder approx- imates f using autoencoder structure with an internal width one. Cai (2023) use different approach to show Wmin max(de, dy. 2} for leaky-RELU networks to uni- (TTEP(<) = 1 if 2 0 and STEP(#) otherwise.
1.2. Summary of contributions
In this work, we study the minimum width enabling univer- sal approximation of [P([ 1]d=, ,Rá) using general activa- tion functions. Specifically, we consider activation functions a such that an alternative composition of a and aftine trans- formations can approximate the identity function and binary step function; we call such functions squashable (see Defini- tion 1). Using the squashability of an activation function o, we show that the minimum width of networks to univer- sally approximate LP( 1]d,, Rd))) ) is max/d2. dy) unless d dyy = 1 (Theorem 2). We also show Wnin = 2 when d dy = 1 if the squashable function o is monotone.
Our result can be used to characterize the minimum width for a general class of practical activation functions, by show- ing their squashability. For example, we show that any non-affine analytic function (c.g., non-affine polynomial,
2

---

Minimum Width for Universal Approximation using Squashable Activation Functions
SIGMOID, lanh, sin, exp, etc.) is squashable (Lemma 4). Furthermore, we also show that wide class of piece- wise continuously differentiable functions including leaky- RELU and HARDSWISH are also squashable (Lemma 5). Hence, our result significantly extends the prior exact mini- mum width results for RELU and its variants.
neither empty nor degenerate (e.g. [a, a] = {aa).
2.1. Fully-connected networks
Throughout this paper, we consider fully-connected neural networks. Formally, given a set of activation functions Σ. we define an L-layer neural network f with imput dimension do dz. output dimension dp dy. and hidden layer dimensions d1,- ,..... as
Even if an activation is not analytic or piecewise continu- ously differentiable, can be squashable, i.e., our minimum width result can be applicable. To check the squashability of general functions, we also provide sufficient condition for the squashability: a is squashable if and only if there exists an alternative composition ff of a and affine transformations such that f is strietly increasing and has a locally sigmoidal shape on some proper interval (Lemma 3).
f tz o ØL-1 o t o o1 tto
where te : Rd + RR is an affine transformation and (zl......., Ld,) = (0 ,Ldd, (Id,)) for some ,f,,,...,, E E for all & E [Z... We denote neural network using single activation function (i.e., Σ = { by a "a network" and a neural network using two acti- vation functions 01, 02 (i.c., E = {or, o2)) by a "(01, 02) network". Here, the width w of f is defined as the maximum over the hidden dimensions ".....,.---
We use the coding scheme (Park et al., 2021b) to prove our result. In particular, we construct our decoder face curve that densely fills the codomain of a target function so that supze/ f ([0,1]4e) ((0.1) || - oo is small. We then construct our encoder to map each x: [0, 1jd= neighborhood of (2) for some R (z). Although we use the coding scheme, our constructions largely differ from existing ones (Park et al.. 2021b; Cai, 2023; Kim et al., 2024; Rochau et al., 2024), which highly rely on properties (e.g.. shape and discontinuity) of specific activation functions (e.g.. RELU, FLOOR, STEP, and their variants). Namely, existing proof's do not easily extend to general activation functions. See Section 4 for our detailed constructions.
We say "a networks of width w are dense in LP 3/"" if for any f E II'(X, ))) and ε > 0, there exists a network of width w such that Ilf - fllL> S E. Given an activation function a and da, dy E N, we use Wo.dg.d, to denote the minimum w E N satisfying the following: a networks of width w are dense in [' 1]=,, RR)) but a networks of are not dense. We often drop d2, dy and use Wa if d dy are clear from the context.
width
3. Main results
1.3. Organization
3.1. Squashable activation functions
We first introduce notations and the problem setup in Sec- tion 2. We then formally define the squashability of ac- tivation functions, describe our main result minimum width for universal approximation, and provide sufficient conditions for the squashability in Section 3. We prove our main result Section and conclude the paper in Section 5. Proofs of technical lemmas are deferred to Appendix.
To formally state our main result, we first introduce a class of activation functions a that we mainly focus on. Condition 1. There exists E R such that a is continuously differentiable at z and o' d'(z) 4 0.
Condition 2. a is continuous and for any compact set IC C R and for any E, > 0, there exists a a neteworrk Pe,ç IR - R of width such that
2. Problem setup and notations
max,ext((-G.)) (Pe,(()) STEP(*) S E, Pe.ç is strictly increasing on K., and Pe.c(AC) C [0, 1].
In this section, we introduce notations and our problem setup. For n E N, we use [n|] to denote (1). For S.T C Rd, we use dSam(S)= _ sup,,yes || ماl and dist dist T) e wee ||x ylloo. I is singleton set, (i.e., S = {*}), we use dist (s, T) to denote dist ({*), T). For yorys E Id and S c Rd, (2) _ {a E Rd. dist (z, 8) < r} and (S) é {x E Rd dist (x, 5) < rr. For a function f: Rdd + Rd fz); denotes the i-th coordinate of fz). For n E N. we use fd to denote the n times composition of f. We use e R + R to denote the identity function (((x) x) and STEP to denote the binary step function (STEp(#) = if < 0 and STEP(œ) = 1 otherwise). We note that all intervals in this paper are proper: they are
Condition is that an activation function a has a contin- uously differentiable point with nonzero derivative. This property enables us to approximate the identity function on compact domain by composing with affine transforma- tions as stated in the following lemma.
Lemma 1 (Lemma 4.1 in (Kidger and Lyons, 2020)). For any & > 0, a R -> TR. satisfying Condition I, and compact set KC C I. there exist affine transformations hi, h2 K. +
R such that
sup 11/ 100000 h1) - -x]| S.
zE
3

---

Minimum Width for Universal Approximation using Squashable Activation Functions
p(x)
(())
(())
a
C
a
C
(a)
(b)
(c)
Figure 1: Illustration of construction of squashable function using a network of width 1 that has a sigmoidal shape when
<(x) = x. The intersections of p(x) and ((x)) serve as fixed points. Thus, a can achieve the squashability by iteratively composing p: (z) + for x E (a, c) and (x) + for æ E (c, b) as n co while p" is strictly monotone.
Condition 2 assumes the continuity of a and the existence of network of width 1 (i.e., an alternative composition of affine transformations and σ) that binary threshold function (i.e., STEP) on any compact set, except for small neighborhood of zero (i.e., (-S, ()). One important property in Condition strictly increasing on K. This allows Pr,ç to preserve the information of inputs in K since is bijective K.
Wo max(d2.d) dy} unless the input/output dimen- sions are both Furthermore, it fully characterizes Wa,d, for all dæ. dy if activation function is squashable and monotone. The proof of Theorem 2 is in Section 4.
approximate the
To the best of our knowledge, the exact minimum width enabling universal approximation has been discovered only for few RELU-LIKE activation functions such as RELU, leaky-RELU, SOFTPLUS, GELU (Park et al., 2021b; Cai, 2023; Kim al., 2024). Furthermore, the best known upper bound for general class of activation func- tions was I max max(d, + 2, dy + +1) when o is continuous non-polynomial and continuously differentiable at some point with non-zero derivative (Park et al., 2021b). Our re- sult extends prior exact minimum width results to general class of activation functions (i.e., squashable) including all analytic functions (e.g.. SIGMOID, tanh, sin, exp, polyno- mial) and a class of piecewise continuously differentiable functions (e.g., HARDSWISH). See Lemmas 4 and 5 in Section 3.3 for more details on squashable functions.
that Pe,c should be
Using these conditions, an activation function.
define the squashability of
Definition 1. A function IR - I is "squashable" if satisfies Conditions and 2.
One can observe that width-1 networks using squashable activation function can approximate the identity function on any compact domain and the STEP function on any compact domain except for small open neighborhood.
A class of squashable activation functions covers wide range of practical functions. Condition can be easily sat- isfied: e.g., any piecewise differentiable function with non-constant piece satisfies Condition 1. Furthermore, we prove that any analytic activation function (e.g., SIGMOID, exp, sin.) and a class of piecewise continuously differen- tiable functions (e.g., leaky-RELU and HARDSWISII) sat- isfy Condition 2. We formally state these results and easily verifiable conditions for Condition in Section 3.3.
3.3. Easily verifiable conditions for Condition
In Theorem 2, we have observed that Wa.da.dy can be char- acterized if a is squashable. However, checking whether a given activation function squashable, especially whether satisfies Condition 2, can non-trivial. this section, we provide easily verifiable conditions for Condition based the following lemma.
3.2. Minimum width with squashable functions
Lemma 3. A continuous function R R satisfies Condition 2 if there exist network of width and a, b E with a < satisfying the following:
We are now ready to introduce our main theorem on the minimum width for universal approximation.
is strictly increasing on [a, b) and there exists c E (a, such that
Theorem 2. Let be squashable function. Then, We
max (dz, ) if dz 2 2 or dy 2 2 and wa (1,2) if d, = d, = 1. Furthermore, if a is monotone, then Wa 2
p(z) < 6(x) Vx E (a,c), p(x) 6(x) Vr (c,b) where ((xx) is a line passing (a, p(a)) and (b, p(b)).
if d. dy 1
Theorem characterizes the exact minimum width enabling universal approximation for squashable activation functions:
Lemma 3 provides a sufficient condition for Condition 2:

---

Minimum Width for Universal Approximation using Squashable Activation Functions
can make a network of width 1 that has "sigmoidal shape" some compact domain (e.g., see Figure la), then satisfies Condition 2. We can easily approximate the STEP function using a function with the sigmoidal shape by composing the function and some affine transformations (see Figures lb and Ic). For a more formal argument, see the proof of Lemma 3 in Appendix A.2.I.
Lemma 7 (Lemmas 21 and 22 in (Kim et al., 2024)). For any : R Ih and dz, dy E N, wo 2 max max(dz; dy)- Furthermore, if a is monotone, then wa 2 2.
Lemma 6 implies that for any squashable activation func- tion o, We S max dy, 2}. This is because (1) con- tinuous functions on [0, 1] are dense in L"((O, 1]d=, R3) (Rudin, 1987) and (2) g (0, 1] is compact for all g C(10, 17 ,,d,,,,", Bd). ). ie.. [0, 116, Lemma 7 provides the lower bound of the mini- mum width. Here, the lower bound Wmin 2 max [dz, dy 2} naturally follows since the first/last operations in networks are affine transformations. Hence, combining Lemmas and results in Theorem 2. In the rest of this section, we prove Lemma 6.
Such a sigmoidal shape (or its symmetric variants) ex- ists in various smooth activation functions such as GELU, SIGMOID, lanh, and sin. In addition, for any non-affine analytic function 0, we can always make a network of width that has the sigmoidal shape. Since all non-constant analytic functions are continuously differentiable and have a non-zero derivative at some point, all non-affine analytic functions satisfy Conditions I and 2, i.e., they are squash- able. The proof of Lemma 4 is presented in Appendix A.2.2. Lemma 4. All non-affine functionn from R to IR satisfy Condition 2.
scale the range of g to be in
4.1. Proof of Lemma
To illustrate our main idea for proving Lemma 6, we first define a ò-filling curve.
In addition, a class of piecewise functions also satisfies the condition in Lemma 3. We defer the proof of Lemma 5 to Appendix A.2.3.
Definition 2. Let d E N and 8 > O. We say continuous function f: IR + lt is a "6-filling curve" of D c Rd if
sup dist (v. [(0, 1J)) s 6.
Lemma 5. A continuous function R + R satisfies Condition 2 if there exist e R and 8 > 0 such that
yED
A 8-filling curve of D C R2 can be considered as a weaker version of space-filling manee-filing curve of 'D (Sagan, 2012). While the range of the space-filling curve contains D but the 8- filling curve covers D within ò distance.
a is continuously differentiable on (c 5, e 5) (c), x+ -im d1))) and v lim o' (x) exist, x+ f U,, and oto=
Suppose that we can implement a 5 ō-filling curve h of [o, 1]dv using a a network for some small 8 >> 0, i.e., for each y E [0, 1] there is 2 E [0, 1] such that h(=) R y. Hence, if we can design a network g that maps each x E [0, 1]d= to some 2g such that h(2z) = (zx), then the network h g approximates f Here, 9 and h can be considered as an encoder and decoder: 9 encodes a dy-dimensional vector & to a scalar value z.. that contains the information of (xx) and h decodes 2g to dy -dmmennsioa vector h(zz) that approximates fx))
Lemma 5 states that if an activation function a contains a point such that the left limit of the derivative and the right limit of derivative at that point are different but have the same sign, then a satisfies Condition 2. We note that piecewise functions such as leaky-RELU and HARDSWISHI satisfy the condition in Lemma 5; for those functions, one can choose the point e in Lemma 5 as some break point between consecutive pieces.
While we provide easily verifiable sufficient conditions (Lemmas 4 and 5) for Condition 2, we note that Theorem 2 covers any activation function satisfying Conditions 1 and 2, even if that activation function does not satisfy conditions in Lemmas 4 and 5. We also present additional sufficient conditions for Condition 2 in Appendix A.3.
We explicitly construct networks that approximate the en- coder and decoder. To this end, we introduce the following lemma where the proof is deferred to Appendix C.
Lemma 8. Let a be a squashable function, d, w E N, and K. C IRd be a compuct set. Then, for any e >> and (a, i) network f of width w. there exists a a network g of width w such that
4. Proof of Theorem
We now present the proof of Theorem 2. Theorem 2 is direct corollary of the following lemmas.
sup [1((z)=(()]l>))) <s. aEk
Lemma 6. Let be squashable function, E >> 0, f E C(O0, 1]d, [0, 1]dy), and p 2 1. Then, there exists a o network f [0, d]d= - R.d. of width max(dy, dy, 27 such that
Here, R - R denotes the identity function (see Sec- tion 2). Lemma 8 implies that constructing a (a,c) network of width max(d,, dy, 2} that approximates f is sufficient to prove Lemma 6. Hence, we focus on approximating the encoder and decoder using (a, t) networks.
|| 1 fl S..
5

---

Minimum Width for Universal Approximation using Squashable Activation Functions
Senc (T(11)
Senc (T1,2))
T(122)
T22
T(111)
T22.1)
fene (T2,1))
(T2,)))
Figure 2: Illustration of fdec fenc when d₂ 2, dy = 2, and N = 2. fene first encodes each Ty to bounded interval [ene(Ty). Then, Siee implements ö-filling curve of [0, 1J2. represented by the black curve, to decode each sone (Tw) (colored) that approximates f (Tx) (represented by the light gray area).
We first show that the decoder can be implemented using (a, c) network of width dy. The proof of Lemma 9 is in Section 4.2.
(,) + as - O, one can observe that for any e there exist small enough 7, 6 and large enough N [|
such that I o 1fm S e. Namely, (a,4) network f= f fene has width max (dæ, dy 2} and completes the proof. Given E > 0, our explicit choices of 8, y, N and the detailed derivation of [] o Senc - 1*11 I e can be found in Appendix B.
Lemma 9. Let a be squashable funatiaioliiiiiiin function and 6 > O. Then, there exists a (a, 1) network Sdee= : 10 - [0, 1]d, of width dy that is a 8-filling curve of [0, 1]dy.
Lemma 9 states that for any 5 > O, we can always imple- ment 5- d-filling curve of [0, 1 using (a, t) network fdee of width dy- Further, the implemented network satisfies fdec (I0, II) C [o, 1 regardless of ò.
4.2. Proof of Lemma
In this section, we prove Lemma 9 by showing the following: for each N, d E N, there exists a (a, c) network of width dy that is a (1/N)-fllling curve of [0, 1]d. particular, we inductively construct a (1/N)-filing curve of [o, 1] from d 1. Here, the base case = is trivial: (a,1) network (xx (()) is (1/NN)-illing curve of [0, 1] for all N e N.
We also show that the encoder can be approximated by 1) network of width max 2}. The proof of Lemma 10 is in Section 4.3.
Lemma 10. Let be a squashable function, N E N and Y (0, 0.5). For each v E [N]da let To I 124 14-1+t " and Cy [0, 1]. Then, there exists (0,1) network fene : 10, + [0, 1] of width 22 such that for each V E [N]d=.
We prove the general case (d 2 2) using the inductive step descrihed in the following lemma, whose formal proof is in Appendix D. Here, for N, d N, we use CN.d,v & I
1--1 N N. and v (M1,-.., E [N]d.
fene (Tv) C (c,))).
Lemma 11. Let N, e N and be squashable function.
Suppose that there exist disjoint open intervals I, C [0. 1] for all E [N] and a (a, v) network f: [0, [0, 1]d of width d such that for each æ E [0, 1] and v [N]
Here, (cx) denotes the -bal of radius centered at Cp (see Section 2). The collection T, in Lemma 10 can be regarded as an approximate partition of [o, 1] its elements are disjoint and it covers almost all parts of the domain with small enough y > 0. By choosing a large enough N, the diameter of f (Tx) can be arbitrarily small, i.... f*(x) x (z²) for all æ, x' E Tu. Under this setup, choose co for each v so that (cx) ) Re (Tv). Then, fene in Lemma 10 maps each element Tz in the approximate partition to some small ball centered at co, with diameter 7. Since face is continuous, this implies that for each e Tu. faec feme (x) (x) with small enough 5 for faee and small enough 7. large enough N for Senc. See Figure 2 for the illustration. Here, we note that Sãee Sene is (a.() network of width max(da. dy: 2}.
f(x) =x and f(Iv) C CNN.d,..
Then, there exist disjoint open intervals Jo C [0, for all i E [N]a+1 and a (a, i) network f [0, 1] + 10, 1 of width d + 1 such that for each [0, 1] and & E [N]d+]]
f(x) = and j(Jb) CN.441.
One can observe that (a, networks f and f in Lemma 11 are (1/N)-filling curves of [0, 1ja and [0, 1ju41, respec- tively. Furthermore, our filling curve construction f(x) (x) for the base case satisfies the assumption in Lemma 11 with I, = (²=1 x) for all e N and E [N]. Hence, by Lemma 11, we can conclude that for each N, d e N,
For x of U, Tv. we have face o fa)( E [0, 1]4 (i.c., bounded) by Lemmas 9 and 10. Since pa, (0, 1jdz
6

---

Minimum Width for Universal Approximation using Squashable Activation Functions
M/3
y3
1
I
(b)
92
3/2
71
22
I3
I1
12
I3
1/1
9/1
21
22
(a)
(c)
Figure 3: (a) Illustration of brackets (left), to be intersected with the corresponding cube of the same color (right). (b) and (c) illustrates our network satisfying the properties of á when N = and N 3, respectively.
curve ff of [0, 1J8. f maps each open interval I,, represented by the colored
there exists (a, a) network that a (1/N)-filling curve of [0, 1]s; this proves Lemma
Such a construction also extends to an arbitrary number of Ip by composing (i.e., an approximation STEP). For example, let Σ,, I2, I3 C [0, 1] be disjoint open intervals and let 2; E I,. Then, have
We now briefly illustrate main idea for constructing f in Lemma 11 given f. Suppose that disjoint open intervals I, for all E [N] and corresponding (a, 1) network of width d in Lemma 11 are given. Then, to prove Lemma 11, suffices to construct a (a, network f of width d++ such that for each i [d] and v E [N]].
(bx)) STEP(# 21 (21 23) x STEP(2 22))
0 if x 5 or 22 S <
otherwise
f(x): = f(z), and 2N 1
2 f(I,)a+I
Namely, by replacing STEP by in with small enough E, > 0 (and denoting that function by v6), we have IN 1- 2V C (I+) by the intermediate value theorem (see Figure 3c). We present a more detailed argument for general N, in the proof of Lemma in Appendix D.1.
This implics that if we can construct a (a, network o: [0, R2 of width 2 such that for each / E [N]]
d(x); = x and I
I
C (Io)2,
4.3. Proof of Lemma
then construct ff in Lemma 11 by choosing
We now prove Lemma 10. Our construction of fene consists of two (a, a) networks: fi [0, 1]d= - IR of width d, and f2 R + IR of width Here, fi maps each T, disjoint compact interval fi (Tv) and is designed satisfy f2(fi(T)) (c,) for each Namely, fenc fz fi satisfies f(T.) C B,(cc)
f(x); (f(())) f(x)a+1 ((f(x)))2 and = -o.... ((((r)) for all i E (2.....d).
See Figure 3a for the illustration. We can construct such & using the squashability of a. For example, suppose that N 1 and d = 1 (i.e., there exactly one I,). By Definition for any 5, > and compact there is width-1 network p such that
Construction of f2. The following lemma shows the ex- istence of fo such that fe(fi(T.) c (,,c,) for each EE [N]
IR with [-6,s] K.
Lemma 12. Let K. T be compact interval and II...., IN C K. be disjoint closed subintervals. Then, for any E > 0, squashable a,, and ci...., CN E R, there exists a (a, 1) network f K. > [0. 1] of width such that for each
max (EXX(-C)))
(p(x)) STEP(=)) S.
Then, by the intermediate value theorem, we have
k: E [N].
[.. E] c((-cccC)
sup 1f(x) cxl Se. zET,
This implies that by choosing p(x)) p(a ) for some Zy E I, and K containing I, with small enough e, S > 0, holds that [2] 22 C (II,) (see Figure 3b). In this case, we can choose width-2 (a, 1) network satisfying Eq. (1) Φ(x)1 and d(x)2 p(x).
We prove Lemma 12 by explicitly constructing a (a, t) net- work that approximates a piecewise constant function which maps each interval Iş to Ck- The formal proof of Lemma 12 is in Appendix E.

---

Minimum Width for Universal Approximation using Squashable Activation Functions
9
2/2
G2
2/2
(a)
(b)
2:14
62
C3
22
(c)
Figure 4: (a) Illustration of a function g IR3 - ?R2 that maps sets in a 3-grid 93 of size (2,2, to distinct sets in 2-grid G2 of size (2, 4). (b) Illustration of th : R + R2 Here, the first coordinate of de (x)) is approximately 1 or 0 depending on whether 21 exceeds с or not while the second coordinate is z2. (c) Illustration of our construction of f when G is a 2-grid of size (3. 2) and e2; C3 > 0 are chosen so that all sets in 9 are disjoint in the second coordinate.
Construction of /1. In the remainder of this section, we construct (a,6) 4) network fi of width dz that maps each T, to disjoint compact interval fi (Tu). Here, we assume d, 2 2; if de 1, we choose (z) = (xx). To describe our construction we define a d-grid.
Then, g can be constructed
2i g(x); = (2)a-1)
if S - if = d -1
Φ (x))
if d
Definition 3. A collection of sets g C 2 is a "d-grid" of size (m1, na) E Nd there exist disjoint compact intervals I........ Ii C IR for each i E [d] such that
where @ is (a. network of width 2 in Lemma 13 that maps the 2-grid G' of size (ma-1, na) to some 1-grid of size 1 see Figure 5a for the illustration of when 3.
g = [I, x ... x Ir.ja : E [n:], Vi E [d]}.
Let Ga, = (To V E [N]d+] } be a dz-grid of size (N,...,N), As in the construction of g. we recursively construct 9i for i dr, d,, - 1,.... .,, 2 as (a, t) network of width that maps an i-grid G. of size (N ...., N, Nd.-it) to some (i - 1)-grid G- of size (N, NN,...,NN4:: NNN4.. We then construct Si as fi = 92 o ..... Ga.- One can observe that fa has width de and maps sets in Ga. to distinct sets in some 1-grid.
One can observe that any finite of disjoint intervals is 1-grid and To is de dz-grid. We construct fi using the following lemma. The proof of Lemma 13 is in Appendix F. Lemma 13. Let a be squashable function and G be a 2-grid of size (n1, n2). Then, there exist a (a.1) network
f: K. Th. of width 2 such that {f(S) S g} is 1-grid of size 727
Intuition behind Lemma 13. We now briefly describe our main proof idea for Lemma 13 where the formal proof is deferred to Appendix F. Our construction of f is based on the squashability of a. Observe that by the definition of the squashability (Definition 1), for any compact set K. C TR, there exists a width-1 network that is strictly increasing and approximates STEP on K. (see Condition 2).
Lemma 13 implies that there exists a (a, i) network f of width that maps sets in a 2-grid sets an 1-grid. This implies that for any distinct sets S, S' in the 2-grid, J(S) n f(S") = 0. We now construct fi by using (a, networks that reduce dimensions one by one while preserving the disjointness of each Ty.
We first show that for any d 2 2 and d-grid 9 of size (mi,..., na)), we can construct a (a, 1) network g width that maps sets in the grid a (d 1)-grid of size (ni,...., n na-1na)- Specifically, such ga can be con- structed by using Lemma 13. Let G' be 2-grid defined by considering the last two coordinates of sets in G. i.e.,
Consider a width-2 network the RR2 + H² defined as (x) (p(=1-c),a0 for some R. Then, by choosing proper and K., / splits sets in G into two parts depending on whether their first coordinate exceeds or not. Here, (S), will be close to one the first coordinate of S exceeds c and (S)1 will be close to zero otherwise. We note that by the strict monotonicity of p. the order of the
g' = (((r)-1, ra): : (#1,..., ra) E S) E g).
8

---

Minimum Width for Universal Approximation using Squashable Activation Functions
9
G2
2/2
(a)
(b)
2:11
62
Ca
22
(c)
Figure 5: (a) Illustration of a function g IR3 - IR2 that maps sets in a 3-grid 93 of size (2,2, to distinct sets in 2-grid G2 of size (2, 4). (b) Illustration of th : I + R2 Here, the first coordinate of de (x) is approximately 1 or 0 depending on
whether 2 exceeds с or not while the second coordinate is 22. (c) Illustration of our construction of f when G is a 2-grid of size (3, 2) and e2; C3 > 0 are chosen so that all sets in 9 are disjoint in the second coordinate.
first coordinate of the sets does not change. See Figure 5b for the illustration.
width holds for d₂ - dy 1 if the squashable activation function is monotone. Since all non-affine analytic functions and a class of piecewise functions are squashable, covers almost all practical activation functions. We believe that our approach would contribute to a better understanding of the expressive power deep and narrow networks.
result
Furthermore, we can also change the second coordinate while splitting the first coordinate. For any > 0, by composing the with some invertible affine transformation Ke : 122 + R2, we can construct a width-2 network 6c,e Ne ahe Ke so that
Acknowledgements
if Φ 1
This work supported by Institute of Information & communications Technology Planning & Evaluation (IIPP) grant funded by the Korea government MSIT (RS-2019- II1 I1190079, Artificial Intelligence Graduate School Program, Korea University), the IITP-ITRC (Information Technology Research Center) grant funded by MSIT (ITPP-205-RG 2024-00436857), and the Culture, Sports, and Tourism R&D Program through the Korea Creative Content Agency grant funded by the Ministry of Culture, Sports and Tourism in 2024 IRS-2024-0034502 and 25% by RS NSS-2024-00348-440 the KIAS Individual Grant by the Center for AI and Natural Sciences at Korea Institute for Advanced Study (AP092801), and National Research Foundation of Korea grant funded by the MSIT (RS-2025-005 1524 and RS-2024-00406127 and Global University Project grant funded by GIST in 2025.
(0) BJ
if 21 ~ 0 and œ1 < if Φ and 1 > c.
(1,32 +)
Using such the and 6c,e, we construct f by sequentially separating sets in & based on their first coordinate. First, we apply some invertible affine transformation so that the first coordinate of all sets in G is close to zero (as in the left of Figure 5c). We then split the sets of the largest first coordinate using die with some proper choice of After that, we sequentially split sets in Figure 5c. Lastly, we apply projection onto the second coordinate. For more formal argument, see Appendix F.
5. Conclusion
In this work, we characterize the minimum width enabling universal approximation of L L"((0, 1]d= 1144). In particular, consider general class of activation functions, called squashable, whose alternative composition with affine trans- formations can approximate both the identity function and STEP on compact domains. We show that for networks us- ing a squashable activation function, the minimum width is max (dzz, dy, 2} unless dz = dy 1; the same minimum
Impact Statement
This paper investigates the theoretical properties of neu- ral networks on the minimum width enabling universal ap- proximation. We could not find notable potential societal consequences of our work.

---

Minimum Width for Universal Approximation using Squashable Activation Functions
References
Namjun Kim, Chanho Min, and Sejun Park. Minimum width for universal approximation using ReLU networks compact domain. In International Conference Learning Representations (ICLR), 2024.
Eric B Baum. On the capabilities of multilayer perceptrons. Journal of complexity, 1933:193-200 1988.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Annual Conference on Neural Information Processing Systems (NeurIPS), 2020.
Moshe Leshno, Vladimir Ya Lin, Allan Pinkus, and Shi- mon Schocken. Multilayer feedforward networks with nonpolynomial activation function can approximate any function. Neural networks, 6(6):661-1)))))) 867, 1993.
Qianxiao Li, Ting Lin, and Zuowei Shen. Deep learning via dynamical systems: An approximation perspective. Journal of the European Mathematical Society, 25(5): 1671-1709, 2022.
Yongqiang Cai. Achieve the minimum width of neural networks for universal approximation. In International Conference on Learning Representations (ICLR), 2023.
Zhou Lu, Hongming Pu, Feicheng Wang. Zhiqiang Hu, and Liwei Wang. The expressive power of neural networks: A view from the width. In Annual Conference Neural Information Processing Systems (NeurIPS), 2017.
George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems, 2(4):303-33 -14, 1989.
Yifei Duan, Li Li'ang Li, Guanghua Ji, and Yongqiang Cai. Vanilla feedforward neural networks as a discretization of dynamic systems. arXiv preprint arXiv:2209.10909 2022.
Sejun Park, Jaeho Lee, Chulhee Yun, and Jinwoo Shin. Provable memorization via deep neural networks using sub-linear parameters. In Conference on Learning Theory (COLT), 2021a.
Boris Hanin and Mark Sellke. Approximating continuous functions by ReLU nets of minimal width. arXiv preprint arXiv:1710.11278, 2017.
Sejun Park, Chulhee Yun, Jaeho Lee, and Jinwoo Shin. Min- imum width for universal approximation. In International Conference on Learning Representations (ICLR), 2021b.
Allan Pinkus. Approximation theory of the mlp model in neural networks. Acta numerica, 8:143-195, 1999.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Dennis Rochau, Robin Chan, and Hanno Gottschalk. New advances in universal approximation with neu- ral networks minimal width. arXiv preprint arXiv:2411.077.0 2024.
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approxi- mators. Neural Networks, 2(5):359-336) 366, 1989.
Walter Rudin. Real and Complex Analysis. McGraw-Hill, Inc., 1987.
Guang- Bin Huang and Haroon A Babri. Upper bounds the number of hidden neurons in feedforward networks with arbitrary bounded nonlinear activation functions. IEEE transactions on neural networks, 911:2244222, 1998.
Hans Sagan. Space-filling curves. Springer Science & Business Media, 2012.
Gal Vardi, Gilad Yehudai, and Ohad Shamir. On the optimal memorization power of relu neural networks. In Interna- tional Conference on Learning Representations (ICLR), 2022.
Jesse Johnson.
Deep, skinny neural networks are not uni- versal approximators. In International Conference on Learning Representations (ICLR), 2019.
A Vaswani. Attention is all you need. In Annual Conference on Neural Information Processing Systems (NeurIPS), 2017.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figumov, Olaf Ronneberger, Kathryn Tunyasu- vunakool, Russ Bates, Augustin Židek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-58) 2021.
Dmitry Yarotsky. Optimal approximation of continuous functions by very deep ReLU networks. In Conference on Learning Theory (COLT), 2018.
Patrick Kidger and Terry Lyons. Universal approximation with deep narrow networks. In Conference on Learning Theory ( (COLT), 2020.
10

---

Minimum Width for Universal Approximation using Squashable Activation Functions
A. On activation functions
A.1. Definition activation functions
exp:
exp(x)) 6".
SIGMOID:
1
SIGMOID(2) + +xeep(-1
tanh:
tanh(x) exp(x) - exp(-x)
cxp(x) + exp( -x)
Leaky-RRLU: for a E (0
x x
Leak)-RELUU(T;0)) oz
ELU: for a >> 0
x i > if S 0.
O) = [((xp(()) -
SELU: for À and 0,
SELU(I;A,a) = {x (c(exp(x)-1) if 5 0.
if > 0
GELU:
GBLU(x) x ((x)
where is the cumulative distribution function for the standard normal distribution.
CELU: for a > 0
fx CELU(x; = (alxpp(\/a) - 1) if $ 0.
if x >
SOFTPLUS: for >> O.
SOFTPLUS(Z;a)- log(1 + exp(ax)).
SWISH:
SWISH(2) = SIGMOID(x).
MISH:
MISH(x) tanb((oFrPLus((: 1)).
HARDSWISH:
if < -3
HARDSWISH(2)
i 3 x(x + 3)/6
otherwise.

---

Minimum Width for Universal Approximation using Squashable Activation Functions
A.2. Proofs related to squashable activation functions
In this section, we prove Lemmas 3-5 by constructing o network of width satisfying the conditions listed in Condition where a has the property in each lemma.
A. PROOF OF LEMMA 3
In this section, we prove Lemma 3. We first prove that if a satisfies the conditions listed in Lemma 3, then a is squashable by explicitly constructing network of width satisfying the Condition 2 using the activation a that satisfies the conditions listed Lemma 3. Namely, we now show that for any E. Ç > 0 and compact set K. there exists a a network f : IR - R of
width such that [(x) - STEP(x)] < e for all a E K \ (-C, c). To this end, without loss of generality, we assume that 0, ((x)= = x and K [-M, M] for some M >> 0 and [-M, .M] [].
Then, have p((a, bl) [a. b]. For any N, define thn R - R by
cx(2) (xx).
Then, tin (la bl) C (a, bj and on is strictly increasing on [a, bj. Furthermore, for any n E N, (n( < (ati(()) for x E (0, b) and ain (x) > (n+1(T) for I E (a, 0). We now show that there exists N E N such that i 2 N,
b- 6- b- a)s < (C) < b. Then, since Vo is strictly increasing, (xx) E (a, a + (b- - a)s) for any [-M, -C] and (xx) E (b- - (b -a)e, b) for any
G. ( < a (b a)s,
(2)
x E [S. Then, define a a network f R - R of width 1 by
1 f(x) b (Ux(z)- (xx)))-
Then, Ml) C f(aa,b) c [0 and f strictly increasing, and 0 f(x) S f(-() E for [-M, -[] and 1- < & f(z) 1 for M]. It implies that f is squashable and this completes the proof.
We now show the existence of N E N such that vnn satisfies Eq. (2) if n 2 N. Let 4/n (C). Then, On < an+1 < b for all n E N. Then, by the monotone convergence theorem, there exists L E R such that a < L S b and lim,-+o0 an = L. Here, if L < b, then
lim an+1 = lim p(an)= p(L) > L 2-000
2=000
which is a contradiction. Hence, L = and this guarantees the existence of Ni E such that a 2 N1, then b- (b- a)æ < (C) b. Likewise, there exists N2 E N such that if 2 N2, then a < vn(-() < a + (b a)e. we choose N > max N1, N2), then our a network f of width satisfies Condition 2.
A.2.2. PROOF OF LEMMA 4
In this section, we prove Lemma 4. To this end, it suffices to show the existence of the a network p: :R - R of width 1 such that
p is strictly increasing [0, 1]. p(0) 0 and p(1) 1, and
(0) < 1 and p' (1) < 1.
Then, from the second and third line in the above conditions, one can observe that p(z) < x if x E (0, and p(x) x if (1 - 6, 1) for some > 0. Then, by the intermediate value theorem, the equation p(z) x has at least one solution in (0.1). Here, since p is analytic, there are finitely many solutions cI, , Ch E (0, 1) such that e₁ <... < Ca and p(G) =
for i E [k]. If k 1, then p satisfies the conditions of Lemma 3 with [0, 1] and Φ(x) = x. Otherwise, / satisfies the conditions of Lemma 3 with [0, c2] and Φ(x) = x. It completes the proof.
We now construct such a a network p by considering the following cases: (1) there exists a E R such that of d((a) = 0 and (2) a' (x) # for all x E TR.
12

---

Minimum Width for Universal Approximation using Squashable Activation Functions
We considered the case (1) in Lemma 15 in Appendix A.3. We consider the case (2): a d'(x) * for all x
R. Without loss of generality, a' (a:) > 0 for all x: R. To this end, consider the following cases again: (2-1) there exists e IR such that (= 0 in (c 8, c) and o"(xr) < 0 in (c,c+ 6) for some å > 0 and (2-2) otherwise.
We considered the case (2-1) in Lemma 14 in Appendix A.3. We now consider the case (2-2), Specifically, it suffices to consider the case that there exists a E IR such that (a) > O and (x) 2 0 for x: > a. Otherwise, suppose that o" (x) S 0
for z E R. Then, we can makes a to convex function by taking an affine transformation: oo (2) = (-
Without loss of generality, assume that a 0 and o (0) Then, we define a network + R such that
(x)) x o (bb) (bx))
for b > O. We will assign an explicit value of b later. Then, we have (5() = 0, (1) = 1, and v is strictly increasing on [0, 1]. Then, we construct a a network p. IR + R of width by
p(z) = - +01 - (x)).
Then, p(0) = 0, p(1) = 1, and is strictly increasing on [0, 1]. Furthermore, one can observe that
12 (b)
p0(0)
a(b)
We now show the existence of b such that
8 bao(())o(0))
a(b)2
To this end, consider a function g: (0, oc) - IR defined by
1 o0)
g(x)
x o(x)
Then, one can observe that
x22 xxx)2 )
x20 (z)'
g g((x)
Since x > 0, it suffices to show the existence of b > O such that g'(b) < 0. Since o o²(0) > 0 and o(x) > 0 for all x > 0, it can be easily shown that (xx > o' (0)z for all x > 0. It implies that g(x) > 0 for x > 0. Furthermore, since (xx - 00 as
x - oo, it holds that g(x) - 0 as x + co. Then, there exists M > 1 such that g(1) > g(M) since g(x) + 0 as x - x0
and g(1) > O. Then, by the mean value theorem, there exists b E (1, M) such that
9(M) g(1) gg'(b) < 0. M 1
completes the proof.
A.2.3. PROOF OF LEMMA 5
In this section, prove Lemma 5. To this end, first consider the case that the given activation is piecewise linear function. Without loss of generality, we assume that
or [-1,0) x
o1 (z)
(3)
x E [0,2]
where 0 < a < 1. We now construct a a network p of width as
ar
x E (-1,0)
p(x) = - (1
=
I E [0,1)
a. +
a
x E [1,2].
13

---

Minimum Width for Universal Approximation using Squashable Activation Functions
Since 0 < is easy to observe that di satisfies Condition 2 by Lemma 3.
We now consider the general case. Suppose that It + IR satisfies the conditions listed in Lemma 5. We show this by constructing a network & of width 1 that approximates (ar) in Eq. (3) with = o' (c__ //= within an arbitrary error for any I E [-, Then, we can easily verify that Lemma can be applied to the same construction of a network of width 1 as above, 1 - (b(1 - ()))), and this completes the proof.
now show the existence of such v.. To this end, without loss of generality, we assume that e 0, ( O. d'(c) d'()), and is strictly increasing on (c- 8, 6). For > construct a network 1o of width 1 as
(x) (rar)
By the mean value theorem, for 11 O, there exists d E (rx, 0) such that v/ (ar) xo xd,(d) and for 0 < x I 2, there exists e, rz) such that (x)) zo (e,)) Since o' (xx) is continuous on (c - ô, + 6), it holds that a d'(,) o' occc) and (e,) + d'(4 as + 0, respectively. It implies that
lim (x) = [o x E (-1,0) m00
d'(c)) [0.
Thus, choosing (xx) = ()((x)/o((4)) with sufficiently small r > 0 completes the proof.
A.3. Additional properties for functions to satisfy Condition 2
In this section, we suggest the additional properties for activation functions to satisfy Condition 2. Lemma 14 implies that an activation a satisfies Condition 2 there exists a point where the sign of o" converts from positive to negative.
Lemma 14. Let c E R and > 0. Suppose that a function a R + I such that a is twice differentiable in (c - 0,c 8), (x) in (c - 5,c) and (x) < 0 in (c.c + 5). Then, a satisfies Condition
Proof. To prove Lemma 14, we now choose appropriate a, E R and Φ: R + and apply Lemma 3 with our a, b, and ø We consider a line passing (c, a()) as ạb Since (=(x) of < e and (=x) < 0f
c, we can choose a slope
so that á and p meet once (c 5, e) and (c,c+ 8), respectively. Let a = max(a(c) a(c - 6/2), ae + 6/2) - (c))
and (xx) 3/2 (x c) + (c). Here, one can easily observe that 8f2 cc)).. Without loss of generality, suppose that == a - alc- 8/2). Then, it holds that
((c + 6/2) = (()) - olc 5/2) + o(c) 2
- a(c) = 5
Then, by the intermediate value theorem, there exists b E (c, + 5/2] such that <(b) = ob)) Furthermore, since - 6(c-8/2)
= (c - 8/2), choosing a = c - 5/2 and applying Lemma with our a, b, and completes the proof.
Lemmas 15 and 16 imply that satisfies a condition stronger than the analytic condition in a compact interval, then o satisfies Condition 2.
Lemma 15. Consider a1, 02 E such that (x) nonaffine analytic on x [a1, a2). Suppose that there exists E [ar, 42] such that o' dc(c) O. Then, a satisfies Condition 2.
Proof. It suffices to show the existence of the a network p R of width such that p is strictly increasing on [o. 1]. p(0) 0, p(1) 1, p (0)
and p1(1) 1 (see Appendix A.2.2). Since a is nonaffine analytic function that has zero derivative at some point, b (c, ag] such that is strictly monotone on [c, b] with nonlinearity. Without loss of generality, assume that c = 0, (0) 0 and (1r) is strictly increasing on [0. b). Then, we define a network v IR + R such that
1 o(b)
((x)
(hr)).
Then, (0) 0, v1) 1, and v is strictly increasing on [0, 1]. We now construct a network p by
p(x) ((1 (xx)).
14

---

Minimum Width for Universal Approximation using Squashable Activation Functions
Then, p(0) 0, p(1) 1, and p is strictly increasing [0, 1j. Furthermore, one observe that
(x: v (1 (x))
Then, we have (0( (1) 0 since (0) = 0. It completes the proof.
Lemma 16. Consider 01, 02 IR such that o (z) is analytic
[01,02]. Assume that there exists [a1, 02] such that
20)(2)
2 ("(x) + x.
Then, a satisfies Condition 2.
Proof. In this proof, (rr) (x) is defined as n-times derivative: (x)) = d'ao(a) dxn
We only need to consider the case
o' (xx)) 0 and of ((2 0; see the case (1) and (2-1) in Appendix A.2.2.
Consider an arbitrary 20 E (01,a2) For b E (01 To. a2 zo). define wi : (0 - e, +c) + R as
(xz) o(b+ + xo) o (20))
(a(bx + 20) o(z))).
Then, (60) = (1) 1. Define p
p(x) = 10(1 (x)))
Then,
)0()== p'(1) ='())=' 82'0 xo) (o(b + xo) - o (xn))²
It is sufficient to find value b such that p0(0) p((1) 1. Define g as
g(x) = 1 o d'(xn)
x
olx mo) (za)
Then, as
1
(f((10) + xa)
1
2²0'(x 20 (Ir) x o (zo))² 1)
d'(x)
x²
+
(To(s sa) (xx)))
x2
(a
it is sufficient to find number such that g((x) < 0. Then, there exist smooth functions h, h1, he such that
o' d'(z))
d'(zo)
g(x)
I a(z + xo) - (20)
0² 0'(20 (2)) x² (xo) 0/2 + xz((2)
x + ((0) x²
+ (())
xd +
a'(o)) x4
Ma)
('(0 2
(zo)
(20)
M(x) x2
x (zo) 1 (8) (m))
+ 1o2 (x)
2n (x0)
(x))
o( (mp) 3
+ (
( x2 +
h(r))
x3
20 20'(2o)
1 +
0 (mg) x
+ h (x))x²
(20)
(20)
(2o) 2
Then, (x) Of
(2) (xo)
σ (33) (3)()
(zo)
(4)
20'(x0)
20(2) (x.u)
Assume that the above inequality is not satisfied for any 2o (ar,
(0), a2); that is, for any (a),a))
(2) (x)
o (3)
20 (x)
30 (x))
15

---

Minimum Width for Universal Approximation using Squashable Activation Functions
Then, for any 01 < 31 < y 02
dx 2 l (x) y (2)
d'(v) o'(x))
o (12
dx
2 log
2 log
20((x)
my (x)
0 (x1)
(2)
d'
d'(y)
which leads
012) (
xn) 52 d'(x1)
o' (x
d'(2)3
for any a1 21 < 02. Thus,
2 d'))..
o(0)(a)) (z - x1)
o'(,))
201
We lastly present Lemma which implies that strictly monotone has limit, then satisfies Condition
Lemma 17. A continuous function o : IR R satisfies Condition 2 if a has strictly monotonicity and there exists sim (x) or lim,---o (x)).
Proof. Without loss of generality, we assume that (a) is strictly increasing and lim a(x) = 0. consider the two cases: (1) lim (x)) a oo, and (2) lim (xx) oc.
For the first case, easily verify that satisfies Condition 2 by composing affine functions before and after
p(x) = o (Mx)
where M is sufficiently large. We now consider the second case. Suppose that lim (xx) oc. We construct network & of width such that
((x) o(1) x (a(1) - (()))
Then, it is easy to observe that v is strictly increasing, lim (bxx) = and lim 1x(x) = 0. Then, we can consider á as in the first case. Hence, a is squashable and this completes the proof.
16

---

Minimum Width for Universal Approximation using Squashable Activation Functions
B. Our choice of 6, y, N
We first choose sufficiently small 8 > 0 so that S E/(dy" 31/r)) And then, choose small enough y 0 so that I <p/ (3d and Saee (7) Σ e/3l+1/2, Lastly, we choose large enough N E N satisfying diam(f"(T.) (y-((11 100--,,, NN) I =/(d) /p x 31/p for each v E [N]] Here, Wae and w denote the modulus of continuity given function f in the p-norm: [x) - f('')lpp I wy[[| - x' (lp) for all x, E [0, 1]4s. Then,
f 10,jur I (a) f (x)llddua
dee Senc - 1
||eee o Senc - (r)) WWsenne To I| o SSene()) 8 (r))
[o.1j/>\Uee(rr To
1 IIfdoo o souc(x) - (x))llpdu
- dy x pdd
[0.1]4r
U To
E
METANM
DEIN]A
S dy x (1 101 21)))))
+
Σ
(llfdec o fenc(x) fdee(co)lpp (f))ll)
To
wE(N)d
S 2d + Σ f
(We (7) + d./r (diam(f (Tv) + 8)) dud.
T,
VE(N
y + (wi (7) + d//o (diam(f" + 8))² 0
where cy is chosen so that dist (faeo(cu): (Tx)) S for each v E [N]d.. This leads us to the statement Lemma
17

---

Minimum Width for Universal Approximation using Squashable Activation Functions
C. Proof of Lemma 8
In this section, we prove Lemma 8. Since f K + Rdy a (a, network of width w, we can express f - R2 as follows:
f = 1 e 1 0 ... 01
where tg: Rd- - de is an affine transformation, and (x(x) = (x).) Pe,d, (x)) for Pe.1. ... Pe,dd E {a,1) for all E [L]. Since a satisfies Condition 1, by Lemma I, for arbitrary compact set and for any 5 > 0, there exist affine transformations h1 : > IR and h2 : + R such that
| o u(x) <
for all x E C; we will assign explicit value to 8 later. We denote hi a o h22 as ō. We note that this lemma can be applied for any given compact set. Since we are considering a compact domain and a continuous activation function, the error arising from replacing L with ā can be reduced.
To this end, we choose network y by applying same affine transformation 4,, br and ē:
g = tu L o t- 0/1 t
where (>x) = (x), ... Pe,d, (1r)) with مe, = if pe,: and مr,, if pe.: = for e E [L] and E [de]. We denote Sa and ge by the first layers of f and g with the subsequent affine transformation respectively. i.e..
ft de-1 t--1 01 to and ge = tt o w o .... 0 tt
Then, for each E [Z] \ {1} and for any x E K. it holds that
[[f(()) - |rx ||te Φ1-1 f Le o th 1 (x) O1l (x) t o g (2))(
(1 f(-1(x) de-1 -e-1 1 9 -e-1 1x-1(=))
Here, we note that any E [L], Wi, well-defined since Le uniformly continuous on Rdd-1, Then, by the definition of d and o, it holds that
| o (z) - 1 S max [(g)-1(z).) - ((ge-1()))) <
<E(de-1
Furthermore, since we are considering the compact domain and Φ continuous, War-, well-defined and
|| o (c-1( - dr- o 1 S w (1]Se-1(x)) (x)
Hence, we have
[[f(()) ge(x)ll== = (1 f(-1(x) de- 1 + |l o 9)-1((x) - - o 1
I we, (W (11fe-1 1 8)
(5)
a [4]\ [1). By iteratively applying Eq. (5), we have
[[f()) - 20 S we, (wor_ (11f(-1(z) - 9L)=1())1= +
& why (War (... (wi, (Wer (wrs (8) + 8) + 8). 5).
Consequently, by choosing sufficiently small > 0, we can reduce this within arbitrary error > 0 and this completes the proof.
18

---

Minimum Width for Universal Approximation using Squashable Activation Functions
D. Proof of Lemma 11
In this section, we prove Lemma 11. show Lemma 11, we construct (a, network f width + 1 as follows: for each i [d] and v E [N]]
f(x); - f(x); and
1
C f(Iz)a+1.
(6)
2N
2N
Then, since f is continuous, for each v E [N] and j E [M] there exists J(ms) C In such that
i(Jus))at j - 1
Furthermore, since J(rr) Iy for each (V1,... ,ua) [V]] and [N], it can be easily observed that
f(J(u))) C
1/3 1
N N
for all E [d]. It implies that j(Tvr) C CN.d41.r and this completes the proof.
We now construct a (0, network of width d + satisfying Eq. (6). To this end, we first present the following lemma. Lemma 18. Let a E S and -2. 2k E [0,1] such that 24 + 2j for all i + j. Let >> such that 7 minyy 12 - 2/2 Then, there exists a (o, network f : [0, 1] - IR? of width 2 satisfying the following:
10 f(z)1 11 C (z)) all [k].
22. 1 2N
1(0, 1)) C 00
One observe that Lemma 18 allows prove Lemma 11 directly. We choose 7 > 0 and 2, Iy for each such that (zv) C Iv. Applying Lemma 18 with our choices of zv 's and y, we construct (a, network Φ [0, 1] IR² of width 2 satisfying the conditions listed in Lemma 18. Then, we complete the proof by constructing f in Eq. (6) as follows:
f(x) = o(f())) f(z)441 = =0f(()) and f(z); ......... ........l(())) for all i (2....,00 .d}.
D.1. Proof Lemma 18
Without loss of generality, we assume h = 2m for some m E N and = 20 < 21 < 22 22m < 22m+11 = 1;
otherwise, we can add an auxiliary žk+1 IR such that 24 2k41 < Let X {21, 22, 22m) }. Dx, = [0,
Fm (z: 7, + 7), and Ax = U (221-1,
To construct f in Lemma 18 using (a, 1) network, we use the Condition that for any compact set C, a can approximate STEP except for the neighborhood breakpoint. We first construct (STEP, t) network h [0,1] {0, } of width 2 such that
M(x) = -fo i E Ax otherwise
and then we construct (a, c) network f : [0, 1] R² of width 2 such that f(x) and [f(x)2 h((2) 1/2N except for the neighborhood of each E X. Since is continuous function, can observe that such f satisfies the conditions listed in Lemma 18.
We first construct h in Eq. (7) as follows: h = hm+11 where Am+1 (x) is recursively defined as
h (x: STEP(x Zm+1)
hr(x) STEP(x Zm-6+2 (Zm-642 - Zm+e) (
(8)
19

---

Minimum Width for Universal Approximation using Squashable Activation Functions
From Eq. (8),
STEP(I STEP(I-2m-E41) 2m 2m-2+2 STEP(I- 2m+t)
De-1(2) = 0 h (x) =
ha(a))
for any & E {2,-.., + 1}. One can observe that he forms additional breakpoints 2m-242 and Am+e, and for any I E [2s 2411 where E {m - L + 2, m + 2). the values of hz(x) alternates with and 1 as iinereases. Hence, he(z)
Eq. (8) can be rewritten by
{1
E [2m-2+26, 2m 2m-2-2241), Ak [e 1 1] or x 2 2m+t
he(x)
otherwise
for any E .2. + 1}, which implies that hm+1 is equal to h in Eq. (7).
We now construct (a, o) network f of width 2 based on h. It suffices to show that for any e > 0 and & E [m + 1] there exists a (a, i) network fe [0, 1] + I22 of width such that
C1. (x))
on [0, 1].
C2. [f((r)2 - he ha)) < E for x E Dx
C3. fe(1o, 1)) c [0, 1]?
Then, choosing f fm+1 with < 1/(2N) completes the proof: Cl and C3 directly imply the first and third conditions of Lemma 18, respectively, and C2 guarantees that fm+1 satisfies the second condition of Lemma 18 from the definition of Dx,, and h. We prove this via mathematical induction on L. We first consider the base case, = 1. Since a satisfies Condition 2, there exists a network p of width 1 such that
Ip(x) STEP(x)) <e
for all æ E (0,1]\(-7<>) and p(10, 11) c [0, 1]. Then, we construct a () network f(); ;0, + R of width 2 as
f fx) =x fi(2)2 = p(a - 2m-1).
Then, one can easily observe that (()) satisfies C1-3. We now consider the general case, l 2 2. From the induction
hypothesis, for any > there exists () network ft-1 [o, + R22 of width 2 such that fe-1(x) = x, ( (x)1 and fe- ((o, 1J) [0, 1J2. Since satisfies Condition 2, for any compact C, there exists a network p of width such that
Ip(x) STEP(x)) < <e/2
for all C (-y,)). We now construct fe [0, 1] + IR? as
f((z)) Se-1(x) fe(x)) = p( - 2m-8+2 + (Zm-842 - 2m+r)/e-1(x))
Here, by the induction hypothesis, fe = æ. Thus, we can simplify this to
f((x); xx fe(x)2 = plx 2m-242 + (Zm Emm-et2 Zm -m- (x)22
which is just the substitution of STEP Eq. (8) by p. Here, one can observe that fo satisfies Cl. Then, for any x E Dx
fe(x)2 he(x))
<lp(2- 2m-e+2 + (2m-442 2m+e) STEP(x 2m-8+2 + (2m-e42 (m+e)he-1(z))
<lp(2 - 2m-4+2 + (2m-e+2 2m++)..f p(æ 2m-e+2 + (2m-8+2 2m+t)he-1(2))
+ lp(x= - 2m-842 + (Zm-442 Zm h - STEP - 5m- + (Zm-
(9)
Here, we note that the second term of Eq. (9) is bounded by E/2 since
2m 2+2 + (#m-842 2+2 Zm + ( (x) & (-7.7)
20

---

Minimum Width for Universal Approximation using Squashable Activation Functions
for all Dx,zi since (x) 1, then 2m-642 + (2m-E-2 zm+e)hee -mm of x =m- Hence, we have
(up up(l(zmee22 ( + 8/2 < up(l(2m-t2 =m-e) + + <E [e(())) - -
-
by choosing sufficiently small 5 > 0. implies that fe follows C2. Lastly, we can easily observe that fe(z)1 x E [0,1]
and fe()))2 [0, since p(x:) E [0, 1] for all C. It implies that satisfies C3 and this completes the proof.
E. Proof of Lemma 12
In this section, we prove Lemma 12. To this end, without loss of generality, assume that K C [0, co) and c1, E (0, 1); there exists G; such that c; = or G = 1, then we can substitute of = E/2 or e/2 respectively and approximate them within error e/2. Let g dist [(1, ... ck); {0, 1)). Then, one can observe that { >> 0. addition, we assume that for any and y E I.+1. Since I are disjoint, for any E [N - 1], there exists z(1) E R such that sup I x ()) < inf I,+1. Let (0) min KC, (N) max K. and
[AV 1], x y for all
dist (z.(),, .I)) dist (z00), [+t) iE[N-1]
7
min
(10)
In this proof, we construct (a,1) network f K + [0, 1] of width 2 such that for any &t E [NV]
sup 1f(z) cs nn rEIn
where 7 = ei.
To this end, we construct two (a, networks h1 : K. + R and h2 + I of width 2 such that
C1. for each h E [N], suPzeI, lh (x)) cal S n/2
C2. for any x E U In. 1h2 o hz()) h1 (x) s n and h2 o h1C) C [0, 1].
Then, one can observe that hn maps input to near the corresponding e if E Ig. and h2 bounds the codomain of h1 while the approximation for piecewise constant is preserved. If we choose f = h2 h1, then such f satisfies the desired conditions.
We first construct h2 satisfying using the property of that can approximate STEP. To this end, we consider (STEP, network K - of width 2 approximating the given piecewise constant function, and then we construct a (a, network h of width 2 approximating g in U Iz.
We now construct (STEP, 1) network g approximating piecewise constant function. To construct such g, we compose
(STEP, i) networks 91," " .NN R IR. of width 2 such that each di shifts by sufficiently large length L, > 0f a
()) Here, for each i [N]. L, is defined as x (c++b) b) where a > max(1, (rrN))/)) /n) and b = xNN)_ - minne[N] G
which implies that each (ar) = x + L, > (N) for x E (2((-11, .()). i.e., we construct each yi such that
(c1 + b) (,)))
+
2 + a x (02 + b)
E (2)))
(x)
(11)
x + x (a +) x (x(i-1), ,(1)
otherwise
for all i e [N]. Then, we define g as follows: g = Jeut a IN o 1NN1 O ... o 91 where Jeur : R + ER is defined as
1
Doue (1)
-x b.
a
21

---

Minimum Width for Universal Approximation using Squashable Activation Functions
Then, one can easily observe that
C1 + &
x E [r:00, ,x(1))
x E [x(1) xl())
02 +
g(x)=
CN + 2 a E (1( x(N)
Since a > 4x(N)/n /v,, it holds that |z//|| < 1/4 for all x E K. Thus, y approximates the piecewise constant function within an error 1/4.
We now construct (STEP,) 1) networks 91. .. IN 99 satisfying Eq. (11). For each E [NV] we define 9: : R R as
(x(x)
(a: b)STEP(-((1) x!")).
One can observe that ge shifts x by a x (a + 6) i < (1)) Here, we note that since a x (c: + +b > x(N), the values shifted by 9i for some E [N] are not shifted again, resulting that 9 shifts only x E (x(i-1), ,(()). Thus, our g can approximate given piecewise function within an error 1//2.
We now construct (a, c) network f2 of width 2 approximating 9 USEN Iz Since is squashable, then for any compact set C' and a > there exists a network p R R such that p is increasing on C, p(C) C [0, 1]. and
(Ø(x) STEP(x)) <a
for all x E C) (-3,3) B) where 0 min(7,8) (Eq. (10)). We will give explicit value to later. We now construct (o, c) network by of width 2 as follows:
h1 = Jeut o fN o ..... o fi where
f((z) = r + (c: + b)p(p)ppppppp (x x 20))) Vi [N].
(TEP(-(e--(0))) < l for x E C (x)) Then, one can observe that (x) = lez + bllp((((x (x- r()))
and E [NV] For the notational simplicity, we denote d; = |c, + bla. We note that for any i,j E [N] and E I,
9j oggggggg gi & (z(x
(12)
Eq. (12) holds since gj o...o 91 maps I to x E I, or value out of K. (r + + x (a. +b x + x()) from the definition of a and b) and | < y. Then, for any i E [N] and x E I, it holds that
I/1 (x) - g(x)l = Igeu o SN ....o o fxx Jeut o 9N o og
S Wgnue (IfN
fi(x) 9N o .... o 91 (z)
I Waue (I
fi(r) fN IN-N1 0 ................ (I) fN o 1N-1 (z IN (z))) fi(z) 9N-1 9N-10....g((3]] (z))) + IN 9N-1 o... (z) gN 91 (z)))
I Waen (W (1fN
Here, IfN 9NN-1 ... 00 (x) 9N ... )r(x)) < &N from Eq. (12). Thus, by conducting this procedure iteratively, we have
(1)(x) - g(+)]
< Wirm (WSN (JN- fi(z) - -x-1 o.... og(()))) + +Nn]
< p (W (W (SN-20..... f(()) 9N-20 gl (x)) + 6N- + dn)1
< W (W]x (w (61) + ..)...) ... + أx)l | n/4
by choosing sufficiently small a > which leads us to have sufficiently small 8, for all E [N]. Consequently, for any i [N] and I, we have
(2) cil s (x) -())))g(>))+++ -(x))++ al < 1 n/2.
(13)
22

---

Minimum Width for Universal Approximation using Squashable Activation Functions
Hence, h satisfies CI.
We now construct l22 satisfying C2. We suppose that there exists E Æ such that h1 (u) O; we will discuss the case that there exists E K. such that h (v) > 1 later. To this end, we consider (o, i) network of width 2 that iteratively adds some constant to the region such that (x) < O. Namely, it suffices to show that for any s' > 0, there exists (a, 4) network v : + R of width 2 such that
if 2 n/4, then (x(x) - 2| S e'
if n/4). then (xx) E [0, 1). and
x s 0, then (x)) - 2 1/2.
Then, let h2 N for some N₁ E N such that N12 |innex hi (z)), then we obtain
h 111 (2) = N h1 (x) E [0,
for all x e KC such that h2 (x)
Furthermore, since 17 S { and h2 satisfies C1, h1(x)= 2 1/2 for any x E U I, Thus, if we choose sufficiently small > 0 such that e' < <n/(4N,,, then
1/2 o (x) h1(x)l = 11b o h1 (1(x)) - h1(2)l
I lab o h1 h1(2)) N o (1(1))+++.... p h1(x) - h1(2))
I N2E' s 11/4 $ n/2.
Here, for each i E [ 1]. w0* o hz) 2 n/4 since wp o h1 (ar) C (br((1) and hu (x) 2 n/2. It guarantees that
jei hz - v' - o hı (z) S for each E [N1]. Hence, h2 satisfies C2. there exists v E XC such that hn (v) > 1, then the same argument can be applied with the choice of th1 (2) = 10 z).
We now construct such v using the property that a network can approximate STEP, Since is squashable, for any 8' and compact set D, there exists network pt D + R such that
let (x) STEP(x)) < 8
for all x E (-n/,,,,,))) We choose 8' such that 8 min( 1/4). Consider a (a,s) network ei of width defined as
2 vxx) = x (-
Then, can easily observe that -xl of 2/ +xx) E (0,n/4+2/03 c [0, if E n0,n/4), and
(+(x)) (x + 2/3)| < 28'/3 1/6 if æ < 0 which implies (xx) - x 2 1/2. It completes the proof.
23

---

Minimum Width for Universal Approximation using Squashable Activation Functions
F. Proof of Lemma 13
In this section, we prove Lemma 13. To this end, we construct a (a, i) network f of width that maps for each S E & to a disjoint interval. Then, since f continuous, {f(S): : S E 9} a 1-grid of size n, and this completes the proof. Before we illustrate our proof, we define the additional notation used in this proof. Since & is 2-grid of size (n1, n2), there exist compact intervals (a1, bil,-.. |an, b [d]; 81,, ... ,[dng Vog] satisfying the following:
b; < a;+1 and b/ < a's+1 for each i E [m - 1] and j E [ng - 1]. respectively,
for any S E 9, there uniquely exist E [mx] and j E [mz] such that S (a,, bi] x lag. 1/}.
For each i E [m1] and j E [m2], let Usg = [as, b,] x [aj, bg1 V; = U,
1 = min {a'z+ - b/}}
jE(m-1]
and T, = b'm a1- We write e
(1) and 62 10 RR2. For (1,2) and b E R, we use H(e,, - far R² b + b 0}. We first consider a (ar,) network h₁ KC + R2a of width 2 defined as
hn hx(r)) - p( ci), hn (x)2 1(x22)
where e1 (In-1,an) and p is a network of width such that (p(x) STEP(x)) < <Con on bul. We will assign an explicit value to Ç. Then, one can observe that
h1 (V.) c Bg(H((,, 1)), (v.). C Be(H(er,,)) 0)) for all i [n 72-11 1].
(14)
Furthermore, since he (z)1 is strictly increasing on IC, the ordering of V with respect to the first coordinate is preserved: i < j. then 21 < 3/1 for all x E Vr y E Vy. We then iteratively apply some (a,6) networks h2. , so that for each i E [ni], h4 maps V to (H((ee,, -1)) and shifts Vni-i+1 by sufficiently large length such that the images of V; are disjoint for the second coordinate.
We now formally construct such (a,4) networks h2, ... Any. See the following lemma where the proof is deferred to Appendix F.1.
Lemma 19. Let { 0 and r > O. Let Xo C Be (H((e, 0)), X, c (e(H(e),,1)) and y C (HH(e, 0)) be compact sets in R² such that >> I1 for all E No and E V. Then, there exists a (a, 1) network f IR2 + R² of width 2 satisfying the following properties:
for any No U X1 [f(z)( x22 2rE,
for any EV [f())2 (12 + +r)l < 2rg,
f(Xo) C (H(ee, 0)) and f()), ((X)) C Be(H((e,, -1)), there exists strictly increasing $ IR + I such that f(x) = ((x1) for all a E No.
Lemma 19 implics that there exists (a, t) network of width 2 that maps y' to in H(e,, -1) with approximately shift for the second coordinate by From Eq. (14), we can apply Lemma 19 with
X = U (V.), X: = :Von), y-h1(Vn (2,
iE (E(m
r= L= 1 and { = Ç. Then, there exists a (a, t) network hy of width 2 that maps the points of Xi and X1 approximately identically while shifting the second coordinate of y by + 1. Here, one can observe that we choose a sufficiently small <> then (2g (Vr,)) )) and he -1) are disjoint for the second coordinate by our choice of r. Furthermore, from the third and fourth lines of the properties listed in Lemma 19, Lemma 19 can be applied iteratively with the recursive choice of No, X, )/, r and & in Lemma 19. In particular, by the fourth line of the properties from Lemma 19, the ordering of V/'s with respect to the first coordinate is preserved while Lemma 19 is applied. Thus, among the sets contained in Xg. we can choose ) as the set that is the highest with respect to the first coordinate.
We now construct such (a, 1) networks h2. .......hu .hn RR + R² as follows: for each & E [nx] \ {1), hy is from Lemma 19 with the choices of
24

---

Minimum Width for Universal Approximation using Squashable Activation Functions
X Uy o............................... o1
11 = Ule[k-1] /4-11111111 (i(Viiitt))
y h o ................................ h1 (Vm
r = Th where ry (k- 1) + 1) and 5 =
Then, construct (0 network f K IR of width
f(z) po hn, .... o (2)
(15)
where p: P: - R is projection onto the second coordinate: p(x, y. We now prove that if we choose sufficiently
small <> such that
f1
1 1 22' 2
2 min
then for each E [n1] and j E [m2]. f(4y disjoint.
We first show that if i.j E [ni] such that i < j, then fx)) f(y) for all V; and y E Vj. and then we prove that for each i [n1], j.j E [m2] such that <J,, then f(x) < fv) for all z E Uij and y Us.
We first consider E V; and Vj. From our definition of f (Eq. (15)) and Lemma 19, one can observe that
[f(z) - (x2 + Tm 7
<
2r S 2 If(w) - (y2 + rni-5t1)l <
2n S 2
k=2
Since Fni-it1 - -n1- j+1 2 + 1, the above equation implies that
f(x) fly) Fmi-it1 Tm Tm1-#+1 - (y2 - - 11 - - 1= 0
We now consider x E Usy and y Uiye. As in above, we have
If(x) (x2 + Fmi ++1 2r 2'
ffv) (y2 mn
Σ 2r S
k=2
Since 1/2 22 > 1) by the definition of 17, we have
f(y) fxx) > 1/2 22 17 0
and this completes the proof.
F.1. Proof of Lemma 19
In this section, we prove Lemma 19. Let b IR such that 01 < < yn for all x (21, E Mn and y (v1, v2, y and
1 min(y: b,b mnly E V Xo).
We note that such b is well-defined and n because æn y1 for all Xo. y y and Xo, y compact. Since is squashable, for any compact set K,, there exists network p R + R of width I such that
lp(x) STEP(>)) <
for all x E K KC (-n,1)) Let 0]
[ 0 r 1
Then, one can easily observe that A-1
-r 1
We now define functions fi, fa, E22 + 12a2
f1(x) Ax, f2(x) (p(x) - (), (x22), fa(x) --
25

---

Minimum Width for Universal Approximation using Squashable Activation Functions
for allall: (11,"". Fa) E R", respectively. We define function f R² IR
f(z) (fs o f2 (z)
for all x E RR. Then, f is a (a, 1) network of width and
f(x) - p( b), f(x)2 - 22 + r(plai - b) x1).
(16)
We now show that our f satisfies the properties listed in Lemma 19. One can easily observe that f satisfies the fourth property of Lemma 19. Thus, we consider the first-third properties. From Eq. (16), we can classify the image regions corresponding to each input region.
We first consider z E Ng. Since Xo (e(H(ee,,)) 0]) and b 77. have æ1 E (-E,E) and p b) (0, Thus, it holds that f(x)1 E ( -1)) and [(x]² F2l < 2rf. We now consider E A1. Since Xi C ( -1)) and Ti > b 7,, we have æ E (1 - E,1 and p(rı - b) E (1-E, 1). Thus, f(z) E Bg (H((e,,, 0)) and [f(x) 22 < 2r Lastly, let y E y. Since P1 C (g 0)) and y1 > b+ 1). we have y1 C (-f.E) and p(3 b) E (1 - E.1). Thus, f(y) C (e(H(e), and [f()) - (82 + +r] < 2r.. Conclusively, f satisfies all properties listed in Lemma 19 and this completes the proof.
26


Reviews:
[{"review_id": "kUyQzn2lOD", "reviewer": "Reviewer_tEim", "summary": "The paper extends previously established result that two neurons per layer are sufficient for universal function approximation in an unbounded depth neural network; the extension establishes the known result for a wider family of activation functions, the so called \"squashable\" functions, which includes sigmoid, exponential and other functions.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for their positive evaluation and valuable comments. We address the reviewer’s concern and clarify our contribution below.\n\n> The result itself is an incremental contribution, extending known results that already covered most popular activation functions, to wider set of activation functions. \n\nWhile prior works derived tight minimum width for ReLU and its variant, we believe that studying general activation functions is not incremental since (1) prior results do not cover popular non-ReLU-like activation functions such as sigmoid, tanh, and sin, and (2) minimum width results for general activation functions can be applied to any new activation functions that can be used in future as long as they satisfy our conditions. We believe that our result can be used to better understand the fundamental limit of narrow networks.\n\nIn addition, we would like to note that our result is not a simple extension of prior works that only consider ReLU and its variants (Park et al., 2021b; Cai, 2023; Kim et al., 2024). For example, the proof of the Leaky-ReLU result by Cai (2023) heavily relies on the property of Leaky-ReLU: it contains at least one break point and is strictly monotone near that point. Namely, their proof technique cannot be directly extended to general activation functions such as nonaffine analytic functions. Likewise, Park et al. (2021b) and Kim et al. (2024) rely on the properties of ReLU (and its variants): it maps the negative inputs to (approximately) zero while the positive input is preserved by an (approximate) identity map. However, general activation functions do not have such properties, and how to extend their constructions to general activation functions was unclear.\n\nTo show tight bounds for general activation functions, we propose a novel condition on activation functions (Condition 2) that is satisfied by all nonaffine analytic functions and a class of piecewise functions, and explicitly construct a universal approximator of the minimum width based on that condition. We will add this discussion to the final draft.\n\nWe would be happy to clarify any concerns or answer any questions that may come up during the discussion period.", "claims_and_evidence": "The paper is very dense mathematically.  Having not gone through the proofs in detail I can't tell, but at a high-level glance it looks like a rigorous mathematical treatment.", "methods_and_evaluation_criteria": "The paper is theoretical, no empirical results.", "theoretical_claims": "No, I did not check the proofs in detail.", "experimental_designs_or_analyses": "The paper is theoretical, no experiments.", "relation_to_broader_scientific_literature": "Assuming the math (which I didn't verify) is correct, it's a nice theoretical result to extend the proof of the width minimum to a greater set of activation functions. Practically it's probably of little use, since (as far as I know) these super-thin networks are terrible generalises (in general).  Also, given that the existing proof already covers a wide variety of ReLU functions, the extension is not that surprising - sure if relu-like function can do with 2-neurons per layer, it seems reasonable that sigmoid would too.  I am unable to appreciate the mathematical aspects, so it is possible that the technique of the proofs has something interesting to offer - as in, it might be of more interest how this has been proven, rather than what has been proven.", "essential_references_not_discussed": "I don't have any additional references to offer.", "other_strengths_and_weaknesses": "I can't comment on the originality of the proofs provided.  The result itself is an incremental contribution, extending a know results that already covered most popular activation function, to wider set of activation functions.  As far as a very mathematical treatment goes, the work is well presented and the paper well written.", "other_comments_or_suggestions": "No other comments or suggestions.", "questions_for_authors": "No questions for authors.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RFKpCyKBz3", "reviewer": "Reviewer_cndk", "summary": "This paper considers a class of activation functions and provides the minimum width required for the corresponding neural network to have the universal approximation property.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for their time and effort to provide valuable comments. We address all comments of the reviewer below.\n\n> While previous papers have not exhaustively covered all activation functions, it is straightforward to deduce results for many activation functions from existing findings. \n\nWe do not think that tight minimum width results for general activation functions (e.g., nonaffine analytic) can be straightforwardly deduced from existing results that only consider ReLU and its variants (Park et al., 2021b; Cai, 2023; Kim et al., 2024). For example, Cai (2023) showed the minimum width for Leaky-ReLU networks using the following two results: (1) Neural ODEs can approximate any $L^p$ functions [1], (2) Leaky-ReLU networks can approximate any neural ODEs [2]. Here, (2) requires an activation to either have a strictly monotone breakpoint or have two asymptotic lines with different nonzero slopes. Namely, their proof technique does not directly extend to general activation functions such as nonaffine analytic functions. \n\nIn addition, Park et al. (2021b) and Kim et al. (2024) proved the minimum width for networks using ReLU or ReLU-like activation functions using the properties of ReLU (and its variants): it maps the negative inputs to (approximately) zero while the positive input is preserved by an (approximate) identity map. However, general activation functions do not have such properties, and hence, how to extend their constructions to general activation functions is unclear.\n\n[1] Li et al., \"Deep learning via dynamical systems: An approximation perspective.\" Journal of the European Mathematical Society, 2022.\n\n[2] Duan et al., \"Vanilla Feedforward Neural Networks as a Discretization of Dynamical Systems.\" Journal of Scientific Computing 101, no. 3 (2024): 82.\n\n> The proof strategy in this paper is essentially no different from those in earlier works, especially the encode-memorize-decode techniques considered in Park (2021) or the earlier bit extraction techniques. Therefore, I believe this paper is an incremental contribution and is not suitable for publication in a top conference like ICML.\n\nWe strongly disagree that our proof strategy is not different from those in earlier works. As we mentioned in the previous response, the existing constructions by Park et al., (2021b) highly rely on specific properties of ReLU and its variants, which cannot be found in general activation functions. To show the tight minimum width for a general class of activation functions, we propose a completely different condition for activation functions called *squashable* (Definition 1), and our proofs are based on this condition. \n\nSince we are using different properties of activation functions, our constructions of the encoder and decoder significantly differ from prior ones. Our decoder construction is based on a novel filling-curve argument (Definition 2) that did not appear in prior constructions. Our encoder explicitly slices the input domain along each coordinate via a width $d_x$ network using the squashability, while prior constructions use either a larger width $d_x+1$ (Park et al., 2021) or only prove the existence of a proper slicing (Kim et al., 2024) based on the properties of ReLU. \n\n> The cited papers are not discussed in detail in this paper, especially Park (2021), Cai (2023), and Kim (2024).\n\nWhile we compared our bound with the bounds in (Park, 2021b; Cai, 2023; Kim, 2024) in Sections 1.1–1.2, following the reviewer’s comment, we will add the above discussions and thoroughly compare our proof techniques with prior ones in the final draft.\n\n> The activation functions considered in this paper are related to the Step function, yet the paper does not mention the study on Step activation functions in Park (2021).\n\nPark et al. (2021b) showed that if a network can use both ReLU and Step, the minimum width for universal approximation under the supremum norm is $\\max\\\\{d_x+1,d_y\\\\}$. Here, they use the discontinuity of the Step function, which enables them to derive the tight minimum width under the supremum norm. On the other hand, our activation functions of interest (i.e., those satisfying Conditions 1 and 2) are continuous and do not have additional ReLU; hence, the proof completely differs from the existing ReLU+Step network result. We will add this discussion to the final draft.\n\n> The first time I encountered the equation $\\max\\\\{d_x,d_y,2\\\\}$ was in Cai (2023), but it is not listed in Table 1.\n\nFollowing the reviewer’s comment, we will add the bound $\\max\\\\{d_x,d_y,2\\\\}$ by Cai (2023) to Table 1 in the final draft.\n\nWe would be happy to clarify any concerns or answer any questions that may come up during the discussion period.", "claims_and_evidence": "I think the conclusion of this paper is reliable.", "methods_and_evaluation_criteria": "I think the method in this paper is credible. However, there have been many discussions on such problems in the literature. In particular, the idea of this paper is obviously borrowed from the existing literature, such as Park (2021) and Kim (2024), but it is never compared and discussed in the paper.", "theoretical_claims": "I did not check the proof of this paper.", "experimental_designs_or_analyses": "This paper is a theoretical article and does not require experiments.", "relation_to_broader_scientific_literature": "I feel that this paper is an incremental generalization of existing research.", "essential_references_not_discussed": "The cited papers are not discussed in detail in this paper, especially Park (2021), Cai (2023), and Kim (2024).", "other_strengths_and_weaknesses": "I was hastily assigned to review this paper. I did not verify the details of the proofs, but I believe the conclusions are correct. However, the results of this paper did not impress me. The minimum width under the $L^p$-norm has already been thoroughly studied in excellent works, such as Park (2021), Cai (2023), and Kim (2024), which have analyzed many different cases. While previous papers have not exhaustively covered all activation functions, it is straightforward to deduce results for many activation functions from existing findings. The proof strategy in this paper is essentially no different from those in earlier works, especially the encode-memorize-decode techniques considered in Park (2021) or the earlier bit extraction techniques. Therefore, I believe this paper is an incremental contribution and is not suitable for publication in a top conference like ICML.", "other_comments_or_suggestions": "1. The activation functions considered in this paper are related to the Step function, yet the paper does not mention the study on Step activation functions in Park (2021).\n\n2. The first time I encountered the equation $w_{\\min} = \\max(d_x, d_y, 2) $ was in Cai (2023), but it is not listed in Table 1.", "questions_for_authors": "Null.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Yuyz5wftGj", "reviewer": "Reviewer_HPdg", "summary": "The paper under review proves universal approximation of neural networks with a fixed squashable activation function and minimal width equal to the maximum of input and output dimension on compact sets with respect to  the L^p-norm, hence any L^p-function with values in R^{d_y} can be approximated in this norm up to an arbitrarily small error. This result extends prior results, e.g., of Cai (ICLR 2023) that derived the same width, however with a family of activation functions (leaky ReLu with variable slope), from that one can choose one in each activation. The paper applies an encoding-decoding scheme similar to Kim, Min and Park (ICLR 2024), however in one dimension less. to this aim the authors show the existence of approximately space filling curves implemented by neural networks as decoder. Furthermore, an approximate encoder network is designed that maps regions in the input space to the portion of the curve, that is decoded to an approximation of the function value of a continuous function on this input space. Therefore, the composition of encoder and decoder provide a suitable approximation of the given function in L^p. As continuous functions are dense in L^p, this result suffices to prove the claim. The authors also state sufficient conditions for an activation function to be slashable.\n\n# # will lower the score by one as I reassess novelty, but I stilll give a weak accept because this paper has still something to ell in terms of generalizing activations.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive evaluation and thoughtful feedback. We address all comments of the reviewer below.\n\n> It does not occur to me what is the sense of the first assertion in lemma 7. Isn't that obvious?\n\nAs the reviewer pointed out, $w_{\\sigma}\\ge\\max\\\\{d_x,d_y\\\\}$ is a trivial lower bound. We wrote it to formally show that our upper bound on the minimum width is tight. We will write that this bound can be easily derived in the final draft.\n\n> Lemma 10, the symbol for the ball should be introduced.\n\nThe notation $\\mathcal B$ for the ball is defined in the first paragraph of Section 2 (Line 118). Nevertheless, for better readability, we will add a pointer to this definition after Lemma 10 in the final draft. \n\n> How does one see that ReLu is not squashable in order to explicitly show that there is no contradiction to Lu et al (NeurIPS 2017)? A formal argument given in the appendix would be solicited.\n\nHere, we clarify the following claim: “Our minimum width $\\max\\\\{d_x,d_y,2\\\\}$ for squashable functions does not contradict the lower bound $d_x+1$ for ReLU networks in (Lu et al., 2017) since ReLU is not squashable.” If this comment is not what the reviewer considered, please let us know.\n\nWe note that the lower bound $d_x+1$ in (Lu et al., 2017) considers the entire Euclidean space $\\mathbb R^{d_x}$ while we consider a compact subset in $\\mathbb R^{d_x}$. In fact, the tight minimum width for ReLU on a compact domain is known to be $\\max\\\\{d_x,d_y,2\\\\}$ (Kim et al., 2024), which is smaller than the minimum width $\\max\\\\{d_x+1,d_y\\\\}$ for ReLU on $\\mathbb R^{d_x}$ (Park et al., 2021). Namely, we can have the minimum width $\\max\\\\{d_x,d_y,2\\\\}$ for ReLU regardless of its non-squashability.\n\n> This is only a preprint, should however be mentioned: [1] Dennis Rochau et al., New advances in universal approximation with neural networks of minimal width. The authors should comment shortly on their minimal internal depth 1 achieved, as this has implications for the universal approximation theory of autoencoders, cf [1]\n\nWe appreciate the reviewer for suggesting an interesting connection between our results and autoencoders, and providing an intriguing paper. According to Dennis et al., we also found that our construction has an autoencoder structure (i.e., minimal internal width 1). We will cite this paper and add a related discussion, including a new lower bound for the supremum norm in (Dennis et al.), to the final draft.\n\nWe would be happy to clarify any concerns or answer any questions that may come up during the discussion period.", "claims_and_evidence": "All claims are supported by detailed mathematical proofs. in addition, the authors explain and illustrate te underlying ideas in a transparent manner.", "methods_and_evaluation_criteria": "The methods are from mathematical analysis and approximation theory.", "theoretical_claims": "The theoretical claims are credible and supported by detailed proofs, in which I did not find any errors.", "experimental_designs_or_analyses": "dies not apply", "relation_to_broader_scientific_literature": "The paper deals with the classical topic of universal approximation.", "essential_references_not_discussed": "This is only a preprint, should however be mentioned:\n[1] Dennis Rochau et al., New advances in universal approximation with neural networks of minimal width, https://arxiv.org/abs/2411.08735\nNamely, this paper also deals with minimal width and minimal internal width for LReLu functions with variable slope. The results of the paper under review, however, are stronger as they work with a fixed activation function.", "other_strengths_and_weaknesses": "The paper more or less settles a classical topic on universal approximation with minimal width in a very satisfactory manner. Namely that the minimal width for neural networks with fixed slashable activations are universal approximators in L^p the theoretically minimal width w=min{d_x,d_y}. This is attractive as it covers almost all used activation functions up to ReLu and the standard FCNN. The ideas are expressed very clearly and concisely and are supported by nice illustrations. I therefore think that the paper stands for a natural endpoint of recent developments in this direction.\n\nI don't see any major weaknesses.", "other_comments_or_suggestions": "* It does not occur to me what is the sense of the first assertion in lemma 7. Isn't that obvious?\n* Lemma 10, the symbol for the ball should be introduced.\n* The authors should comment shortly on their minimal internal depth 1 achieved, as this has implications for the universal approximation theory of autoencoders, cf [1]", "questions_for_authors": "How does one see that ReLu is not squashable in order to explicitly show that there is no contradiction to Lu et al (NeurIPS 2017)? A formal argument given in the appendix would be solicited.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LAC07n9is6", "reviewer": "Reviewer_TQ5s", "summary": "The paper investigates the exact minimum network width required for universal approximation of $L^p$ functions on a cube $[0,1]^{d_x}$ by neural networks utilizing *squashable* activation functions. Squashable functions are defined in this paper as activation functions that, when composed alternatively with affine transformations, can approximate both the identity function and binary step function. The paper proves that the exact minimum width required is $\\max\\\\{d_x, d_y, 2\\\\}$, with special conditions when $d_x = d_y = 1$. It provides conditions under which common activation functions (non-affine analytic and certain piecewise functions) are squashable, significantly extending prior results.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive evaluation and thoughtful feedback. We answer the reviewer’s question below.\n\n> In practice, both and norms are too weak because derivatives are not guaranteed to converge. For example, can we state the universality in a Sobolev space?\n\n\nWe thank the reviewer for this interesting question. For the Sobolev $W^{1,q}$-norm with small $q<d_x$, by the Sobolev embedding theorem, the Sobolev space $W^{1,q}(\\mathbb{R}^{d_x})$ can be embedded into $L^p(\\mathbb{R}^{d_x})$ with $q=pd_x/(d_x-p)$. Hence, in this case, universal approximation under $W^{1,q}$-norm reduces to our result. However, we do not know whether our minimum width bounds extend to $q\\ge d_x$. We think finding the minimum width under general Sobolev spaces is a very interesting problem and would like to pursue it as a future research direction.\n\n\nWe would be happy to clarify any concerns or answer any questions that may come up during the discussion period.", "claims_and_evidence": "- Claim: For squashable activation functions, the exact minimum width for universal approximation is $\\max\\\\{d_x, d_y, 2\\\\}$.\n  - Evidence: The claim is rigorously supported by explicit constructions and proofs (Theorem 2 as a corollary of Lemmas 6, 7, 8, 9, and 10).\n\n- Claim: Non-affine analytic functions and certain piecewise functions (e.g., leaky-ReLU, HARDSWISH) are squashable.\n  - Evidence: Detailed proofs provided for these classes (Lemmas 4, 5).", "methods_and_evaluation_criteria": "The paper’s methodological framework, which includes defining the concept of squashable activation functions and establishing theoretical criteria, is appropriate and sensible for addressing the minimum width problem for universal approximation. Evaluation criteria through formal proofs are rigorous and clear.", "theoretical_claims": "The main theoretical claim is the precise characterization of minimum width for universal approximation for squashable functions. The correctness of proofs appears solid and well-structured.", "experimental_designs_or_analyses": "n/a", "relation_to_broader_scientific_literature": "The paper builds significantly upon prior literature about universal approximation theory, particularly works focusing on ReLU-like activations. It substantially generalizes the known exact minimum width from ReLU-specific results (Park et al., 2021b; Kim et al., 2024; Cai, 2023) to a broader class including analytic and piecewise continuously differentiable activations.", "essential_references_not_discussed": "The paper appears thorough in citing relevant literature. No obvious missing essential references were identified.", "other_strengths_and_weaknesses": "Strength: Figures 2-4 help understanding the idea behind the proof.", "other_comments_or_suggestions": "good job", "questions_for_authors": "In practice, both $L^p$ and $\\sup$ norms are too weak because derivatives are not guaranteed to converge. For example, can we state the universality in a Sobolev space?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper provides novel insights on universal approximation properties of neural networks with certain activation functions. All reviewers seem to agree that the paper is mathematically sound. However, there were big discussions about the paper's relation to previous work and the way the previous work is discussed in the submission. Three out of four reviewers see a significant contribution in the submission and are fine with the way the previous work is presented. The fourth reviewer initially expressed strong disagreement with that assessment. After a long discussion, the reviewer also raised the score to "weak accept", while still voicing doubts. At the end, I follow the opinion of the majority of the reviewers, recommending accept. However, I urge the authors to take the criticism seriously, even the parts that initially sound extreme. They should definitely improve the discussion about the relation to previous work, including proof strategies in previous work, in the revised version.
Decision: Accept (poster)