Optimal Minimum Width for the Universal Approximation of Continuously Differentiable Functions by Deep Narrow MLPs
Authors: Geonho Hwang
Venue: NeurIPS 2025 poster ()
Keywords: Universal Approimation
Sobolev norm
Continuously differentiable functions
deep narrow MLPs
deep learning
Primary area: theory
TL;DR: In this paper, we investigate the universal approximation property of deep, narrow multilayer perceptrons (MLPs) for $C^1$ functions under the Sobolev norm, specifically the $W^{1, \infty}$ norm.

Abstract: In this paper, we investigate the universal approximation property of deep, narrow multilayer perceptrons (MLPs) for $C^1$ functions under the Sobolev norm, specifically the $W^{1, \infty}$ norm. Although the optimal width of deep, narrow MLPs for approximating continuous functions has been extensively studied, significantly less is known about the corresponding optimal width for $C^1$ functions. We demonstrate that \textit{the optimal width} can be determined in a wide range of cases within the $C^1$ setting. 
Our approach consists of two main steps. First, leveraging control theory, we show that any diffeomorphism can be approximated by deep, narrow MLPs. Second, using the Borsuk-Ulam theorem and various results from differential geometry, we prove that the optimal width for approximating arbitrary $C^1$ functions via diffeomorphisms is $\min(n + m, \max(2n + 1, m))$ in certain cases, including $(n,m) = (8,8)$ and $(16,8)$, where $n$ and $m$ denote the input and output dimensions, respectively. Our results apply to a broad class of activation functions.

Optimal Minimum Width for the Universal Approximation of Continuously Differentiable Functions by Deep Narrow MLLs
Geonho Hwang
Department of Mathematical Sciences Gwangju Institute of Science and Technology Gwangju, Buk-gu 61005 hgh21340gist.cc ac.
Abstract
In this paper, investigate the universal approximation property of deep, narrow multilayer perceptrons (MLPs) for C²2 functions under the Sobolev norm, specif- ically the W norm. Although the optimal width of deep, narrow MLPs for approximating continuous functions has been extensively studied, significantly less is known about the corresponding optimal width for C' functions. We demon- strate that the optimal width can be determined in a wide range of cases within the C' setting. Our approach consists of two main steps. First, leveraging control theory, we show that any diffeomorphism can be approximated by deep, narrow MLPs. Second, using the Borsuk- Ulam theorem and various results from differ- ential geometry, we prove that the optimal width for approximating arbitrary C functions via diffeomorphisms is min(n + m, max(2n + m)) certain cases, including (n, m) = (8, 8) and (16, 8), where n and m denote the input and output dimensions, respectively. Our results apply to broad class of activation functions.
Introduction
min(n + m, max(2n + 1,m)
(1)
|/ - -lllwwee = Σ | - -0))gl
(2)
lalSk
The choice of neural network architecture plays a crucial role in determining performance. However, in practice, architectural decisions are often made through trial and error. It is therefore important to provide theoretical guidance on what should be avoided and how to select appropriate width and depth based on the input space, target function, and specific tasks. The universal approximation property (UAP) refers to the ability of deep learning models to approximate given class of functions. Since deep networks must approximate general functions perform specific tasks, the UAP has received considerable attention as a theoretical foundation. While various forms of universal approximation theorems exist depending on the network type and its characteristics, one actively studied setting is the universal approximation property of deep, narrow multilayer perceptrons (deep, narrow MLPs), which reflects the practical scenario where networks are deep but relatively narrow in width.
MLPs with fixed width and arbitrarily large depth exhibit different universal approximation behavior depending on whether their width exceeds critical threshold (Johnson 2018 Kidger & Lyons 2020). This threshold called the minimum width, and numerous studies have investigated upper and lower bounds for this threshold based the input dimension n, output dimension m. and the choice of activation function.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

The most intensively studied case involves the approximation of continuous functions under the uniform norm. Notable results include the upper bound n + m + c(o), where c(or is a constant depending on the activation function, shown by Hanin & Sellke (2017); Kidger & Lyons (2020).
More recently, Hwang (2023) improved this upper bound to max (2n + 1,n).
Jounds,JJhnnon 20188; Cai 2022); Kim et al. (2023) proved that the minimum width must be least n + 1 or m + S2m depending on the setting. However, few studies have succeeded
For lower bounds, Johnson
in narrowing the gap between known lower and upper bounds. Among the few, Park et al. (020);;
Hwang (2023) proved optimality in specific cases: the minimum width is for (n, m) (1, and
for (2r(2,)) (2,
Beyond the uniform norm, there has also been research under other norms. Park et al. (2020)
established the optimal minimum width of deep, narrow MLPs with ReLUU activation in the 7.p norm.
However, research on general norms beyond the L, and uniform norms remains limited.
However, there has been scarce number of papers that study norms involving derivatives of functions in the setting of deep narrow MLPs. Many deep learning techniques directly penalize the difference between the derivative of the target function and that of the network. These include Sobolev Training
(Czarnecki et al. 2017). Physics-Informed Neural Networks (PINNs) (Raissi al., 2019), and Generative Adversarial Networks with gradient penalty (Gulrajami et al. Arbel et al. 2018).
In this work, we determine the minimum width required to approximate continuously differentiable functions in Sobolev spaces. Specifically, we focus on approximation with respect to the 1.,00 norm. Compared to the uniform norm, the topology of the Sobolev norm W1.c0 is finer, enabling tighter lower bound estimates. For the upper bound, tools from control theory allow us to match the upper bounds known in the uniform norm setting. Using these ideas, we compute both upper and lower bounds for approximation in Sobolev spaces. In some cases, the lower bound coincides with the upper bound, thus identifying the optimal minimum width. This includes interesting cases such as (8, 8) and (16, 8). The exact pairs to which our result applies can be found in Theorem 5.9
Our contributions are as follows:
We show that deep, narrow MLPs can approximate arbitrary diffeomorphisms with respect to the Sobolev norm W1..00 (Theorem 4.1)
We precisely characterize the additional width required to approximate arbitrary continuously differentiable functions compositions of diffcomorphisms and linear transformations. (Definition 4.3 and Theorem 4.6)
Using these results, we prove that the known upper bounds n + m and max (2 + 1, m) under the uniform norm also hold under the Sobolev norm W1.0s, (Theorem 5.1
We prove that these upper bounds are also lower bounds for infinitely many combinations of n and m, and therefore. these values represent the optimal minimum width in those cases. (Theorem 5.9
Related Words
In this section, we review previous studies on the universal approximation property (UAP). Cybenko (Uben.Cyyenkoo (1989) proved that a two-layer MLP possesses the UAP in the space of continuous functions. This result was extended by Leshno et al. 1993) to more general activation functions.
While these early results focus on two-layer networks, subsequent research has investigated the UAP of deep, narrow MLPs. Hanin & Sellke (2017) established a universal approximation theorem for deep. narrow MLPs with ReLU activation, providing both lower and upper bounds on the minimum width. Johnson 2018 showed that width of least n n is required for networks with monotonic activation functions. Kidger & Lyons (202)) proved that width of n + m + 1 suffices for general non-polynomial activation functions, while + m + 2 is sufficient for polynomial activations. Park et al. (2020) demonstrated that the optimal minimum width is three when n = 1 and m = 2 with ReLU. Cail (2022) showed that a width of at least max(n,, m) is necessary for general activation functions. Kim et al. (2023) proved lower bound of m + 1 12mmmmm Hwang (2023) established an upper bound max 2n + 1,m) for networks using the Leaky- activation and showed that the optimal minimum width is four when m 2.

---

There have also been investigations of the UAP under norms other than the uniform norm Park et al. (2020) showed that the optimal minimum width max(n + 1, m) in the Lp R space for ReI.U networks. Additionally, Kim et al. (2024) demonstrated that in the [1) 1J", IR"" setting, the optimal minimum width becomes min n, 2)
In addition to studies on MLPs, there has been significant progress in understanding the universal approximation property of residual networks (ResNets). Lin & Jegelka (2018) demonstrated that even ResNets with one-neuron hidden layers can serve as universal approximators, highlighting the expressive power that arises from their residual structure. Aiizww et al. (2020) extended this line of research by analyzing both ResNets and ODENets, providing rigorous mathematical results along with supporting numerical experiments. More recently, Tabuada & Gharesifard (2022) investigated ResNets from a control-theors perspective.
Beyond function value approximation, some universal approximation theorems also consider deriva- tives. Li (999) proved that a two-layer MLP can approximate arbitrary derivatives of a function, provided the activation function is sufficiently smooth.
However, these results do not cover the Sobolev norm in the context deep narrow MLPs. this paper, we provide partial answer to this open question by establishing results under the W1 norm.
3 Notation and Definition
In this section, we introduce the notations and definitions used throughout this paper: N denotes the set of natural numbers, and No- = N U {0}. (r) denotes the open ball in I² centered at the origin with radius For set A C Rd A denotes the closure of A with respect to the Euclidean norm.
For two open sets V CU C Rd, we say that is a precompact subset of U V C Rd is compact and U. We denote this as E U. For sets A, B C IRd, the Minkowski sum is defined as A + B (x + + y | A
For a d-dimensional vector æ E Rd, we denote by x; the i-th component of x; in other words, (æ1, 222 ...,, F.). Similarly, for a function f X + R², write denote the i-th component function, so that (x) = (z),..... (x)). We use Piz to represent the (j - i + 1 1)-dimensional subvector (z,, Fi+1. "it,....,,,, Ty). For vectors x. y E Rd, the dot product is denoted by x y E R and detined as r- := i== Z4ys. For vectors we define the operation a
(Z1, Id,) gd and y (VI,..., No.) E Rd, y (I1 [I...... Fd, 91 .1..... Md.) +dl+ Similarly, for functions f: X Rd and g X + Rdd, we define I / g X - Rdi+da by (J @ gz) = f(x) Φ g(r).
Let Aff denote the set of affine transformations from Ra to R For function f X Y and a subset X' X, we write flx to denote the restriction of f to the domain X'. For r E No. the space (";;; Y) denotes the set of functions that are r-times continuously differentiable. For U C R" and mixed partial derivative gr gr1-. dxj"
TT) E NT. the space ("U R') consists of functions f such that the
exists and is continuous. For k E No and r E [0, 1). the space Our"" ".."""""""""
Ckk (U; ) consists of functions whose k-th order partial derivatives are Hölder continuous with exponent In particular, CO (U; RT') denotes the space of Lipschitz continuous functions. We define cld (U; RTT) as the space of locally Lipschitz continuous functions: that is, Cbd (U;
for every precompact set V E U, there exists a constant Lv such that [[f())) - f(y)]l x | yll for all x, y E V. We denote the Lipschitz constant of f on V by Lv (f).
3.1 Sobolev Space
We define the Sobolev space follows: We denote the weak derivative of by Du.
Definition 3.1 (Sobolev Space), Let n, le E N, p E N (oo), and let U C R be an open set. The Sobolev space W'PP(U) is defined by
WkP(()) = {u E IP(U) D IP(U for all multi-indices with |a| $ k).
(3)
equipped with the norm
|klllwooe = Σ ||
(4)
3

---

The vector-valued Sobolev space W WU, for m, k E N is defined as
W == {u= (ui,... Um) un E W {U)}
(5)
with the norm
[
(6)
More specifically, we focus on the following local Sobolev space, considering compact domains: Definition 3.2 (Local Sobolev Space). Let U C. R" No. and E [1, oo). The local Sobolev space W (U;R") is defined as the projective limit:
(U; (R)) )
lim (V*(VV
loc
VeU
where the right-hands side given explicitly as
( e II WT*(VRR")
fvlv= for all V22 c V
(8)
VEU
The local Sobolev space is equipped with the relative topology inherited from the product topology of the spaces W (V;
In this paper, focus the Sobolev norm W1,0000 It is well known that W: C... See Theorem 4.5, p.155 in Evans (2018) for details. is also known that for convex domains, the Sobolev and Lipschitz spaces coincide (Theorem 1 in Heinonen (2055): if V Rd is convex, then (1)x((V) = (V). Moreover, there exist constants C₁ >> 0 depending only on d and n such that (see Theorem p.279 and Theorem 6, p.281 in Evans (2022)))
Ci (V S (f) + [I S
(9)
For convenience, we will always take the continuous representative among functions that differ only on a set Lebesgue measure zero.
For set of functions A Wl.P(U; )",, we denote by A W1.00 A the closure of with respect to the norm closure in the local Sobolev topology by A W- loc
IIw Similarly, for a set functions A C W.1. WU; (U;R")) R"", we denote the
Al
3.2 Activation Function
We adopt the commonly used condition on activation functions, as proposed by Kidger & Lyons (2020).
Condition 1. There exist constants o: E IR and e E R__ such that a nonlinear activation function a is C² function the interval (a c, e), and of (a) # O.
The ReLU activation function is defined as
ReLU(x)) = x
if 2 0, if
(10)
0
and the Leaky-ReLU activation function is defined as
LRa(x) = f
if x 2 0, if r <
(11)
Bx
We consider MLPs with sets of activation functions. For example, MLPs with the Leaky-ReLU activation function select an activation function from the following set at each layer:
LR (LR, R+)
(12)
We use the symbols and to denote an activation function and set of activation functions, respectively. We define Leaky-ReLU-se activation functions as follows:

---

Definition 3.3 Leaky-ReLU-likee A set of activation functions Σ is called Leaky-Rela Leaky-ReUU-like if and only for each B R,, there exists a C' activation function σ Σ such that
[)(a) 1,
(
lim
lim
3,
(13)
x
and
sup
Do (x) 11
0. B-1
(14)
We also denote the identity function by id:
id a
(15)
Activation functions applied to vectors operate componentwise. For set of activation functions Σ. define nd as
x = { : - R E
(16)
3.3 Deep Narrow MLP
We define the set of deep, narrow MLPs with set of activation functions E, arbitrary depth, input dimension n, output dimension m, and at most w intermediate dimensions as A (The exact
definition provided in Appendix A.I For a singleton activation function o, define:
A (o)
(17)
For natural numbers zero-padding inclusion Jm,n Rmm + R as:
N. define the natural projection Pn.m R" R and the
Pn,m (T), Unn = (.1,..., Fmm)
(18)
Im (11, ................................ Fmm (x1, Tmyy ,,....00 0)-
(19)
Any function f A n,m,w can be decomposed as:
f Pue.n P 9 qu,w:
(20)
where 9 E A 0,00000 Note that if E A E 0,00000 then their composition 92 also belongs to
Σ
From this point on, we will use the notation refer to either single activation function or a set of activation functions E, depending on the context.
3.4 Subsets of Diffeomorphisms
We define the of diffeomorphisms. For definitions of concepts from differential geometry, Appendix A.2
Definition 3.4 (Diffeomorphism: (U)),, Let U Rdd be an open subset, and let be non-negative integer infinity. Then D' (U) denotes the C" -diffeommorphisms from U to TRd..
4 Universal Approximation
4.1 Problem Formulation
W-'. Our primary goal is to identify the minimum width wmin E N such that any continuously differen- in the topology
tiable function f E C* (R"; can be approximated by elements of A
n.m,an wcm.on
1000
of 1, (IR"; RT)) In other words, our aim is to determine the value w lon
wh..0
gin
(n, m,m, σ)) such that
min
11.00 min (n, a) min { l N C² (R": R')
w.o0
A
(21)
n.m.l
w min (n, m. a)) denotes the minimum width for which MLPs of this width and arbitrary depth can approximate O functions to any accuracy in the 11 norm.
W1.00
5

---

4.2 Diffeomorphisms and Continuously Differentiable Functions
Our proof strategy is divided into two parts. First, we approximate diffeomorphism using deep narrow MLPs with a small additional width. Next, we show that any continuously differentiable function can be approximated by a composition of affine transformations and diffeomorphisms, and we rigorously estimate the required width. In this subsection, we aim to prove the following theorem, which asserts that a diffeomorphism can approximated by deep narrow MI.S.
Theorem 4.1. Let be of non non-polynomial -'llfttttoon, LR. ReLU, Leaky-ReLLe ReLU--LLeU-- activation functions. Then. for any natural number E N, the following relation holds:
D³ DRRR.) S d,d.d-a((a)
luc
(22)
where
0 o = LR or a is Leaky-ReLU--H
a(o) = {1 if = ReLU or a is non-polynomial -funcctoo
(23)
The theorem states that deep, narrow MIPs with small additional width can approximate arbitrary diffeomorphisms. The proof relies on techniques from control theory. Approximating an entire diffeomorphism directly using a neural network is challenging. To address this, we interpret a diffeeomorphimm the solution of an ordinary differential equation evolving over time. In other words, the existence of a diffeomorphism implies the existence of continuous flow connecting the identity map to the diffeomorphism. The direction and magnitude of this flow are determined by vector field. The problem then reduces to approximating this continuous flow step by step, which is equivalent to approximating vector field. Deep, narrow MLPs approximate such flows over sufficiently small time intervals by leveraging the universal approximation property. Then, by approximating the flow generated by this two-layer MLP using deep narrow MLP, complete the argument. The full proof is provided in Appendix C.T
Now, we introduce quantity O(n, m) such that any continuously differentiable function from [0, 1J"d to R can be approximated by a composition of affine transformations and (2 m) -dimensionall diffeommrppismss We further show that this width is optimal. To this end, we begin with the following lemma
Lemma 4.2 (Theorem C Palaiss (1960). Let n, m E N with n S m, and let f K =00, R be a smooth embedding. Then, there exists a smooth diffeomorphism F IR + IR" such that the following equation holds:
F Jn.m =
(24)
The lemma implies that any smooth embedding can be decomposed into affine transformation followed by diffeomorphism. Now, let Emb(X, Y) denote the set of smooth embeddings from X Y. We define the quantity M(n, m) as follows:
Definition 4.3 (9(n, m)).
An, m) = min { E No pt.m Emb( (0, 11"" ()),)
(10,
(25)
where the closure is laken with respect to the C* -norm.
Using the lemma above and the definition of Q(, m)), we state the following theorem:
Theorem 4.4. Let a be one of non- non-polynomial C1.l -function LR, ReLU, or Leaky-Reis ReLU- activation functions. Then, for any natural numbers and m, the following relation holds:
loc
C² (R";R"' C A n.m.a(n,m))a((e)
(26)
where
0 ifo LR or a is Leaky-ReLU-like
(
(27)
i = ReLU or a is a non-polynomial C Ct.1-function
The proof of the theorem is provided in Appendix D.I The preceding theorem shows that 2(n, m) is a sufficient width for approximating functions with n-dimensional input and m-dimmensoonal output. Conversely, the following proposition demonstrates that 2(n, m) is also a necessary width for such approximation.

---

Proposition 4.5. Let be set of non-decreasing C' activation functions. Then, for natural numbers and m, the following relation holds:
C² (R"; & n.m.f(mmmm)]1
loc
(28)
The proof of this proposition is provided in Appendix D.2 By combining the previous theorems with this proposition, derive the following result. This theorem demonstrates that the purely geometrically defined quantity Q(n,, m) has a fundamental connection to the minimum width of deep narrow MIPs.
Theorem 4.6. The following relation holds:
W...00 W Winn (n, m, Qm m)
(29)
for a Leaky-ReLU-iike a in which every element is increasing, and
Q(n, m) S w. (n, o) S 92 m) 1,
(30)
for a set of C increasing activation finctionns a.
5 Calculation of S2(1, m)
In the previous section, we showed that (nn, m) nearly determines the minimum width required for universal approximation. this section, we provide general bounds for S2n, m) and compute exact values for specific cases.
5.1 Upper Bound of Q(n, m)
We begin by establishing the following general upper bound.
Theorem 5.1. The following relation holds:
Q(n, m) S min(n + m, max(2n + 1, 1,))).
(31)
Proof. The inequality Q(n, m) < + m follows directly from the definition of Ω2n, m). N(n, m)
<
max (2n + 1, m) is by Lemma 5.2
Lemma 5.2. Consider natural numbers and m where m 2n. Let f ('";""R")) be continuously differentiable function. Then, for bounded open set U C Rh and a positive number c C R_, there exists a smooth embedding g U Ih. such that
|f 1ff-9/liiee gl <E
(32)
Proof. This is direct consequence of the transversality theorem. (See Chapter 3, Theorem 2.1 of Hirsch (2012) for details)
In some cases, improve the general bound established above.
Lemma 5.3. For even the following eqaution holds:
22k, 2h - 1) 2k.
(33)
have l(k, 24 1) > 2k. Thus, it suffices to prove that Q(k, 2k 1) < 2k. As immersions are dense in C¹ (B. R2k-1), it is enough to approximate an immersion f. By Corollary 3.2 of Lashof Smale 1959), there exists smooth embedding & such that ||P28,26-1 og f fllws.> (U)Rm) c. Note that while the original result is stated for the uniform norm, the same proof applies directly in the C norm setting.
Proof. By Kim al. (2023).

---

5.2 Lower Bound of (2(2,
In this subsection, we present lower bound for certain cases, which coincides with the upper bound established in the previous section, thereby yielding the optimal minimum width. To prove the lower bound, we require an argument of the following form: Given function f R + R"", there exists <> O such that if the codomain dimension of another function g is small, then the concatenation f o g cannot be an embedding. To this end, we construct a function f whose seff-intersection is transversal and has the structure of a sphere S" If all antipodal points on the sphere are mapped to the same value by f. then we can apply the Borsuk theorem.
Lemma 5.4 (Borsuk- Ulam Theorem). Let h 5th - R" be continuous function Then there exists a point x E 5th such that
M(x) h(-
(34)
The Blas theorem states that every continuous map from an n-dimensional sphere to IR maps some pair of antipodal points to the same point. Now, suppose we have embedding S" + R" and a map f such that f(x) = f(-x) for all antipodal points I E S Then, for any function g R' IR,, there exists pair of antipodal points on S" that are mapped to the same value by g. Therefore, the map G = f o g cannot be injective, and hence cannot be an embedding. This leads to the conclusion (2,,, m) 2
r 1.
The difficulty, however, lies the fact that we must consider a map such that Pp o G is small, rather than requiring exact equality Pmtrr, G = f. The following lemma guarantees that the diffcomorphic structure of the self-intersection is preserved under small perturbations in the C'l norm.
Lemma 5.5 (Ehresmann's 's Lemma for Intersection). For n, E N with 2n m. consider precompact set U C. 12" and a C' function f: f: U + R" in transversal position. Then there exists > 0 such that the following holds: Consider arbitrary g E C² (U; satisfying
Ilf glwwi.xxUURR) <c
(35)
Define the diagonal A U X U as
A (1x,0) EU xU x EEU..
(36)
Define F.U x U A R as
F( u) = f(x) f(y).
(37)
Similarly, define ğ as
f(.,v) = g(x) glv). Then, there exists a C¹l diffeomorphism +b f (0) + g (0) such that
(38)
(xx T ((f(y.x))
(39)
where T denotes the involution (I. v) (y. 2).
The proof of Lemma 5.5 is provided in Appendix E.1 Now, the only remaining task is to construct a function with such self-interscction structure. The following two lemmas provide results for specific cases.
Lemma 5.6. Assume that there exists a submersion f IRP=11 x (-1,1) - R". Then, the following relation holds:
Q(n, m) = n + m.
(40)
Proof. Let f RIPa11 x (-1,1) + R be a submersion. Then, there exists lifting f 1 x (-1,1) + Rm such that for a canonical two-to-one covering S x 1-) + RP x (-1, 1). have p f = f. Then, as all antipodal points have the same / values and the intersection is transversal, follows from the previous arguments that 2(n, m) 2 n + m. This completes the proof.
Lemma 5.7 (Projective Space Submersion Lemma). For n E N, consider a, b E No such that n +1 2i0-b x G, where 0 S b S and e is an odd number. Then, for any natural number m E N satisfying m Sa + 29, RIn (-1, 1) can be submerged into a
8

---

Proof. By Theorem B of Phillips exists section in F (M) where Fm (MT) is the bundle of m-frames tangent to M. This condition is equivalent to the existence of m linearly independent vector fields. By Theorem 1.1 of Davis (2012). the maximum number of linearly independent vector fields on IRP equals Sa + 2 where + = 2 x for 0 S b < 3 and an odd number E N. Therefore, IP" x (-c, c) has Sa 2 independent vector fields, and thus can be submerged into 88**".
1967). there exists a submersion M - R" if and only if there
We can also verify that an immersion with an 4m, which implies that (n, m) 2n + 1.
intersection exists when 3n + < 2m S
Lemma 5.8 (Theorem 3 of Miller (1969). Given n and mn + 1 < 2m S 4n 2. Then if
0 (mod@2n-ml there exists a transversal immersion 5 -> Rm++1 with self-intersection RP2n- m m-1 Here, Cm defined as
2
if 8r,
24++1
if Sr 1,
Cm
24r+2
if m = 8r + or 8r +3
(41)
24r43
im Sr + 8r + 5, 8r + 6, or 8r + 7.
By combining all the results, we obtain the following theorem. Theorem 5.9. 2 f a, E Ny satisfying S S 3 and m S Sa + 2b,
Q(n, m) = m + n.
(42)
If 33 <m= < 2n and + = 10 (20de22n- mm
M(n, m) 2n + 1.
(43)
2n + 1 m,
((n.m) m.
(44)
Remark 5.10. At first glance, the dependence of the optimal minimum width on the parity of the imput and output dimensions may appear somewhat artificial. However, Lemma 5.3 and Theorem 5.9 together yield the relation
2k, 2k - 1
if i is even,
1(k, 2k 1)
1)
(45)
if is odd,
which provides strong evidence that this parity dependence may be a fundamental property.
Limitation
Although our results yield optimal values in many cases susc f?8. 8) = 16 and 2216 8) 24-they do not apply to all combinations of n and m. Our analysis is asymptotically valid primarily when m is either much smaller than significantly larger, specifically in the regime where 3n < 2m. Developing a theoretical framework that addresses the intermediate regime not covered by theory would be compelling direction for future research. Furthermore, determining the exact lower bound in cases where n + is not divisible by a power of 2 remains an open and intriguing problem.
Also, our analysis non-constructive and asymptotic in nature. particular, we not provide explicit rates of approximation or quantitative bounds on the depth required for network to achieve a given precision. As a result, our results establish existence guarantees but leave open the practical question of how deep network must be to approximate a target function within prescribed accuracy. This limitation stands in contrast to constructive approximation results that do provide explicit dependence on approximation error.
Furthermore, our work does not characterize the role of the smoothness of the target function in the approximation behavior. It is natural to expect that smoother functions should be easier to approximate, and indeed prior studies have demonstrated this by analyzing approximation rates in terms of Sobolev smoothness classes (Schmidt-HHeber 2020). Incorporating such smoothness- dependent considerations into the analysis of deep, narrow networks remains an important direction for future work.

---

7 Conclusion
In this study, we investigated the minimum width of deep narrow MLPs required to approximate continuously differentiable functions under the Sobolev norm. Our analysis established optimality broad range of cases. However, our proof techniques rely on the robustness the topological structure under small perturbations in the derivatives of the target functions and therefore do not directly extend to the uniform norm. Nonetheless, the structure the proofs suggests that similar bounds may still hold under the uniform norm. Developing more refined algebraic topological tools to rigorously bridge this gap presents an interesting direction for future research.
Acknowledgments and Disclosure of Funding
This work was supported by National Research Foundation of Korea(NRF) grant funded by the Korea (OV))) TRS-2025-0051525 RS-2024-004061277, and Global University Project grant funded by the GIST in 2025.
References
Aizawa, Y., Kimura, M., and Matsui, K. Universal approximation properties for an odenet and a resnet: Mathematical analysis and numerical experiments. arXiv preprint arXiv:2101.10229, 2020.
Arbel, M., Sutherland, J.J., Binkowski, M., and Gretton, A. On gradient regularizers for mmd gans. Advances in neural information processing systems, 31, 2018.
Biagi, and Bonfiglioli, A. An Introduction to the Geometrical Analysis of Vector Fields: with Applications to Maximum Principles and Lie Groups. World Scientific, 2019.
Cai, Y. Achieve the minimum width of neural networks for universal approximation. The Eleventh International Conference Learning Representations, 2022.
Caponigro, M. Orientation preserving diffeomorphisms and flows of control-affine systems. IFAC 2011.
Proceedings Volumes,
Cybenko, Approximation by superpositions sigmoidal function. Mathematics of control, signals and systems, 2(4):303-314, 1989.
Czamecki,, W. M., Osindero, S., Jaderberg, M., Swirszz7, G., and Pascanu, R. Sobolev training for neural networks. Advances in neural information processing systems, 30, 2017.
Davis, D. Vector fields rp" xrp". Proceedings of the American Mathematical Society, 140(12): 4381-4388, 2012.
Evans, L. Measure theory and fine properties of functions. Routledge, 2018.
Evans, L. C. Partial differential equations, volume 19. American Mathematical Society, 2022.
Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., and Courville, A. C. Improved training of wasserstein gans. Advances neural information processing systems, 30, 2017.
Hanin, B. and Sellke, M. Approximating continuous functions by relu nets of minimal width. arXiv preprint arXiv:1710) 12788, 2017.
Heinonen, J. Lectures on Lipschitz analysis. Number 100. University of Jyväskyla, 2005.
Hirsch, M. W. Differential topology, volume 33. Springer Science & Business Media, 2012.
Hwang, G. Minimum width for deep, narrow mlp: A diffeomorphism and the whitney embedding theorem approach. arXiv preprint arXiv:2308. 15873, 21887,2202 2023.
Johnson, Deep, skinny neural networks are not universal approximators. arXiv preprint arXiv:1810.00390 00393, 2018.
10

---

Kidger, P. and Lyons. T. Universal approximation with deep narrow networks. In Conference on learning theory, pp. 2306 2327. PMLR, 2020.
Kim, N., Min, C., and Park, S. Minimum width for universal approximation using relu networks on compact domain. arXiv preprini arXiv:2309. 10402, 2023,
Kim, N., Min, C.,, and Park, S. Minimum width for universal approximation using relu networks compact domain. In The Tweifth International Conference on Learning Representations, 2024.
Lashof, R. and Smale, S. Self-intersections of immersed manifolds. Journal of Mathematics and Mechanics, pp. 143-157, 1959.
Leshno, M., Lin, V. Y., Pinkus. A., and Schocken, S. Multilayer feedforward networks with nonpolynomial activation function can approximate any function. Neural networks, 666):861-867, 1993.
Li, X. Simultaneous approximations of multivariate functions and their derivatives by neural networks with one hidden layer. Neurocompurting, 12(4):327-343, 1996.
Lin, H. and Jegelka, S. Resnet with one-neuron hidden layers is universal approximator. Advances in neural information processing systems, 31, 2018.
Miller, J. G. Self-intersections of some immersed manifolds. Transactions of the American Mathe- matical Society, 136:329-338, 1969.
Palais, R. Extending diffeomorphisms. Proceedings of the American Mathematical Society, 11(2): 274-277, 1960.
Park, S., Yun, C., Lee, J., and Shin, J. Minimum width for universal approximation. arXiv preprint arXiv:2006. 08859, 2020.
Phillips, A. Submersions of open manifolds. Topology, 6(2):171-206, 1967.
Raissi, M., Perdikaris, P., and Karniadakis, G. E. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-707, 2019.
Schmiet-Hieber.. J. Nonparametric regression using deep neural networks with relu activation function. 2020.
Tabuada, P. and Gharesifard, Universal approximation power of deep residual neural networks through the lens of control. IEEE Transactions Automatic Control, 2022.

---

A Definitions and Notations
A.1 Sets of Neural Networks
For set of activation functions E, the set of MLPs denoted by A do.d is defined as:
No dy = {f Rd + Rdv
W; E Aff 9s E Ed, f WN 9N-1 o... 91 Wi]
Note that, in general, an MLP can have different activation functions in each layer. If the set Σ is singleton, i.e., {o we omit the set notation and simply write:
=W do do..d....d
N da doo
(46)
We define the set of deep, narrow MLPs with input dimension n, output dimension m, and at most intermediate dimensions as:
U
U
N dx.. dx
(47)
m,
NENo
A.2 Some Definitions from Differential Geometry
Definition A.1 (Diffeomorphism). For natural numbers d, r E N and open sets U1, U2 C R function f Un - U2 is C" Cr-diffeomorphhism if and only it is bijective, r-times continuously differentiable, and its inverse f-1 is isr--i r-limes continuously differentiable.
Definition A.2 (Immersion). Let M and N be smooth manifolds, and let f M - N be a C--map. The map f is called an immersion for every point E M, the differential
dlp T.M + T
(48)
is injective.
Definition A.3 (Submersion). Let M and be smooth manifolds, and let f : ::M + N be a C* map. The map f is called submersion if. for every point M, the differential
dfp T.M N
(49)
surjective.
Definition A.4 (Embedding). Let M and N be smooth manifolds, and let f :-M- M N be a C² -m-ma.. The map f is called an embedding if it is an immersion and a homeomorphism onto its image f(M). where I [(M) is equipped with the subspace topology from N.
Definition A.5 (Transversality). Let M, N, P be smooth manifolds and let f: - P. g: P be smooth maps. We say that and are transverse (written t g) for every pair of points p M, q E N with (p) = g(q), we have
(f)(T,M) + (Tg(T,N) = TT
(50)
That is, the images of the differentials p and q together span the tangent space of P at f(p) g(g)-
B Practical Lemmas
In this section, we present several useful lemmas that are employed throughout the paper. The composition of functions is addressed by the following lemma.
Lemma B.I. Let - in the loc (TEm; R') topology and gi 9 in the w/ (R"; topology. Then fi o gs + f og in the W.1.00
lnc. (R"; R!) topology.
Proof. It is sufficient to prove that, for each V E R." the Lipschitz constant of fog ft on V converges to zero as i increases. Choose a sufficiently large number in such that for any 2 io. we have [1g gil [ (V:Rm) < 1. Then,
Cv foog) < Lv Lv ...offfff....... f
S LG gi) Lo(V))+mmm)( (1) i0000 0, (51)
where g(V) B1) is the Minkowski sums.
12

---

This lemma implies that if each function can be approximated by neural networks in the local Sobolev topology, then their composition can also be approximated in the same topology.
We can apply a partial activation function using the following lemma.
Lemma B.2. For natural numbers n, m, w N, and an activation functiooo a satisfying Condition the following relation holds:
loc
A
(o.id) n.M,w
(52)
R2 77, W
Proof. For each f E m. can be represented as:
(o.id)
= Pu,m ogo gn,ws
(53)
(o. where E A w.w. By the definition of 1,w,,0 there exists a natural number E N such that the following equation holds:
WN N IN-1 0... o W1,
(54)
where W, E Aff 95 E fo, id}' for each E [I, By Lemma B.1 We, 9: E A loc
for
lor
loc Obviously,
each E [L, NN. the composition is also in Ag www. again, leading to f E A
n,m.w
W; Ag 1,1000 loc and it is sufficient to prove that A w,,0,
lor
3 {or id) For g E fa, id) consider
C [1, wIN such that (x) = (x) if i E I and (z(z) E IR and R such that o' (aa) + 0 and
if i & I. By Condition I there exists C² function in (a 6, + t). For an arbitrary
precompact set V E IR and sufficiently large M so that M C (a 6, + for any E V. the following relation holds:
M (a (a+ + -()a)a))
M M+000 0.
(55)
('(a)
(V)
M(ola+ (o(a++)-oo(m) (aaa
loc Define fa E Nf,1..
Because
11,11 the identity function
+ E N1,111
(xx) M(x) M(o(a+ a'(a)
if E
(56)
if $ 1
and concatenation M E No w.w as (x) = pact V E R.
(m(w)) Then, for arbitrary precom-
l|g - ام 1w
M+ 0.
(57)
loc and this completes the proof. Therefore, A w0000n
C Proof of Theorem
C.1 Main Proof of Theorem 4.1
The theorem is proved using the following lemma, which states that any continuously differentiable function can be approximated by two-layer neural network.
Lemma C.1 (Theorem 2.1. of Li (996)). Let K be compact subset of R 2 1, and f Cmm (-) (K)) where in Ct (IR, where n max { [lm.]
for 1 < S q. Also, let be any non-polynomial functiioo function 1 < i
S q). Thenfor any e > O, there a network
N(x)
Σ Go x+0)), EER,
(58)
1=
where &i E R. w; E Re, and 0; E RR, such that
p D <,, k N for some iiiiia
(59)
The following two lemmas state that any arbitrary increasing function can be approximated using Leaky-ReLL ReLL and Leaky-ReLU-Aa activation functions, respectively.
13

---

Lemma C.2 (Increasing Functions to Leaky-ReLU), For any increasing C' function f.
loc
f AR 1,1,1
(60)
The proof of Lemma C.2 provided in Appendix C.2
Lemma C.3. Let Σ {a B R2 be of Leaky- acfivation functions. Then, for any increasing C² function f.
loc
(61)
The proof of Lemma Corollary C.4 (Generalization of Activation). For natural number d N and any increasing, C activation f p. the following relation holds;
is provided in Appendix
C.3 The two lemmas yield the following corollary.
Daddd C a,dd
loc.
(62)
where a is the Leaky-ReLU or a set of Beakk-ReLU--ikk activation functions.
The following lemma a technical result used to approximate a vector field with deep narrow MLPs.
Lemma C.5. For t, R. and w E RR,, define ft Rd Rdd as:
f (FI...., 2a)+ (I1....., Zd tanh(( x x+ b)), Let be the Leaky-ReLU positive real number E R.,, such that, for [t] < 8, the following relation holds:
(63)
of Leaky-ReLU-lik activation functions. Then, there exists
loc
fo A d,d.d
(64)
The proof of Lemma C.5 is provided in Appendix C.4 The following lemma states that any smooth diffeomorphism can be approximated by flows generated by (timeddependent) vector fields. The definition of vector field is as follows:
Definition C.6 (Flow of Vector Field), Let Rd x R + Rd be function that is Lipschitz continuous with respect to and a piecewise continuous with respect For each A, consider ODE system
+(t) = f(x(1),t),
(65)
where x : IR Rd. We define a flow map $5.² IRd Rd,, corresponding to f as follows:
of (t)) + +
(66)
For t = 0, we omit and just denote as 07:
(67)
We define the maximal domain M2 C. Rdd x R as the set which satisfies (x, E M, if and only if also well known that if f is Ch -function with respect to x and then (x) also C -f-ffnctioon with respect to x and t.
the solution (f(xx) is well-defined. It is well known that M2 is an open
(See Theorem B.41 of Biagi & Bonfiglioli (2019) for example.)
When we consider Df, we only consider a Jacobian with respect to x::
Df t) = Dxf((xtt..
(68)
Lemma C.7 (Theorem of Caponigro 2011)). Any orientation preserving diffeomorphism can be represented by a flow map: For any diffeomorphism f E >>> (RR) with det(Df) > 0, there exists
flow map d'p generated by ODE system à = F(x, t) with smooth vector field F Rd R - Bd such that the following equation holds:
s= oz
(69)
If two vector fields close, then the flows they generate also close.
14

---

Lemma C.8. Consider C functions fi, : R R R.. Define two ODE systems i = f((2, t)
for = 2.2 and let o ;= o' f be a flow map defined by each fú. Assume that V x 10, C Morfor precompact set V E TRd and T E 4. Define V as
v {2) E R I [0,T], + Bx(1). Then, for any c E R4, there exists positive number ò E (0,1) such that if
(70)
[lf( - ff((, (6.tt) (e (V: < 6,
(71)
for all t [0, T]. then,
|0 - 25 wi. <c
(72)
The proof of Lemma C.8 is provided in Appendix C.5
Now, we approximate a wwo-layered-MLL--. vector field using deep narrow MLPs.
Lemma C.9. For i NI, and C',, -funccttos U Rd x
R + R let
v(x,t) = Σ (z,
(73)
For a real number T E R. and a precompact set U E Rd, assume that x 10, C MG.
Consider n E NN tt == At = +,
fi,k 1 + (x,
(74)
To = JNN o JN o ..... S1,k3
(75)
and
S = T3 ..... T1.
(76)
Then, there exists a natural number no E N such that if n 2 no, the following relation holds:
100 S, flll.. < e.
(77)
The proof of Lemma C.9 is provided in Appendix C.6
By combining all the lemmas, we prove the theorem.
Proof of Theorem 4.1 By Theorem 2.7, p.50 in Hirsch 2012), we only have to consider D (IRd).
f is an orientation reversing diffeomorphism, o f is orientation preserving where g E A LR defined as:
d,d,d is
g (21, ma) + (-I1, 22, 23.
(78)
Therefore, only consider orientation preserving diffeomorphism D (R.). Consider
arbitrary precompact set V @ gd and R_. By Lemma C.7 there exists ODE flow induced by F E C (Rd x R; Rd) such that
/= oz
(79)
By Lemma C.8 there exists 8 E R, and V e Rd such that if [F(,tt) - F2(-, by (U,R)) <
for all E [0. 1]. then, ||0] 11 (V < Consider compact set K such that V C K. By
Lemma C.I there exists F2 Rdd RR + Rd such that
|F
DF DFalll= <
(80)
where F2 is represented as
N F2 Σ cap(w; x att b;),
(81)
where a; by, C2 E IR, and w, E Rdd Here, tanh if a is Leaky- or Leaky-ReLU-like, and equals if an activation function satisfying Condition [ As both F and F2 are C! functions,
|| [[F())) Fal: t) (V.Re) 8,
(82)
15

---

for all & E r). Thus, 110b - (bg.lwi < 2. Then, by Lemma C.9 number no so that if 2 no. then, for is == Kr At /.
there exists natural
fi : x + x + Atesp(u; x + attx + b.), Tk == Jn,k o fn- o ..... fi.es
(83)
(84)
and
Sk = o 0...0. Tı.
(85)
the following inequality holds:
10t Sullwi. (V.R.)
(86)
For the Leaky- ReLU or Leaky- like o, by Lemma
C.5 there exists i E [1, & E [1, nl, and
loc
6G,k E 12+ such that if |4| < 6,, then, Je, E A d,d.d
Choose sufficiently large n so that | < 6
for all, each fa, Ad, loc Then, Sn, E A d,d.d
loc
_For satisfying Condition
1
fL. E Ag d,d,++1
thus, S, E 2q loe Thus, 110b the proof.
Sn (xll < for Sn E A d,d,d+((o)
loc This completes
C.2 Proof of Lemma
Proof. 111,1 is the set of strictly increasing piecewise linear functions with finite segments. Consider any increasing C' function
L.R.
RR R, compact set K
C R. and positive real number E R...
We will construct a function o E U such that ||o - fll fllw < e. Consider a closed interval [a, b] 5 K.
Then, there exists a natural number n E N such that [](x)) - (v) 7 and [D]((x) - DJ(v) < i
for - y J. Define f U as a piecewise linear function with breaking points a 16-a)i/n for 0 < i < n, which has the same values with in each breaking point. For all x E K and the
closest breaking point y E [a, bj,
x yl c. Then,
(a(x)) fx)) < lo(x) - [v)] - lo(y) - f(v) + (f(x) - f(x)) < 2'
(87)
And for two adjacent breaking points yo: 1/1 such that x [yo- vi].
-Df(=)-D(<>)) - Do(()) = f - f(y)) Do(x) = DDf(c) - Df(x)) < 6
(88)
y1 y/o
for a E (Yo. y1) by mean value theorem, almost everywhere. Therefore, ()a( (())) < C. lor
Because the selection of compact set K C IR is arbitrary, 1,1,1 and this completes the proof.
C.3 Proof of Lemma
Proof. As strictly increasing C¹ functions are dense in the set of increasing C¹ functions in C topology, we only need to approximate a strictly increasing C' function f. Consider an arbitrarily small error R, and open interval (a, b). It is sufficient to prove that there exists g A ii,1.1 such that
|l ا (l...((,,b)))) <. Define L1, L2 E I4, as uniquely determined value as follows:
(89)
[41,2] = (Df(x) E R. x E 6].
(90)
Define b: R, + R as
b(8) sup Do - -1 8-1 0
(91)
Choose a sufficiently small e' E R4 so that (6L2 + 4)b(1+e')++
< e. There exists a natural number
N N such that if ||æ - y/1 < N. then, [l. f(y)l < 1 and [Df(x) - Df(())) min (1,)). Define & as a piecewise linear function with breaking points a; a + (b - aaj/N for 0 < M N, which has the same values with a in each breaking point. Then,
||
h hww. ((a.b)))
(22)
16

---

Now, it is sufficient to prove that there exists function / E 1T,,, such that
||/ h (Wi.x(((,,))))) <.
(93)
Define % as
-0441)- - [(ai)
(94)
0:+1 03
so that T. be the slope of ft in (as, (441). We use mathematical induction on n to prove the following: There exists fn.m Ai,1, such that
1. | fn.m (.a.a1)((a m0 0,
2. there exists natural number M such that if m M, then, || D
2.
3. and, for an arbitrary d E (0, +), I1.mmmm - (In (x - an+1) + (aaaa///)a///)////sssssaaa/aaaa
For n = 0, there nothing to prove. Assume that the induction hypothesis is satisfied for n. Define
fn+1. E 1i,11 as
(m(Jn,m(2) [(n+11))) + f(aat1)-
fntim (1) = 7n+1
(95)
Yn
As
d (m))
m=+00 (x) in C -topology.
m
LR
(h f(an+1))) + fan+11
fn+1m
1n+1 m-+00
7/n
m
fh.
in (a, an+1)
(n -mmtil 0n++1) + (an+1)
in (aa+1, = 7at1 (2--22) + [(x+2) (96)
with CO -opooloyy Now, it is sufficient to prove that the derivate-related assumptions. Dfuti, be calculated as
Dfn+1m(()) 7nt1 Do ou (m(fn,m(z) - -f ((nn+1)))DDnmml Tot1
(97)
Then, for any ò E 1 and [a, On+1 0).
sup #E =Eja,an+11 -8
- D (x)
sup
(DSmm())) (x))l
1 Tn-1 0 7m (m (m(fnm(x) - [(an+)))))))) m=
(98)
xE(a,ant11h
And, for x E [an+1 + b], limm sup +8,0) DD 7x 0, and therefore.
lim
sup #660
DDf+im (x) 7a+1l
m=+00
lim
sup
7n+1 Do m (m(fn, f(an+1))) 7n41. (99)
#E ea +6,6
Therefore, the induction hypothesis 3 is satisfied. As h(x)) = 7n+1( - an+2) + f(an+2) for E (n42)
Dh Df Dw ((a +d.a.+a))
m= 0.
(100)
Now, it remains prove that there exists a natural number M' such that if > M'. then D| Df I ((0--111- < c. Choose sufficiently large M' so that if m 2 M then,
sup =E(ant1-8.0
[.D.f.mm - Yall S min 1).
(101)
17

---

Then, for z E (an+1 - 0,a + +0),
Ih+11
Df
76411 7i+1
Do 22 (m(fn,((x) ((ant))))Dfmm(a T+t1
7
7+41 [D
Do
=i (m(fn,m(x)) - - f(an+1))) - - 1/1 (1t2 N+t1
(fum(())
7
7141
7
7i+1 [15mm (1)]lb
71t1 ) + 7/t1 76-1 D Dm (2))
Y
%
$
e') + LLge' s (3L2 2)b(1 e)
2' (102)
and
1l -
& lh4+1 Dfa+im - ~ < (00(2 < <σ (103)
6
Therefore, for x E (on+1 0,0 + 6),
[Db(()) Dintimm<)) <
(104)
By mathematical induction, we conclude that there exists fN.m E Ai,11 and M such that
m 2 M, then,
|/b
<e.
(105)
This completes the proof.
C.4 Proof of Lemma C.5
Proof. For w = (Wi.... wa), if w; = 0 for i E [1,d - 1]s, the last term of fe can be calculated as:
Fd + t tanhh (w + b).
(106)
For sufficiently small 8 E R, and |t| < 8, this function is increasing. Thus, by Corollary C.4 the
following relations holds:
A d,d,d
(107)
d,d
Also, the following relation holds:
A (tanh-",id) d,d.d
(tanh.dd
C Ag
loc
(108)
d,d.d
d.d.d
Now assume that there exists i E [1, 1]N such that w; + 0. Further, without loss of generality, assume that ti * Then, the following functions are elements of 2g
loc
fi v.) + (w b,r2,.... xa),
(109)
fa (Zi,..., x2) + (tanh(z1). (2)), ... zd).
(110)
fa (Zi...., + (ZI...., Td + tx1),
(111)
fa (Ii...., + (tanh (amnh~((z)) (z x2 ......, Id).
(112)
fs (Z1,..., + (21 + wat tanh ( (3 F2, .. Id))
(113)
and
T1 (I--waTs warF - W2.d I2. .2,...,
fo xa) +
(114)
W1
Then, the composition
fa f3 f2 becomes
fa f2 fi (2) (I Fa xa + ttanh(w x + 6)).
(115)
18

---

C.5 Proof of Lemma C.8
Proof. Let L and L' be Lipschitz constant of fi and Df1 with respect to in V. respectively. Restrict ò to
8 < min (1.2TeL7)
1
(116)
We first prove that, for all x E V and E [0. 02 V. Define T as
T T {te [0, V x (t) c M, and x2(x) V for forall V
(117)
Obviously, E T.
2. And T is an open set relative to r]: Assume that T. Then, as x5(x) E V for all x E V, and M and V is open, there exist 61,a, 22 R+ such that if lly - x < 61 choose finite cover
and It' S 62,e, then, o5 (y) V. Because V is compact, [[=} B)(e),)))=es of V. Then, [t, + minges 62,z) C. T, and T becomes an open set.
3. T is closed relative to [0, r|: Assume that T = [0, t) for t (0, It is sufficient to prove that
<<(x) = xx+ 1/ fe(05(z),
(118)
is finite and in V. Define e(x, t) as
(xxt):= -b(())
(119)
Then, the following equation holds:
e(x, 1) = S/' [((0(())))) - f2(05(x), sdds.
(120)
Then, as (f(x), V for s E (0, the following inequalities hold:
le(e, 5 [
s)s + /''
L [etez s)1l dds S öt + T, J [e(a,,))lds s st (121)
where the last inequality is by Gronwall's inequality. As St 8te <1,
(z)== = xi + te(z,t) eV,
(122)
for all V. which leads to E
4. We conclude that T' = =[0,7].
Next, prove that []e(x, f)ll can be bounded. It is already proven by setting 8 29c Then, [|e(,, ()]l < 5
Finally, we will prove that [Dee |) can be bounded.
De(x, t) = / Df(()(e). -D1A()()))))))))( ())(())
(123)
19

---

Then.
(D((, ( (DDA
% (D(((( -
+
+ * (e0lll...DDDtt-----dddeed.dpddd)
- (
$ J/ L L21le(z, s)(l 7L L De !' LL'öse + - LL LL'I)ee (ldds
/ [Deea l)dss S (L
S 5 (Id 1) + 8 8T. + LT!
1) +1 L() eLL't (124)
where the last inequality is by Gronwall's inequality again. By setting sufficiently small 8, we get t)ll + [[ee(,))) t)ll < e,
(125)
for all x E V and & [0, This completes the proof.
C.6 Proof of Lemma C.9
Proof. Define v c Rd and V/ c Rd as
VO = [x)
(126)
and
v = ((x,t) ER x V
(127)
As O is compact, there exists a positive number 5 E R. such that
V = (V2 + Ba())) x 10, 10--0-1 E
(128)
for all t E [0, Define V c Rd R as
V := RR
(129)
We will conduct all our discussions on V where of is well-defined. Denote the supremum and the Lipschitz constant of e in V as C and L. respectively. Also, denote the supremum and the Lipschitz constant (as operator norm) of De in V as C" and L'.
In this proof, will a big-O notation with respect to At; that is, a function f R - denoted as
f= 0 OAtt)
(130)
if and only if
!s(a\) < cdi'
(131)
where c is constant independent of At and polynomially dependent N, L, will check that
Ioo Te+1 1 (v) (A
(132)
We define ULA E C¹ (RR,, as
ULIiZ At (x,
(133)
And define Uk as
Up = UNA-
(134)
Then, is sufficient to bound two terms:
Jf Un++ 2~ (v)
and
|| Un+ 11l L~ 11
(Vo,
(135)
20

---

The first term can be calculated as
ARl2
| - -
-U++1
e ( s) ds - Atu(;
1.x (V
-11//mm u -0) - -oo,((()ds (x) La (M))
(/ |10 (0)) u(z, (2) ds
Aut1
I sup =eV,, bx
St sup sup =e v., Nx+1]
1 (o .)) v(x, <C(A)) (cLa0 + LD) (136)
where the last equality is by the following arguments: for any hs and x E Vk.
(z) - x||| = =11%% o(0h (1), ( -1 ((o))((0;; - u((, + o( vlx,r()d r)dr
I (x) z||dr + C(s - (t) s C(s tt)e & CAte (137)
S L t/
where the second last inequality is by Gronwall's inequality. Therefore,
a| (0 ,8) v(r, 4x) & CT + LLA,
(138)
and the bound is independent of k. To calculate the second term [[T441 UK+1 m :d) for I N],, define Ti C¹ (R4:R) as
Tix == fi, 0.. fiAt-
(139)
Then, TN, Tk. We inductively bound
|T U UL.KllL- (V
(140)
When 1, Tı, = U1 fı,, and there is nothing to prove. Assume that the above induction hypothesis is satisfied for 1. Then,
T+11,(x) fLI1 Ti,(2)) Ti,(()) + (TV))
(141)
Therefore,
||+ U141,111 10 S sup =EVi-11
[[T+1,,())) -1i1)()))
sup =EV4,11
[T] + (T, (U),(()) (t, II
sup =EVi=11
[TT,,(x)) A1
(x,
$ (At) CNL. (142)
Therefore,
[Te UR UKIL R4) $ (At)² CN
(143)
And thus,
1outte Tht1 _ (V)) (At)) (CN2L+L+e* = c1 (At)2
(144)
Now, define eg : IRd R as
Ek e= 2b St.
(145)
We restrict At sufficiently small so that
eLt 1 ci (At) min ():
(146)
21

---

Under this assumption, we use the mathematical induction on l to prove that Sx (z) E Vix for an arbitrary & E [1. mly. It is obvious when k 0. Assume that the induction hypothesis satisfied for k == ko. For E U and k S ko.
(z)+1(2)]l = [1 - (x)]l = 110 ot (x) - Th+1 o Sa(z)]
$ llotetatts o o.t (z) 2irite+s o Sx + 1 Sx(2) Tk+1 o (x(x)
S L (0t [|(x())]] + 11 -k++tlll ( < LAtt)le (x)) Atc (At).
(147)
Then, for any k S I
A7
|lex = (1 + ( + (1+ LAr)* -1 AlC S elr el
L a (Ai) 2 (148)
LAt
As (1)) T + ex+1(x) E VAN the induction hypothesis is satisfied. Also, (()=)I=(UURR)) < 2'
Now, we bound Dest First, we bound derivative D ( (p. Ta). For arbitrary s, E [0, and E V2, consider the following equation.
(5)(x) --) ((((00(0)))))
(149)
Apply derivative to both sides, and we get
-1, (c) (z) $ S L' (x)l
[ID)t'()) Iall = -all=
S T. T' (x) - Tall + dI' S dI. (t - s)e s dI'(t - s)el', (150)
where the last inequality is by the Gronwall's inequality. Denote the last constant as :' == dl''t'r;
that is,
(Dot'(s) - Tall s (t s).
(151)
Calculate the Lipschitz constant of De
fx41 For 8, E [1x, (4+1) I. and x, E Vee.
s 11, (o))) - f. (v), (v) (v))
|| Dot - Dot.
S. (o(o (x)dr Du( (u(d) (s)
+ |1.
Do( r (w)
S + (t -)) I' [D))"(<x) (v)"(()) dr + 2T' / (x))*(x) - -5 (v)ldrr
(1 + L'(t - -s] L' |Do (z) Do (v) dr + 2 vll
S 2T/ w ((((()))))))) < AL'C vlle?, (152)
where the second last inequality is by Gronwall's ' inequality. Denote the last constant as L/2 := eL' that is.
(x)*(*) Dø [v)] s LaAt|Iæ 1allz-vll.. -vll-
(153)
Now we calculate the followings:
D DDogeAAk Akil DTx+
= (V) OAL))
(154)
22

---

Do (x) DUk+1(()) - 1
-111 De (x), (z) - Du(r, ts)ds
Ak-1 Du (z)) Do(x, ds (ltt |D (x), (D (x) Ia)l ds
1
0 (A(?). (155)
We the mathematical induction on I to prove that
| -UUL (Vo,_, = =
(156)
When / 1, Ti. U1. fi,k, and there is nothing to prove. Assume that the above induction hypothesis is satisfied for l. Then,
DT++1A(() Dfut) = (Ta AtDu (7 (157)
Therefore.
|Di++I, DULIIMIL 200 S sup
(x) DU1+IA
=eVi-11
< sup
II(Ta AtDut41 (TIA.()) (111 ((DU,)))) + AtD f (x-1)))
< sup |TTi, DU + At (H (11 (x.,t -EViz-1
=AtA (158)
Therefore, the induction hypothesis is satisfied. For any E U and k; we have
|Ee+ (x)ll = D (D (z)) (x) (4))
||
(6th Do) (x) - Dot (6. (z))
+ Mooty-tu120 (x) (S{)) (S(r))
+ ((.
(1599
(159)
For the first term, we have
Dotpte (0'' (z') (x) - Do tx+1 (ot (x)]
$ D (x) DS*(2))) (x)]l + (D (6 (x:) - Ta) (Do - (>)(1))
$ (1 + (ex (160)
For the second term, there exists a constant 02 E R4 satisfying
- Defte (4 (S)))t
Lv,, (Dole.t+t) [IDS < [= < (x)+ (161)
For the last term, have
Dobl (S. - (=0)) =
(162)
Then, by selecting a sufficiently small At, we have
[[Dett1 (x)]l (1 IAA)) (x]]]
$ (1 (e++ +o < cliT-1
< c
es
(A()
2' (163)
for a constant Cg E RR,, We conclude that
[|]++|()||| + De(+1(()) <e,
<<,
(164)
and this completes the proof.
23

---

D Proofs of Approximation Lemmas
D.1 Proof of Theorem 4.4
Proof. Consider a function f (R"; and precompact set E R It sufficient to prove that, for any E 4,,, there exists function f E A (..m..nn.m)+aad such that f (V:Rm) < 6. Because A H.m. is closed under affine transformation composition, we only need to consider V satisfying V (0, 1)". By the definition of [(n,, m), for any E RR,, there exists embedding 9 E Emb((0, 1|", RR such that
|l P o (10)=(0).)):.R))) <
(165)
Because (2,, m) 2 n, by Lemma 4.2 for qn (n,mm) : (x1,.....00 In) (x1,.... En. O.... 0), there
exists smooth diffeomorphism G such that = G ((,,m) By Theorem 4.1 there exists an MLP H E A (00
such that
6 ||G - -10pp <
(166)
2
Then,
PQ(m.)).m Ho 9n,((mmm)) Pf(m.m). o G mm..
(167)
Therefore,
|| P H 06 <
(168)
P H. (n,9(m,m) E Ag N.m This completes the proof.
D.2 Proof of Proposition 4.5
Proof. For non-decreasing C activation function a, there exist smooth, strictly increasing activation loc
functions on that converge to a in Wr topology. Therefore, Ag,,d
loc C Ad,, (T.]
making
sufficient to consider only smooth, strictly increasing activation function a.
For f E Ag -m it can be decomposed as:
= Pf(n.m) 1,m o o o Qn.0(n,m)-1:
(169)
where 9
Ag (ta.))-L000oam--RR.....
As -2mm
loc
9 91. E Emb((0, 1)"h 1 1oc Therefore,
Doc (1)))
have:
f110111 Pi(x,))-1,m Emb((0, 1)", RR((.)) loc
(170)
and as the selection of f f A (m is arbitrary, we get the following:
Emb((0, 100m) 1oc
Ag Ag ( (0.13m
C P 1.m
(171)
As ((n, - < ((n,m) by the definition of ((n,m):
Pmmm.m)-..
1)", R.((..))1)) 2 C¹ ((,
(172)
and thus,
A Ni.. (0.1)m
(0,1))))
(173)
Therefore, we have CL(RR., Rm) g A n.m.A(mmm)-1
loc
This completes the proof.
24

---

E Proofs of Topological Lemmas
E.1 Proof of Lemma 5.5
Proof. Define F:
as
([), = (a, (f))) + a) - g(v)))
(174)
Then F is a proper submersion for sufficiently small Then, DF z, y) can be calculated as
1 DF (()) - (())) g(v)) aDJ(x) (1- Dg(r)) -aDS(y) Dg(u))
0
(175)
Consider vector field X, in (-6, 8) U U for [1,m 1|N which satisfy the following:
(DF)X; Ci.
(176)
where &ç the i-th coordinate vector. Then, define G F (RRmt)) - Rm+ x F-) as
G(z) = (F(z), 2x X
F())m11
Øx,
F(())).
(177)
Then, G has inverse G-1; : R x F (0) + F (Rutl) which can be calculated as
am+1 G G 12,1,,,2.....11. f 1 2) x x m1 (a)),
(178)
for x E IR and x E F- (0). Then, for the projection Rm+ x F (0) R=+ the following equation holds:
F=FoGT
(179)
Therefore, F (ct is diffeomorphic to (c2) for c1, C2 E R!*!
Note that the above diffeomorphism G can be defined for all X; that satisfy Equation 176
We set X1 as
X1 = (DF) (DF(DF)))"*e E1
(180)
Then, ox, the diffeomorphism between (0) = [0] (0)x5-1(0) and F (1)0,))==1))000000))))))
Let X, be represented as
XI(a,I,,) Mi
(181)
MM
is enough to prove that (a y, x) = M2(a, x, 11) Let
A(x,y) f(x) f (g(x) - 9(1)),,
(182)
and
B(a, B(x) == aD((x) + + -1- (gg())
(183)
Then, A(y,z) A(x
(FF)TDF can be represented as
AT DF(DF)== A AA" B()
[1
(184)
Then.
+ B(E + - A
+ + B(y T
(DF(DF)))- =
- B((() + B(y))(u)) A
(B(c)B(r) + B())
(185)
X, = (i x. v) M2(a, x.
= (DF)T (DF(DF)T) 61 =
B(c)) T - B(y)B(y)T)-1A B(y)(B())B()) + B
(186)
Mi (a. = M2(a, x. y). And this completes the proof.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
No justification
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
No justification
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each thcoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
26

---

Justification:
No justification
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [NA]
Justification: The paper does not include experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
27

---

Answer: [NA]
Justification:
The paper does not include experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ctiipe.ccppulli//uuiii http:/uwdiesseeee CodeeSubeeiieeeee ssion ceiiionPoicyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper speciffy all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [NA]
Justification: The paper does not include experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: The paper does not include experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
28

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [NA]
Justification: The paper does not include experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics /tttui://euurpss..
Answer:
[Yes]
Justification: No justification
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e-g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, haaerssithcoode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
30

---

Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
31

---

The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non non standadd components.
Please refer to our LLM policy hattps://neurips... 200ps//ee///0ooeeeeeee/2/22///// for what should or should not be described.
32


Reviews:
[{"review_id": "Y4LVv5cLZk", "reviewer": "Reviewer_LpbM", "summary": "This paper studies the minimal width required for an MLP to be universal for functions from $\\mathbb{R}^n \\rightarrow \\mathbb{R}^m$. The authors improve upon existing results in two ways. First, they consider universality with respect to the $C^1$-norm. Second, they use tools from algebraic topology to obtain new upper and lower bounds for a variety of values of $n$ and $m$.", "questions": "In Theorem 4.1, when you say that $\\sigma = LR$, you mean that the activation function can be chosen to be a Leaky ReLU with a different value of $\\beta$ at each layer, right? Otherwise, if $\\sigma$ is a fixed Leaky ReLU activation function this bound appears to contradict the lower bound proved in [Johnson 2018]. This led to a bit of confusion while reading.\n\nHas the quantity $\\Omega(n,m)$ been directly studied in the algebraic topology literature before? I see that numerous related problems have been extensively studied. I think it might be nice to include a table with all of the known results and open problems related to $\\Omega(n,m)$.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: The problem of determining the minimal width required for universality is an interesting and important theoretical problem. I believe the authors have made significant progress on this problem by relating it to a quantity, which they call $\\Omega(n,m)$, that appears to be a natural object of study in algebraic topology. The authors also extend existing results by considering universality with respect to the $C^1$-norm instead of just $L_p$-norms.\n\nWeaknesses: Some of the writing is a bit unclear. For example, the authors consider neural networks where the activation layer in each layer can change within a class of activation functions (such as the leaky ReLU). I personally think it would be more clear if the results focused on the case where the activation function is fixed. The varying activation function considered does not seem to be essential and it makes the paper a bit harder to follow.", "was_revised": true, "final_justification": "This is an interesting theoretical paper. The authors have answered all my questions and I have no concerns. I stick by my assessment of accept.", "rebuttal": "**Regarding the varying activation functions)** Thank you for the valuable advice.\nIn fact, the use of varying activation functions was primarily motivated by the elegance of Equation (33) in Theorem 4.6:\n$$w_{\\min}^{W^{1,\\infty}}(n, m, \\sigma) = \\Omega(n, m)\n$$\nWe believe that, for certain activation functions, a set of varying activation functions could potentially be replaced by a single fixed activation function. However, we have not been able to find a concrete example demonstrating this. We consider this an interesting direction for future research.\nWe will also improve the writing related to the activation function to better convey the content.\n\n**Regarding the contradiction to [Johnson2018])** Yes, in the case of $\\sigma = \\mathrm{LR}$, the activation function can be a Leaky ReLU with a different value of $\\beta$ at each layer. However, this does not imply a contradiction with the result of [Johnson 2018], even when the activation function is fixed. This is because Theorem 4.1 concerns the approximation of diffeomorphisms specifically, rather than arbitrary continuous functions.\n\n**Algebraic Topology Literature)** To the best of our knowledge, there is no existing reference that explicitly defines the quantity $\\Omega$. However, several studies offer valuable insights that contribute to understanding or computing the value of $\\Omega$. We have made an effort to compile and list all known values of $\\Omega$ in the paper, based on our current understanding of geometry and topology."}, {"review_id": "079RqXOhqj", "reviewer": "Reviewer_5C4H", "summary": "This paper studies the minimum width $w_{\\min}$ for which the class of deep, wide networks (i.e. neural networks consisting of any number of hidden layers, each of width at most $w_{\\min}$) can universally approximate $C^1$ functions in the $W^{1,\\infty}$ Sobolev norm. Using tools from control theory and differential geometry, the authors demonstrate an upper bound on $w_{\\min}$. This upper bound is also shown to be optimal (i.e., a matching lower bound exists) for some combinations of input dimension $n$ and output dimension $m$.", "questions": "- I am confused about the domains of approximation on which the result holds. What does it mean for MLPs to be able to universally approximate $C^1(\\mathbb{R}^n; \\mathbb{R}^m)$ functions with respect to $W_{\\textrm{loc}}^{1,\\infty}(\\mathbb{R}^n; \\mathbb{R}^m)$? The local Sobolev space $W_{\\textrm{loc}}^{1,\\infty}(U; \\mathbb{R}^m)$ in the paper is only defined for compact domains $U \\subset \\mathbb{R}^n$. Is the idea that any $C^1(\\mathbb{R}^n; \\mathbb{R}^m)$ can be universally approximated in the $W_{\\textrm{loc}}^{1,\\infty}(U; \\mathbb{R}^m)$ norm on any compact domain $U \\subset \\mathbb{R}^n$?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- This result is interesting, nontrivial and, to my knowledge, novel. The role of depth in neural network expressiveness remains an important open question, and a more complete understanding of the approximation power of fixed-width, infinite-depth networks seems relevant in addressing this. The consideration of universal approximation of the functions themselves and their derivatives also seems novel and I can see that this could be relevant for many applications.\n\nWeaknesses:\n- The problem should be motivated more explicitly. At least a few more sentences of the introduction should be devoted to making the case of why the fixed-width, infinite-depth regime is interesting to study, and why controlling approximation of derivatives is interesting (are there particular applications where this is of interest? Does derivative control imply anything about generalization? etc.)\n- The presentation/exposition in the paper could be improved. Some specific comments:\n    - Some of the notation throughout seems cumbersome and not especially necessary. For instance, would be clearer and easier if $\\nabla_{d_x, d_y, n}^{\\Sigma}$ were just defined in words as \"the set of MLPs with activation function(s) $\\Sigma$ of arbitrary depth and hidden layer width at most $m$.\" Then the symbolic definition in equation (16) and the definition of $\\mathcal{N}_{d_0, \\dots, d_N}^\\Sigma$ can just be dropped.\n    - Regarding notation, it would also be clearer to express the problem statement and the meaning of universal approximation in words rather than using the containment/closure notations. For instance, instead of, or in addition to, defining $w_{\\min}^{W^{1, \\infty}}$ symbolically as in (25), it should be explicitly stated in words that this is the minimum width for which MLPs of this width and arbitrary depth can approximate $C^1$ functions to any accuracy in the $W^{1, \\infty}$ norm. This makes the paper's main result much easier to digest.\n    - The authors should not assume that the reader has much familiarity with concepts from differential geometry. Terms like immersion, submersion, embedding and transversality should be defined in-text in the most elementary way possible (partials of all orders are continuous, Jacobian is full rank, etc.). The formal definition of vector field flow probably doesn't need to go in the main paper; it would be better to motivate it intuitively when describing the proof strategy for Theorem 4.1 and safe the formal definition for the appendix. If any of the concepts can be illustrated with a figure, this would be helpful.", "was_revised": true, "final_justification": "I thank the authors for their response and for their answers to my question about the approximation domain. I maintain my recommendation for acceptance, encouraging the authors to make the revisions/clarifications that we discussed.", "rebuttal": "**Regarding the motivation)**\nThank you for the valuable advice. We will add the following paragraphs to the introduction:\n\n1) *The choice of neural network architecture plays a crucial role in determining performance. However, in practice, architectural decisions are often made through trial and error. It is therefore important to provide theoretical guidance on what should be avoided and how to select appropriate width and depth based on the input space, target function, and specific tasks. In particular, from the perspective of approximation power, identifying the minimum width required for universal approximation is a key question.*\n\n2) *However, there has been a scarce number of papers that study norms involving derivatives of functions in the setting of deep narrow MLPs. Many deep learning techniques directly penalize the difference between the derivative of the target function and that of the network. These include Sobolev Training [1], Physics-Informed Neural Networks (PINNs) [2], and Generative Adversarial Networks with Gradient Penalty [3].*\n\n**Regarding the presentation)**\nThank you for the detailed advice. \n* We will defer the equational definition of deep narrow MLPs and the definition of the set of MLPs to the appendix. \n* We will also incorporate a verbal explanation of $w$, as suggested by the reviewer. \n* Additionally, we will include definitions of immersion, submersion, diffeomorphism, embedding, and transversality in the appendix. \n* The formal definition of the flow of a vector field will be defered to the appendix.\n* We will focus mainly on explaining the concepts or strategies of the proofs, rather than including all detailed proofs.\n\n**Domain of the approximation)**\nThank you for the insightful question. \nIn our setting, $U$ is not assumed to be compact; it is typically an open set, such as $\\mathbb{R}^n$.\nThe space $W_{\\textrm{loc}}^{1,\\infty}(U; \\mathbb{R}^m)$  is defined via local behavior: for a precompact set $V\\Subset U$, we consider the restriction of functions to $V$ and $W^{1,\\infty}(V; \\mathbb{R}^m)$ norm.\n\nIn other words, the two statements are equivalent:\n* $C^1(\\mathbb{R}^n; \\mathbb{R}^m)$ can be universally approximated in the $W_{\\textrm{loc}}^{1,\\infty}(\\mathbb{R}^n; \\mathbb{R}^m)$ norm.\n* For any precompact set $V\\Subset \\mathbb{R}^n$, $C^1(V; \\mathbb{R}^m)$ can be universally approximated in the $W^{1,\\infty}(V; \\mathbb{R}^m)$ norm.\n\nIn short, this is simply a precise way of stating that we can approximate any $C^1$ function in the Sobolev norm on any compact subset.\n\nWe will clarify this point in the revised version of the paper to avoid confusion and to make the domain of approximation more explicit.\n\n**References**\n\n[1] Czarnecki, Wojciech M., et al. \"Sobolev training for neural networks.\" Advances in neural information processing systems 30 (2017).\n\n[2] Raissi, Maziar, Paris Perdikaris, and George E. Karniadakis. \"Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations.\" Journal of Computational physics 378 (2019): 686-707.\n\n[3] Arjovsky, Martin, Soumith Chintala, and Léon Bottou. \"Wasserstein generative adversarial networks.\" International conference on machine learning. PMLR, 2017."}, {"review_id": "oFRMGYGWqB", "reviewer": "Reviewer_8NUv", "summary": "The paper studies the minimum width requirement for deep narrow MLPs to approximate continuous differentiable ($C^1$) functions. It shows (Theorem 5.1) that when the width is at least $\\min(d_x+d_y, \\max(2d_x+1, d_y))$, deep MLPs can approximate any $C^1$ functions under the Sobolev norm. The quantity is also shown to be a tight lower bound when the input and output dimensions $d_x$ and $d_y$ satisfy certain conditions (Theorem 5.9). These bounds together characterize the optimal minimum width for the universal approximation of deep narrow MLPs for a wide variety of input/output dimension configurations.", "questions": "1. The tight lower bound applies to continuously differentiable functions with finite $W^{1,\\infty}$ norm, which is more restrictive than continuous functions with finite Lipschitz norm. Is this additional assumption (differentiability) necessary for the lower bound, or is it just a proof artifact so that you can invoke the results from control theory/differential topology?\n\n2. If we assume higher-order smoothness, i.e., $W^{k,\\infty}$ with $k>1$, is it possible to get a better bound that depends on $s$?", "limitations": "Partially addressed. The authors have mentioned several limitations, e.g., the lower bound only applies to input/output dimensions satisfying the conditions in Theorem 5.9. However, there are some other limitations that are not mentioned: \n\n1. The result is non-constructive and asymptotic in depth, and there is no explicit dependence on the precision, so we do not know the rate of approximation, i.e., how deep we require the network to be to approximate a function, which is important for understanding the extent of universal approximation. There are works on constructive approximation that provide more explicit dependence, while they typically require wider networks [1]. \n\n2. Similar to 1, the result does not characterize the dependence on the smoothness of target functions, which is again crucial for understanding the fine-grained behavior of universal approximation. In principle, one would expect that smoother functions are easier to approximate by MLPs, which is shown in e.g. [2] by considering the order of smoothness (e.g. $k$ in Sobolev $W^{k,\\infty}$).\n\n[1] Yarotsky, Dmitry. \"Error bounds for approximations with deep ReLU networks.\" Neural networks 94 (2017): 103-114.\n\n[2] Schmidt-Hieber, Johannes. \"Nonparametric regression using deep neural networks with ReLU activation function.\" The Annals of Statistics 48.4 (2020): 1875-1897.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength: \n\n1. The paper provides sufficient technical background for readers to digest the main results.\n\n2. The proof based on diffeomorphism is well presented. Converting the problem of approximating continuously differentiable functions to discretizing an ODE by MLP layers is interesting. \n\n3. Showing a tight lower bound matching the upper bound has great theoretical significance. \n\n\nWeaknesses: \n\n1. The results are asymptotic in depth and do not provide explicit dependence on the property of the target function, making the result less significant. See Limitations.\n\n2. There are too many different notations, which could be confusing. For example, $n,m,d,d_x,d_y,d_0, d_N$ are all used for dimensions, sometimes different notations are referring to the same thing. I understand that some notations are for general background definition, but at least Section 1 ($d_x,d_y$) and Section 6 ($n,m$) can be made consistent.\n\ntypo: \n\nline 276: redundant parentheses in $\\Omega(n, m)$.", "was_revised": true, "final_justification": "I have no additional questions and the authors' response well addresses my previous questions/concerns. I will keep my score for recommendation.", "rebuttal": "**Notations)** Thank you for pointing out. We will ensure that the notation $d_x, d_y$​ is used consistently throughout the paper.\n\n**Typo)** Thank you for pointing out the typo. We will fix it. \n\n**Essentialness of $C^1$)** We believe the assumption is essential.  \nSince $L^{\\infty}$ is not separable (i.e., it does not have a countable dense subset), neural networks with continuous activation functions(which form only a countable-dimensional function class) cannot approximate all functions in this space.  \nSimilarly, the space of continuous functions with bounded derivatives is also non-separable.  \nAs a result, neural networks with $C^1$ activation functions cannot approximate certain functions in that space.  \nTherefore, it is necessary to restrict the function space to a $C^1$-function space with a countable basis.\n\n**Extension to Higher-Order)** \nThank you for the insightful question. We believe that if $k \\geq 2$, the same optimal minimum width as in the case $k = 1$ (our setting) still holds.  Since the topology of $W^{k,\\infty}$ becomes finer for $k \\geq 2$, the lower bound remains valid.  For the upper bound, we expect that Lemma B.7 can be generalized to the $k \\geq 2$ case, albeit with some tedious calculations.\n\nWe believe that the main obstruction to proving the upper bound lies in Lemma B.2.  To approximate affine transformations using nonlinear activation functions in the $W^{k,\\infty}$ norm, additional arguments are needed.  For instance, in the case $k = 2$, if there exists a point $x$ such that $\\sigma'(x) \\neq 0$ and $\\sigma''(x) = 0$, a result analogous to Lemma B.2 can be established.  \nSuch an activation function may potentially be constructed by composing simpler activation functions.\n\nExtending these results to the case $k > 2$ would be an interesting direction for future research.\n\n**Limitations)** Thank you for pointing out the limitations. We will add to the paper noting that our result is non-constructive and asymptotic in nature."}, {"review_id": "HD2swnQsTN", "reviewer": "Reviewer_g2vW", "summary": "The paper attempts to find the minimum width necessary such that a set of neural networks of arbitrary depth are still universal approximators. Instead of being universal in the set of continuous functions, only continuously differentiable functions are required to be approximable. The norm used for approximation is the uniform Sobolev norm of first order. \n\nThe first central result is Theorems 4.1 in which it is demonstrated that arbitrary diffeomorphisms can be approximated in the aforementioned sense requiring a width of $d$ or $d+1$. It is then shown that compositions of diffeomorphisms and affine embeddings can be approximated based on a certain embedding number. This yields an abstract approximation result for general C^1 functions in Theorem 4.6. \nThe embedding numbers are then calculated in Section 5. Corresponding lower bounds are also identified, albeit only in special cases (Theorem 5.9)", "questions": "1. Could the authors add an overview of results in the literature on resnets and explain the novelty of their proof idea?\n2. Can the significance of the results be better described?\n3. Can they improve the misleading statements?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The results are certainly interesting, the proof idea based on first apprpoximating diffeomorphisms is insightful and the arguments are mathematically sound. \n\nOn the other hand, I see a couple of weaknesses:\n\n1. The novelty could be explained a bit better. To what extend is the argument given here new? Has it never been used in similar approaches, e.g. when proving universality statements in other norms? There is also quite some literature on showing universality of Resnets, where similar ideas are likely developed. I think these should be discussed in the related work section.\n2. The significance is not that strongly supported. In some sense, this only extends known universality results to a more obscure norm on a more obscure function space. Why should one care about this? Are C^1 functions particularly important in some application? Is the W^{1,\\infty} norm used too assess the quality in prominent learning problems? \n3. Some statements seem intentionally vague when they do not need to be. In particular, the claim \"we prove that the optimal width for approximating arbitrary $C^1$ functions is [...] in infinitely many cases.\" This use of \"infinitely many\" is misleading, because these are actually very few cases for which a certain combinatorial condition is satisfied. A more honest statement would be more appropriate.\n4. The proof of Lemma A.1 is slightly inaccurate. The composition of W^{1,\\infty} functions is generally not well defined, because they are only defined almost everywhere. The issue is that, for f \\circ g, the function g could map a non-zero set to a zero set. This can easily be remedied by assuming that in any composition one takes a continuous representative of the outer function which is possible since f in W^{1,\\infty}. Hence, Lemma A.1 works perfectly, well, it just needs a footnote somewhere explaining how to define the composition.   \n5. Some minor issues:\n- In the formula under line 143, f(x) should be replaced by f.\n- In line 162 the \"a\" before Lipschitz should be removed.\n- Some brackets are wrong in line 276. \n- Line 309-310 seem to be copy-pasted from a different paper, because they seem hardly fitting for this manuscript.", "was_revised": true, "final_justification": "The reviewers have very precisely addressed my concerns. I think the paper is very good, but not ground-breaking. Therefore, I think the score 5 is appropriate.", "rebuttal": "**Novelty)** Thank you for your comment. The idea of approximating diffeomorphisms using deep narrow MLPs and control theory was first introduced in [1]. And later in [2], the authors applied this property to study universal approximation under the uniform norm. However, their conclusion was partially incorrect due to a flawed treatment of diffeomorphisms. In [9], the concept of approximating continuous functions using diffeomorphisms is rigorously estabilished. However, the authors do not provide lower bounds that are applicable in practical settings. Moreover, due to their proof technique being tailored to the uniform norm, the result cannot be directly generalized to the Sobolev norm setting.\n\nIn our work, we adopt a framework based on control theory and extend the results to the Sobolev norm setting. We believe that introducing the Sobolev norm into the minimum width literature is itself a novel contribution, as it offers a useful tool—such as the robustness of topological structures—for establishing lower bounds. Demonstrating this property is also one of our key contributions.\nMoreover, we provide a rigorous characterization of the relationship between the optimal minimum width and $\\Omega(n,m)$ in the Sobolev norm. \n\n**Most importantly, by employing concepts from real projective spaces and the Borsuk–Ulam theorem, we establish—for the first time—a nontrivial optimal lower bound in cases that had not been previously proven.\nTo the best of our knowledge, this is the first instance in which such a bound is derived using geometric concepts, including real projective spaces and the Borsuk–Ulam theorem.\nWe believe this is where the novelty of our work lies.**\n\nRegarding the ResNet case, we will add relevant references [3–5] to the related work section. In particular, [4] employs a similar control-theoretic technique for approximation. If there are additional references we have missed, we would appreciate your suggestions.\n\n**Significance)** Thank you for pointing this out. We will add the following sentences to the introduction:\n\n*However, there has been a scarce number of papers that study norms involving derivatives of functions in the setting of deep narrow MLPs. Many deep learning techniques directly penalize the difference between the derivative of the target function and that of the network. These include Sobolev Training [6], Physics-Informed Neural Networks (PINNs) [7], and Generative Adversarial Networks with Gradient Penalty [8].*\n\n**Vague Conditions)** We will revise the statement as follows:\n\n*In some cases, the lower bound coincides with the upper bound, thus identifying the optimal minimum width. This includes interesting cases such as (8,8) and (16,8). The exact pairs to which our result applies can be found in Theorem 5.9.*\n\n**The proof of Lemma A.1)** Thank you for identifying the error. We will revise the proof and add a footnote as suggested by the reviewer.\n\n**Minor Issues)** We will correct these as noted.\n\n**References**\n\n[1] Duan, Yifei, et al. \"Vanilla Feedforward Neural Networks as a Discretization of Dynamical Systems.\" arXiv preprint arXiv:2209.10909 (2022).\n\n[2] Li, Li’Ang, et al. \"Minimum width of leaky-ReLU neural networks for uniform universal approximation.\" International Conference on Machine Learning. PMLR, 2023.\n\n[3] Lin, Hongzhou, and Stefanie Jegelka. \"Resnet with one-neuron hidden layers is a universal approximator.\" Advances in neural information processing systems 31 (2018).\n\n[4] Aizawa, Yuto, Masato Kimura, and Kazunori Matsui. \"Universal approximation properties for an ODENet and a ResNet: Mathematical analysis and numerical experiments.\" arXiv preprint arXiv:2101.10229 (2020).\n\n[5] Tabuada, Paulo, and Bahman Gharesifard. \"Universal approximation power of deep residual neural networks through the lens of control.\" IEEE Transactions on Automatic Control 68.5 (2022): 2715-2728.\n\n[6] Czarnecki, Wojciech M., et al. \"Sobolev training for neural networks.\" Advances in neural information processing systems 30 (2017).\n\n[7] Raissi, Maziar, Paris Perdikaris, and George E. Karniadakis. \"Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations.\" Journal of Computational physics 378 (2019): 686-707.\n\n[8] Arjovsky, Martin, Soumith Chintala, and Léon Bottou. \"Wasserstein generative adversarial networks.\" International conference on machine learning. PMLR, 2017.\n\n[9] Hwang, Geonho. \"Minimum width for deep, narrow mlp: A diffeomorphism approach.\" arXiv preprint arXiv:2308.15873 (2023)."}]

Author rebuttal:

Meta-review: This paper studies the minimum width  for deep  narrow networks for approximating $C_1$ function under the Sobolev norm $W^{1,\infty}$ norm. The novelty is in studying minimal width for $C_1$ function as continuous functions has been studied previously. 

Authors leverage control theory and differential geometry to provide a minimum width in this setup.

All reviewers agreed on the good quality of the work  : 
* Reviewer LpbM and other reviewers suggested adding motivation to why the $C_1$ function are important, authors gave in the rebuttal motivations such as Physics inspired neural networks, Sobolev Training and gradient regularizers in GAN. Note citation [3] that authors provided does not do any gradient regularizer, the citation in GAN with gradient penalty is rather: 
Improved Training of Wasserstein GANs. Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, Aaron Courville
On gradient regularizers for MMD GANs. Michael Arbel, Danica J. Sutherland, Mikołaj Bińkowski, Arthur Gretton
and many other works such as Sobolev GAN etc ...
* points raised by  Reviewer 5C4H on presentation and the clarity , to the authors please incorporate this feedback in the final manuscript. 
* Reviewer g2vW raised similar concerns on vague statements and found an error in the proof of Lemma A.1 which does not compromise the lemma. Please fix these issues. 

Overall this is a good paper that contributes to the understanding of deep narrow networks.
Decision: Accept (poster)