Universal approximation power of deep residual neural networks via nonlinear control theory
Authors: Paulo Tabuada
Bahman Gharesifard
Venue: ICLR 2021 Poster (2021)
Keywords: Deep residual neural networks
universal approximation
nonlinear control theory

Abstract: In this paper, we explain the universal approximation capabilities of deep residual neural networks through geometric nonlinear control. Inspired by recent work establishing links between residual networks and control systems, we provide a general sufficient condition for a residual network to have the power of universal approximation by asking the activation function, or one of its derivatives, to satisfy a quadratic differential equation. Many activation functions used in practice satisfy this assumption, exactly or approximately, and we show this property to be sufficient for an adequately deep neural network with $n+1$ neurons per
layer to approximate arbitrarily well, on a compact set and with respect to the supremum norm, any continuous function from $\mathbb{R}^n$ to $\mathbb{R}^n$. We further show this result to hold for very simple architectures for which the weights only need to assume two values. The first key technical contribution consists of relating the universal approximation problem to controllability of an ensemble of control systems corresponding to a residual network and to leverage classical Lie algebraic techniques to characterize controllability. The second technical contribution is to identify monotonicity as the bridge between controllability of finite ensembles and uniform approximability on compact sets.

Published as a conference paper at ICLR 2021
UNIVERSAL APPROXIMATION POWER OF DEEP RESIDUAL NEURAL NETWORKS VIA NONLINEAR CONTROL THEORY
Paulo Tabuada
Bahman Gharesifard
Department of Electrical and Computer Engineering University of California at Los Angeles,
Department of Mathematics and Statistics
Queen's 's University
Los Angeles, CA 90095
Kingston, ON, Canada
tabuada êee. ucl a.cla.ccm edu edu
bahman. gwwresaarreaae fardeeeessrr
ABSTRACT
In this paper, we explain the universal approximation capabilities of deep resid- ual neural networks through geometric nonlinear control. Inspired by recent work establishing links between residual networks and control systems, we provide a general sufficient condition for residual network have the power of univer- sal approximation by asking the activation function, or one of its derivatives, to satisfy a quadratic differential equation. Many activation functions used in prac- tice satisfy this assumption, exactly or approximately, and we show this property be sufficient for an adequately deep neural network with n + neurons per layer to approximate arbitrarily well, on compact set and with respect to the supremum norm, any continuous function from R to R". We further show this result to hold for very simple architectures for which the weights only need to assume two values. The first key technical contribution consists of relating the universal approximation problem to controllability of an ensemble of control sys- tems corresponding to a residual network and to leverage classical Lie algebraic techniques to characterize controllability. The second technical contribution is identify monotonicity as the bridge between controllability of finite ensembles and uniform approximability on compact sets.
INTRODUCTION
In the past few years, we have witnessed a resurgence in the use of techniques from dynamical and control systems for the analysis of neural networks. This recent development was sparked by the papers (Weinan 2017; Haber Ruthotto 2017, Lu et al. 2018) establishing a connection between certain classes of neural networks, such residual networks (He et al.) 2016), and control systems. However, the use of dynamical and control systems to describe and analyze neural networks goes back least the 70's. For example, Wilson-Cowan's equations (Wilson & Cowan 1972) are dif- ferential equations and so is the model proposed by Hopfield in (Hopfield 1984). These techniques have been used to study several problems such weight identifiability from data (Albertini & Son- lag 1993 Albertini et al. 1993). controllability (Sontag & Qiao 1999 Sontag & Sussmann 1997).
and stability (Michel
1989 Hirsch 1989).
The objective of this paper is to shed new light into the approximation power of deep neural net- works and, in particular, of residual deep neural networks (He al. 2016). It has been empirically observed that deep networks have better approximation capabilities than their shallow counterparts and are easier to train (Ba Caruana 20014, Urban et al. 2017). An intuitive explanation for this fact is based on the different ways in which these types of networks perform function approxima- tion. While shallow networks prioritize parallel compositions of simple functions (the number of neurons per layer is measure of parallelism), deep networks prioritize sequential compositions of simple functions (the number of layers is a measure sequentiality). It is therefore natural to seek insights using control theory where the problem of producing interesting behavior by manipulating a few inputs over time, i.e., by sequentially composing them, has been extensively studied. Even though thhoreec techniques have been utilized in the literature to showcase the controllabil-

---

Published as a conference paper at ICLR 2021
ity properties of neural networks, the best of our knowledge, this paper is the first use tools from geometrie control theory to establish universal approximation properties with respect to the infinity norm.
1.1 CONTRIBUTIONS
In this paper we focus on residual networks (He et al. 2016). This being said, explained in et al. 2018), similar techniques can be exploited to analyze other classes of networks. It is known that deep residual networks have the power of universal approximation. What is less understood is where this power comes from. We show in this paper that it stems from the activation functions in the sense that when using sufficiently rich activation function, even networks with very simple architectures and weights taking only two values suffice for universal approximation. It is the power of sequential composition, analyzed in this paper via geometric control theory, that unpacks the richness of the activation function into universal approximability. Surprisingly, the level of richness required from an activation function also has a very simple characterization; suffices for activa- tion functions (or suitable derivative) to satisfy quadratic differential equation. Most activation functions in the literature either satisfy this condition or can be suitably approximated by functions satisfying it.
More specifically, given a finite ensemble of data points, we cast the problem of designing weights for training deep residual network as the problem of driving the state of finite ensemble of initial points with single open-loop control input to the finite ensemble of target points produced by the function be learned when evaluated at the initial points. In spite of the fact that we only have access to a single open-loop control input, we prove that the corresponding ensemble of control systems controllable. This result can also be understood terms of the memorization capacity of deep networks, almost any finite set of samples can be memorized, see (Yun et al. 2019 Vershynin 2020) for some recent work on this problem. We then utilize this controllability property to obtain universal approximability results for continuous functions in uniform sense, i.c., with respect to the supremum norm. This is achieved by using the notion of monotonicity that lets us conclude uniform approximability compact sets from controllability of finite ensembles.
1.2 RELATED WORK
Several papers have studied and established that residual networks have the power of universal ap- proximation. This was done in (Lin & Jegelka| 2018) by focusing on the particular case of residual networks with the ReLU activation function. was shown that any such network with n states and one neuron per layer can approximate an arbitrary Lebesgue integrable function f I - R with respect to the L norm. The paper (Zhang al., 2019) shows that the functions described by deep networks with states per layer, when these networks are modeled as control systems, are restricted to be homeomorphisms. The authors then show that increasing the number of states per layer to 2n suffices to approximate arbitrary homeomorphisms underlying network already has the power of universal approximation, Note that the results in (Lin) & Jegelka, 2018) do not model deep networks control systems and, for this reason, bypass the homeomorphism restriction. There also an important distinction to be made between requiring network exactly implement function and approximate it. The homeomorphism restriction does not prevent a network from approximating arbitrary functions; it just restricts the functions that can be implemented a network. Closer this paper are the results in (Li et al. 2019) establishing universal approximation, with respect to the I norm, S p < oc, based on general sufficient con- dition satisfied by several examples of activation functions. These results are major step forward in identifying what is needed for universal approximability. as they are not tied to specific architectures or activation functions. In this paper we establish universal approximation in the stronger sense of the infinity norm - which implics, special case, universal approximation with respect to the L norm for 1 < oo.
R" + R" under the assumption the
At the technical level, our results build upon the controllability properties of deep residual networks. Earlier work on controllability of differential equation models for neural networks, e.g., (Sontag & Qiao| 1999), assumed the weights to be constant and that an exogenous control signal was fed into the neurons. In contrast, we regard the weights as control inputs and that no additional con- trol inputs are present. These two different interpretations of the model lead to very different
2

---

Published as a conference paper at ICLR 2021
technical problems. More recent work in the control community includes (Agrachev & Caponigro 2009]. where it is shown that any orientation preserving diffeomorphism on a compact manifold, can be obtained as the flow of control system when using time-varying feedback controller. In the context of this paper those results can be understood as: residual networks can represent any orientation preserving diffeomorphism provided that we can make the weights depend the state. Although quite insightful, such results are not applicable to the standard neural network models where the weights are not allowed to depend on the state. Another relevant topic ensemble con- trol. Most of the work on the control of ensembles, see for instance (Li & Khaneja 2006 Helmke & Schönlein 2014, Brockett 2007). considers parametrized ensembles of vector fields. In other words, the individual systems that drive the state of the whole ensemble are different, whereas in our setting the ensemble consists of exact copies of the same system, albeit initialized differently. In this sense, our work is most closely related to the setting of (Agrachev & Sarychev, [2020a b) where controllability results for ensembles of infinitely many control systems are provided. In this paper, in contrast, we use Lie algebraic techniques to study controllability of finite ensembles and obtain approximation results for infinite ensembles by using the notion of monotonicity rather than Lie algebraic techniques is done in (Agrachev & Sarychev (220aa]b) (b). Moreover, by focusing on the specific control systems arising from deep residual networks we are able to provide easier to verify controllability conditions than those provided in (Agrachev & Sarychev 2020a b) for more general control systems. Controllability of finite ensembles of control systems motivated by neural network applications was investigated in (Cuchiero et al. 2019) where it is shown that controllabil- ity is generic property and that, for control systems that are linear in the inputs, 5 inputs suffice. These results are insightful but they do not apply to specific control systems such as those describing residual networks and studied in this paper. Moreover the results in (Cuchiero et al. (219)) do not address the problem of universal approximation in the infinity norm.
To conclude the review of related work, we note that universal approximation with respect to the in- finity norm for non-residual deep networks, allowing for general classes of activation functions, was recently established in (Kidger & Lyons 2020). In particular, it is shown in (Kidger & Lyons 2020 that under very mild conditions on the activation functions any continuous function f: K R T where K R" is compact, can be approximated in the infinity norm using deep neural network of width n + m + 2. Even though these results do not directly apply to residual networks (due to the presence of skip connections). they require 2n + 2 neurons for the case discussed in this paper where n = m. In contrast, one of our main results, Corollary 4.5 asserts that a width of n + is sufficient for universal approximation.
When preparing the final version of this paper we became aware of the paper (Park et al. 202 published at the same venue and addressing a similar problem. Although the results of (Park al.) 2021) do not directly apply to residual networks (due to the presence of skip connections), they establish that max n + m, neurons per layer suffice for universal approximation. When the case discussed in this paper, both papers provide the same number of neurons n + 1. However, there are several technical nuances across the results (for instance, some results in (Park et all 2021) require layer of step activation functions, in addition to, e.g., ReLUs, whereas others hold even the domain of f is not compact when approximating in the LP norm) that deserve further investigation.
m,
CONTROL-THEORETIC VIEW OF RESIDUAL NETWORKS
2.1 FROM RESIDUAL NETWORKS TO CONTROL SYSTEMS AND BACK
We start by providing control system perspective residual neural networks. We mostly follow the treatment proposed in (Weinan 2017| Haber & Ruthotto 2017 Lu et al. 2018), where was suggested that residual neural networks with an update equation of the form:
z(K + 1) (kk + ((K) (k)) + 6(k)),
(2.1)
where k E No indexes each layer, =kk) E R". and (S(k), (K), b(k)) E Raxn x Rax x R". can be interpreted as a control system when & is viewed as indexing time. In 2.1). S. W, and b are the weights functions assigning weights to each time instant i. and Σ R" R" is of the form E(z) = (z)2)). o(x))........ (.....,((z,) (2,)). where o : R" + R" is an activation function. By drawing an analogy between (2.1) and Fuler's forward method discretize differential equations, one can
3

---

Published as a conference paper at ICLR 2021
interpret 2.1 the time discretization of the continuous-tima time control system:
x(1) = (())(()))()))))) + b()),
(2.2)
where (t) E R and (S(t), W (1), b(t)) R x R x R"; in what follows, and in order to make the presentation simpler, we sometimes drop the dependency on time. To make the connec- tion between the discretization and (2.2) precise, let 10,7 + I be solution of the control system (2.2) for the control input (S. W, b)= [0, + _2*x x R x I", where T IR+, Then, given any desired accuracy E IRt and any norm in RR", there exists sufficiently small time step T E R so that the function 2 {0, 10,1,... [T/T]. + R" defined by:
(0) = x0),
=(k + = =(k) TI( + b(TT),
approximates the sequence (ar (k:) k=o with error e, i.e.:
(<(k)) - - (kT)
for all k E {0. Lo....... [./ Intuitively, any statement about the solutions (.2) holds for the solutions of (2.1) with arbitrarily small error E, provided that can choose the depth to be arbitrarily large since by making T small we increase the depth, given by 1 + [T/T].
2.2 NEURAL NETWORK TRAINING AND CONTROLLABILITY
Given function ʃ R. R" and finite set of samples Esamples RR" the problem of training residual network so that it maps E Saamlees to f(r can be phrased as the problem of constructing an open-loop control input (S, W,,,),) [0, + R=Xn Rux= Rn so that the resulting solution of (2.2) takes the states E Esamples to the states f(x). It should then come as no surprise that the ability to approximate a function f is tightly connected with the control-theoretic problem of controllability: given, one initial state التن! E I" and one final state zi R", when does there exist a finite time E R+ and a control input (S, W, b) [0, + Raxa Ruxa x I² so that the
solution of (2.2) starting at tit at time 0 ends at x at time 7?
To make the connection between controllability and the problem of mapping every I E Esamples to f()) clear, it is convenient to consider the ensemble of d Esspless copies of (222) given by the matrix differential equation:
(1) = (S))) (1) (1) + (()) +00)) (S()) X. b(1)))
(2.3)
where for time t E R,"" the ith column of the matrix (t) E Raxd denoted by (t). is the solution of the ith copy of (2.2) in the ensemble. If now index the elements of Esamples as fx ....., "., where is the cardinality of Esamples and consider the matrices X = [ |x+1 ... [x-]]
[f(")), we see that the existence of control input resulting in solution of (2.3) starting at Xinit and ending at X i.e,, controllability of (2.3). is equivalent to
and n (f(()))(()))
existence of input for (2.2) that the resulting solution starting at x² Esamples ends at f(("")
for all i E {1,.... [I....,d] .d.
Note that achieving controllability of (2.3 especially difficult, since all the copies of (.2 in 2.3 are identical and they all use the same imput. Therefore, to achieve controllability, we must have sufficient diversity in the initial conditions overcome the symmetries present in (.33) ), see Aguilar & Gharesifard 2014). Our controllability result, Theorem 4.2) describes precisely such diversity.
As mentioned the introduction, this observation also distinguishes the problem under study here from the classical setting of ensemble control (Li & Khaneja 2006 Helmke & Schönlein 2014), with the exception of the recent work (Cuchiero et [ggaachrrrcee & Sarychev] b). where collection of systems with different dynamics are driven by the same control input.
3 PROBLEM FORMULATION
Our starting point is the control system:
i(() (())(((()))()))))) (()),
(3.1)
a slightly simplified version of (.2). where (t) E R", (s(t), W b(t)) E R x Raxn x R", and the input S in 2.2) is now the valued function 8; as we will prove in what follows, this model
4

---

Published as a conference paper at ICLR 2021
is enough for universal approximation. In fact, will later sed that it suffices to let assume two arbitrary values only (one positive and one negative). Moreover, for certain activation functions, we can dispense with altogether.
Assumption 1. We make the following assumptions regarding the model (311.
The function Σ is defined as E: : I + (x2)), a (In)), function σ R i.e., DC = ag + arE + azL² with 01, a2, E IR, a2 # O, and & = DD a for some j E No- Here, D'a a denotes the derivative of of order j and Do = 0.
where the activation or suitable derivative it, satisfies a quadratic differential equation,
The activation function σ : RR- + +RR is Lipschitz continuous, Do 2 0. and & = Dåo defined above is injective.
Table 1: Activation functions and the differential equations they satisfy.
Function name
Definition
Satisfied differential equation
Logistic function
a(x) I+e-5
Do a + = Do - 1 Dd + r
Hyperbolic tangent
o(x) cr
Soft plus
(x) +log(1 + cr
Several activation functions used in the literature are solutions of quadratic differential equations Moreover, activation functions that are not differentiable can also be handled via approximation. For example, the ReLU function defined by max(0. z) can be approx- imated by (x)
as can be seen in Table [
+ +e")///". as r > oo, which satisfies the quadratic differential equation given in Table 1 Similarly, the leaky ReLU, defined by oxx) = for 2 0 and oxx) = rw for < 0, is the limit as k - Do of a(x) = ræ + log(1 + ((1-r)kx)/ and the function a satisfies D - k(1 + Da + k(Da)² + krr = 0.
The Lipschitz continuity assumption is made to simplify the presentation and can be replaced with local Lipschitz continuity, which then does not need to be assumed, since o is analytic virtue of being the solution of an analytic (quadratic) differential equation. Moreover, all the activation functions in Table I are Lipschitz continuous, have positive derivative and are thus injective.
To formally state the problem under study in this paper, need to discuss different point of view on the solutions of the control system (3.1) given by flows. A continuously diffcrentiable curve J0, 7] R" is said be solution of (3.1) under the piecewise continuous input
(s, W,b) b) : [0, r] + x Raxa x IR" if it satisfies (3.1). Under the stated assumptions on a. given a piecewise continuous input and state --il RR"" there is one and at most one solution xx() of (3.1) satisfying z00) iii Moreover, solutions are defined for all 7 R We can thus define the flow of (3.1) under the input (s, W. b) as the map o= _ + R" given by the assignment اiنن + x(r). In other words, o (2."it) is the point reached at time r by the unique solution starting at - at time O. When the time is clear from context, we denote a flow simply by o. It will also be convenient to denote the flow o² by Z" when Φ is defined by the solution of the differential equation i = ((x) for some vector field Z R R
We will use flows to approximate arbitrary continuous functions f R" + R". Since flows have the same domain and co-domain, and f R > Ih' may not, we first lift f to a map f R²* + R$. When n > m. we lift / to o f : R² + R", where v Rm - Rn is the injection given by ((x) = (#1..... Tn, ........).......... 0). In this case k- = n. When n < m, we lift f to = =for= : T - T where 1.m + = is the projection (I.,,.... Tns Ta+1....... Fm) (Z), ,x)) In this case A = m. Although we could consider factoring f through map R" + RT i.c., to construct f R + R" so that = go ff as done in, e.g., Li et al. 2019), the construction of g requires deep understanding of f. since a necessary condition for this factorization is fR") S ((R)). f
Constructing g so as to contain f(R") on its image requires understanding what fR²" is and this information is not available in learning problems. Given this discussion, the remainder of this paper we directly assume we seek to approximate a map f R + R".
See the discussion after the proof of Theorem 4.2
Tabuada & Gharesifard 2020
5

---

Published as a conference paper at ICLR 2021
The final ingredient need before stating the problem solved in this paper is the precise notion of approximation. Throughout the paper, we will investigate approximation in the sense the L (supremum) norm, i.e.:
(E)
sup f( zeE
where E RR is the compact set over which is the approximation is going to be conducted and [x(=)]== = m () [x((
Some approximation results will be stated for networks modeled by control system (3.1) with state space R In such cases the approximation quality is measured by |[f - ( (E) where & is the flow of (3.I. Other results will require networks with state space R' for m > n. For those cases the approximation quality is measured by |l - B ø o alln=> (F) where R" + R is an injection and B ET + R" is a projection. These maps will be linear and can be implemented as the first and last layers of a residual network.
We now ready to state the problems study in this paper.
Problem 3.1. Let f Rn + R be a continuous function, Esamles I be a finite set, and & E B, be the desired approximation accuracy. Under Assumption | does there exist time E R and an imput (s, W,b) b): [0 IR Ruxa In so that the flow o² IR"" + R= defined by the solution of 3.1 with state space I= under the said input satisfies:
Il 10 (Enampa)) S 6?
Note that we allow E be zero in which case the flow oJ matches exactly on Esamples i.c., f() (x) for every E Esumples. The next problem considers the more challenging case of approximation on compact sets and allows for residual networks with m n neurons per layer when approximating functions on R".
Problem 3.2. Let f I + R" be continuous function, E C B be compact set, and 8 IR+ be the desired upproximation accurucy. Under Assumption
7 does there exist m E N,
: R" + R", projection 3 Im - R". and imput (s. W,b) : [0, + R x gax" x IR" such that the flow o² : R"" + R defined by the solution of 3.1 with state space Tu++1 under the said input satisfies:
time RR+ injection
|/ - B o o all (=() S
In the next section, we will show the answer to both these problems to be affirmative. For the second problem we can take under the additional assumption of monotonicity which is satisfied by construction when we allow m = n + neurons per layer.
MAIN RESULTS
The proofs of all the results in this section are provided (Tabuada Gharesifard 2020).
We first discuss the problem of constructing an input for 3.1 so that the resulting flow & satisfies <(z) = f (x) for all the points r in a given finite set Eamples C R". We explained in Section 2.2 that this is cquivalentt to determining if the ensemble control system (2.3) is controllable. It is simple to
see that controllability of (.3) cannot hold on all of Raxd, since if the initial state X(0) satisfies (0) X., (0) for some + j. must have (t) (t) for all t [0, r| by uniqueness of solutions of differential equations.
Our first result establishes that the controllability property holds for the ensemble control sys- tem (.3) on a dense and connected submanifold of independently of the (finite) number of copies d, long the activation function satisfies Assumption 1 Before stating this result, recall the formal definition of controllability.
Definition 4.1. A point X E Raxd is said to be reachable from point Xini E Ruxd for the control system (2.3) if there exist T + and a control imput (s, W,b) b) : [0, 7] + Raxa T so that the solution X of (2.3) under said input satisfies X(0) Xinit and ( Xfin Control
system (2.3 is said to be controllable on a submanifold M of Rad if any point in M is reachable from any point in M.

---

Published as a conference paper at ICLR 2021
Theorem 4.2. Let N C Rrxd be the set defined by:
N = AE ARxd
II
(Au - A =0, EE {1, [l.......
Isisisa
Let and suppose that Assumption 7 holds. Then the ensemble control system 2.3 is control- lable on the submanifold M = gaxds. NN.
It is worth mentioning that the assumption of * ensures connectedness of the submanifold weakens
M, which we rely on to obtain controllability. The following corollary of Theorem 4.2 controllability reachability but applies to larger set.
Corollary 4.3. Let M C Tixxd be the submanifold defined in Theorem 4.2 Under assumptions of Theorem 4.2 any point in M is reachable from a point A E Rgx for which:
Ao: + Aoj,
holds for all + j. where i.j E E(),
[I,...,d}.
The assumption Asi # Aaj in Corollary
4.3 requires all the columns of A to be different and is
always satisfied when A [x 122) x x² E Esamples Hence, for any finite set Esamles there
exists flow 6 of (3.1) satisfying f(x) ((z) for all E Esamples provided that f (Eaapple)) C M, i.e., Problem 3.1 is solved with 0. Moreover, since M is dense in Raxd, when S [(Eaamples) C M
fails, there still exists a flow @ of B.T taking ((xr) arbitrarily close to ((r for all E Fsamples i.e., Problem 3.1 is solved for any residual networks as states that almost any finite set of samples can be memorized, independently of its cardinality. See, e.g.. (Yun al., 2019: Vershynin 2020 for recent results this problem that do not rely on differential equation models.
0. This result also sheds light on the memorization capacity of
Some further remarks are in order. Theorem 4.2 and Corollary 4.3 do not directly apply to the ReLU activation function, defined by max{0, 3) since this function is not differentiable. However, the ReLU is approximated by the activation function:
log(1 +e""),
r 0c. In particular, as r--- + oc the ensemble control system 2.3 with (<x) log(1+ ) converges to the ensemble control system (2.3) with (I) = max 2 and thus the solutions of the latter are arbitrarily close to the solutions of the former whenever
large enough. Moreover, E = Do satisfies D = r -f² and DE = re" +(1+= + > 0 for x E R and r > 0 thus showing that & is an increasing function and, consequently, injective.
The conclusions of Theorem 4.2 and Corollary 4.3 also hold if we weaken the assumptions on the inputs of (3.T). It suffices for the entries of and b to take values on set with two elements (one positive and one negative), see the discussion after the proof of Theorem 4.2 in (Tabuada & Gharesifard 2020) for details. Moreover, when the activation function is and odd function, i.e., (--z) - (z), is the case for the hyperbolic tangent, the conclusions of Theorem 4.2 hold for the simpler version of (3.1), where we fix s to be 1.
In order to extend the approximation guarantees from finite set Esamples C F to arbitrary com- pact set F. C R", we rely on the notion of monotonicity. On R. we consider the ordering relation
x' defined by x; x for all E (1,...,M n) and x., o. E R A map f R + R is said to be monotone when it respects this ordering relation, i.e., when x 3 x' implies f(x) I f(x'). When f
continuous differentiable, monotonicity admits a simple characterization (Hirsch & Smith
2006):
0f dz;
2 0, Vi.j E (1.....n..
(4.1)
vector field Z R= Ra is said be monotone when its flow 0J : R" R is a monotone map. Monotone vector fields admit a characterization similar to (4.1. see (Smith 2008):
az: > 0, Vi.j E {1,.... n) ij
(4.2)
da
7

---

Published as a conference paper at ICLR 2021
Theorem 4.4. Let n 1 and suppose that Assumption 7 holds. Then, for every monotone analytic function f IR" - R", for every compact set E C R,, and for every & E R² there exist a time
E Rt and an imput (s, W, W) [0. r] R x Rux" x R" that the flow o R + R defined
by the solution of 3.1 with state space Th under the said input satisfes:
- S..
(4.3)
When the function to be approximated is simply continuous, nomial function using weiersssssssssss Theorem and then embed it into a monotone function. Although such embeddings typically require doubling the dimension, leverage the existence of the projection map 3 to introduce a novel embedding that only requires increasing n to n +
first approximate it by a poly-
Corollary 4.5. Let n > and suppose that Assumption ! holds. Then, for every continuous function f R" + R", for every compact set E C R", and for every E E R² there exist a time T E R+, an injection a R. - R=+1, a projection B B + RR,, and an input (s. b) [0,r] + R x R(a+1)x(x+1) x Th-1 so that the flow o² -R=+1 + Ru+1 defined by the
solution of 3.1 with state space I=++1 under the said input satisfies:
If - BBod tt (E) Se.
It is worth pointing out that, contrary to Theorem 4.4 no requirements are placed on f in addition to continuity. (Agrachev Sarychev| 2020a b), sufficient conditions for the existence of a flow o" satisfying 4.33 are given for more general class of control systems. The assumptions used in The- orem (4.4) are not easy to compare with the assumptions in Theorem 5.1 of (Agrachev & Sarychev) 2020b). However, by employing deep network of width + i.c., by using Corollary 4.5 we rely on much simpler assumptions on the activation functions which are satisfied by the networks used in practice. In contrast, (Agrachev & Sarychev [020b] Theorem 5.1) requires strong Lie algebra approximation property to be satisfied by the ensemble control system that does not appear to be easy to verify.
KEY NOVEL TECHNICAL IDEAS
We find it fruitful to sketch the proofs, available in (Tabuada & Gharesifard 2020). of the main results to give the reader a flavor of some of the novel technical ideas.
In Theorem 4.2, use the notion of ensemble controllability to find the open-loop control inputs that map the finite set of sample Esamples C IR" to the corresponding state under f. The use of Lie algebraic techniques ear control (Jurdjevic 1996). To give the unfamiliar reader a glimpse on this, given two smooth vector fields f1, f2 : R" + R", one can show that the reachable set of the control system á = f (x) + 22 i.e., the set of points that can be reached by selecting and U2 appropriately, is equivalent the one for å un fi (x) + (2)2(2) + uglfi, f2)(x), ah f2(x) the Lie bracket of f1 and f2. Continuing in this fashion, the closure of the set of all brackets generated by S1 and f2, i.e., the Lie algebra generated by these, corresponds to the directions that we can steer the system to. Now the setting considered here is far from this classical setting and three challenges arise when attempting to apply these techniques to (.3). The first one is that we are steering an ensemble of data points, rather than one point, using single control system. The second challenge is that most existing Lie algebraic controllability results are for control systems that are affine in the input, such as the one above, whereas (2.3) is not. To address this challenge, we identified a subset of inputs that, when used in a piecewise constant manner, lead to a control system that is easier to analyze. The third challenge is that computing the dimension of the associated Lie algebra is difficult, since one has to establish linear (in depeedddeeeee of highly nonlinear functions. To proceed, we had to identify which as- sumptions to place on a so that the dimension of the Lie algebra could be computed while allowing for a quite general class of activation functions. Strikingly, by assuming that o satisfies a quadratic differential equation, which still includes many of the known activa- tion functions, we were able to provide a closed form expression for the determinant of the matrix (A.1) in Lemma A.1 in (Tabuada & Gharesifard 2020) thereby showing that, for every point in the open and dense submanifold M, the dimension of the Lie algebra is the dimension of the state space.
study controllability is standard practice in nonlin-
where [/i. [2]xr) = de (x) af2
8

---

Published as a conference paper at ICLR 2021
The novel idea in the proof of Theorem 4.4 is the link between monotonicity and approx- imability in the - sense. We already know from Theorem that deep residual networks finite sample
memorize to arbitrary accuracy any continuous function f evaluated set Psamples- The question left open is, what is the mismatch between f and the flow of (3.1) for points in the domain of f but not in Psamples? Lemma A.3 in (Tabuada & Gharesifard 2020) provides an upper bound for such gap provided that the flow of (3.I) Is monotone. With this upper bound in hand, the proof of Theorem 4.2 shows that when is mono- tone, there exists a monotone flow of (3.1) that approximates, with arbitrary accuracy, f evaluated on Esamples and the result then follows. In summary, the proof of Theorem 4.4 establishes that monotonocity is the key property that allows to extend approximation on finite sample sets (Theorem 4.2) to approximation on compact sets (Theorem 4.4).
Our final step is in Corollary 4.5 where construct embedding of into monotone mapping by increasing the domain and co-domain of by one, i.e., by working on IR*++1 Embedding non-monotone maps into monotone maps typically requires doubling the di- mension. The key novel idea here is to add linear correction to that is compensated by the linear function implemented as the last layer of deep residual network. With this linear compensation it suffices to increase the domain and co-domain of f by one, i.e., n + neurons suffice for universal approximation.
ACKNOWLEDGMENT
The work of the first author supported the CONIX research center, of six centers in JUMP, a Semiconductor Research Corporation (SRC) program sponsored by DARPA. The work of the second author was supported by the Alexander von Humboldt Foundation, and the Natural Sciences and Engineering Research Council of Canada. The authors wish to thank Professor Eduardo Sontag (Northeastern University) for insightful comments on earlier version of this manuscript.
REFERENCES
A. Agrachev and M. Caponigro. Controllability the group of diffeomorphisms. Annales de I'Institut Henri Poincare (C) Non Linear Analysis, 26(6):2503 2509, 2009.
A. Agrachev and A. Sarychev. Control in the spaces of ensembles of points. SIAM Journal on Control and Optimization, 58(3):1579-1596 2020a.
A. Agrachev and A. Sarychev. Control on the manifolds of mappings as setting for deep learning. arXiv preprint arXiv:2008. 12702, 2020b.
C. Aguilar and B. Gharesifard. Necessary conditions for controllability of nonlinear networked control systems. In American Control Conference, pp. 5379-5383, Portland, OR, 2014.
D. Albertini, E. D. Sontag, and V. Maillot. Uniqueness of weights for neural networks. Artificial Neural Networks for Speech and Vision, pp. 115-125, 1993.
F. Albertini and E. D. Sontag. For neural networks, function determines form. Neural Networks, 6 (7):975 990, 1993.
L. J. Ba and R. Caruana. Do deep nets really need to be deep? In Proceedings of the 27th Inter- national Conference Neural Information Processing Systems Volume 2, NIPS' 14, pp. 2654- 2662, Cambridge, MA, USA, 2014. MIT Press.
R. W. Brockett. Optimal control of the Liouville equation. AMS IP Studies Advanced Mathemat- ics, 39:23, 2007.
C. Cuchiero, M. Larsson, and J. Teichmann. Deep neural networks, generic universal interpolation, and controlled ODEs. https://artii.or//ab////0..... 07838, 2019.
E. Haber and L. Ruthotto. Stable architectures for deep neural networks. Inverse Problems, 34(1), 2017.
K. He, X. Zhang. S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.

---

Published as a conference paper at ICLR 2021
U. Helmke and M. Schönlein. Uniform ensemble controllability for one para-meeeerrr families of time- invariant linear systems. Systems & Control Letters, 71:69-77, 2014.
M. W. Hirsch. Convergent activation dynamics continuous time networks. Neural Networks, (5):331 349, 1989.
M. W. Hirsch and H. Smith. Monotone dynamical systems. In Handbook of differential equations: ordinary differential equations, volume 2, pp. 239-357. Elsevier, 2006.
J. Hopfield. Neurons with graded response have collective computational properties like those of two-state neurons. Proceedings of the National Academy of Sciences, 81(10):3088-3092 1984.
V. Jurdjevic. Geometric Control Theory. Cambridge Studies in Advanced Mathematics. Cambridge University Press, 1996.
P. Kidger and T. Lyons. Universal approximation with deep narrow networks. arXiv preprint A5K5:09905.8855 2020.
J-S. Li and N. Khaneja. Control of inhomogeneous quantum ensembles. Physical Review A, 73(3): 030302, 2006.
Q. Li, T. Lin, and Z. Shen. Deep learning via dynamical systems: An approximation perspective. arXiv preprint arXiv:1912.10382 2019.
H. Lin and S. Jegelka. ResNet with one-neuron hidden layers is a universal approximator, In Proceedings of the 32nd International Conference Neural Information Processing Systems, NIPS' 18, pp. 1172-6181, Red Hook, NY, USA, 2018. Curran Associates Inc.
Y. Lu, A. Zhong, Q. Li, and B. Dong. Beyond finite layer neural networks: Bridging deep archi- tectures and numerical differential equations. In International Conference Machine Learning, pp. 3276-3285, 2018.
A. N. Michel, J. A. Farrell, and W. Porod. Qualitative analysis of neural networks. IEEE Transac- fions Circuits and Systems, 36(2):229-25)) 1989.
Sejun Park, Chulhee Yun, Jaeho Lee, and Jinwoo Shin. Minimum width for universal approx- imation. In International Conference on Learning Representations, 2021. URL https:
7/ //operrvii ew. met//forume 1==00xXwwyTF22
H. L. Smith. Monotone Dynamical Systems: An Introduction the Theory of Competitive and Cooperative Systems. Mathematical Surveys and Monographs. American Mathematical Society, 2008.
E. D. Sontag and Y. Qiao. Further results on controllability of recurrent neural networks. Systems & Control Letters, 36(2):121 - 129, 1999.
E. D. Sontag and H. Sussmann. Complete controllability of continuous-time recurrent neural net- works. Systems Control Letters, 3074): 1997.
Tabuada and B. Gharesifard. Universal approximation power deep residual neural networks via nonlinear control theory. arXiv preprint arXiv:2007.060077: 2020.
G. Urban, K. J. Geras, S. Ebrahimi Kahou, Ö. Aslan, S. Wang. A. Mohamed, M. Philipose, M. Richardson, and R. Caruana. Do deep convolutional nets really need to be deep and con- volutional? In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26. 2017, Conference Track Proceedings. OpenReview.net, 2017.
R. Vershynin. Memory capacity of neural networks with threshold and ReLU activations. arXiv preprint arXiv:2001.06933 066988. 2020.
E. Weinan, A proposal on machine learning via dynamical systems. Communications in Mathemat- ics and Statistics, 5, 2017.
H. R. Wilson and D. Cowan. Excitatory and inhibitory interactions in localized populations of model neurons. Biophysical Journal, 12(1):1-24, 1972.
10

---

Published as a conference paper at ICLR 2021
C. Yun, S. Sra, and A. Jadbabaie. Small ReLU networks powerful memorizers: tight analysis of memorization capacity. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer. Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Process- ing Systems 32: Anmual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pp. 15532-15543, 2019.
H. Zhang, X. Gao, Unterman, and T. Arodz. Approximation capabilities of neural ODEs and invertible residual networks. arXiv preprint arxiv:190.112998, 2019.


Reviews:
[{"review_id": "xQOmsK5sCE9", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the universal approximability of residual networks. The main contribution of the paper is to prove that residual networks of width 2n can approximate any continuous function from $\\mathbb R^n$ to $\\mathbb R^n$ in arbitrary uniform error, for activation functions satisfying some differential equation. \n\nIt was interesting for me that only two values of weights suffice for the universal approximation of residual networks. I think that this result can be used to prove the expressive power of compressed/pruned networks.\n\nHowever, I think that there exists a critical weakness in the connection with the existing universal approximation results. As the authors mentioned in the related work section, (Kidger & Lyons; 2020) proved that feed-forward networks of width n+m+1 can approximate any continuous function from $\\mathbb R^n$ to $\\mathbb R^m$ in arbitrary uniform error, for general activation functions. In my opinion, this result can be easily extended to residual networks since residual networks can approximate by any feed-forward network by choosing some large s and small W in Eq. (3.1), i.e., ignore the residual connections. If this approximation is valid, the main contribution given by Theorem 4.4 is significantly weakened. \n\nDue to the weakness I mentioned, my decision tends toward rejection.\n\n------------------------------------------------------------------------------------\nafter reading the authors' response and revision, I raise my score to 6.", "extra_scores": {}, "extra_text": {}}, {"review_id": "l9M84G068cz", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\n  [1] The paper is well-written, clear and organized well. \n  [2] The approach of relating DNN universal approximation problem to controllability of ensemble of control systems is interesting. \n  [3] The results are a bit more general than existing literature in the same topic.\n\n\nMain concerns: \n  The paper contains some interesting ideas and new results, but it seems it is more an incremental work to existing literature and the contribution might be a bit limited and not significant enough. From empirical sense, it has been well observed that such deep residual neural networks has very good approximation capabilities to almost arbitrary functions. From theoretic sense, the existing relevant work by Li et al., 2019 has already established very similar results (also through dynamic system and control theory) for universal approximation capability of such deep residual neural networks, but just in the sense of L-p norm rather than L-infinity norm (where p can be any number between 1 and infinity). Although the authors argue that L-infinity norm is more general than such L-p norm result, the contribution does seems quite incremental to me (especially considering that the p can be any arbitrary large number between 1 and infinity), and I'm not very convinced by the significance of the contribution, in both theoretic sense as well as practical sense. To be clear, I do think that the approach in this paper is somewhat interesting and inspiring and it also contains some valuable and publishable results, but I just not convinced that the contribution is significant enough to be published as a regular paper in such a top conference like ICLR.\n\n-------\nAfter reading the author's responses, I'm ok to promote the rating from 5 to 6.", "extra_scores": {}, "extra_text": {}}, {"review_id": "PJTc_wNg9Df", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors consider residual networks. They interpret a residual network as a discrete analogue of some dynamical control system. Using the results of the control-theoretic problem of controllability they proved the universal approximation property of the deep neural networks.\n\nThe universal approximation property is rather weak; it is usually true for wide classes of models and activation functions. For example, in the case of single-layer networks, it is sufficient that the activation function is not a polynomial, see Pinkus, Allan. \"Approximation theory of the MLP model in neural networks.\" Acta numerica 8.1 (1999): 143-195.\n\nIn this paper, the universal approximation property is proved for a very specific class of activations (satisfying a certain diffur). It is not clear why this is a significant requirement. The authors mention that it is possible to extend the results to other activation functions, and explain how to do this for ReLU, but what to do with other activations is still unclear. Of course, approximation using a dynamic system is a more complex model than a conventional single-layer network, but this requirement looks exotic. \n\nSo, the authors consider a special type of a complex nonlinear model, parametrized by an infinite-dimensional control, for which the natural and rather weak property is proved. \n\nThe proof technology itself is quite interesting and is based on the ideas of Lie algebras. But this is all hidden in the Appendix; if a person reads only the main text, all these ideas are not visible, and the results are not particularly impressive themselves. \n\nAuthors should somehow emphasize their new ideas in the main text, or at least make it more interesting.\n\nReferences to literature are quite relevant in the work and the related work section is OK. \n\nAs a conclusion I can say that\n- the results itself is rather natural and weak. Nowadays for residual networks we have significantly stronger results about their approximation capabilities, see e.g. Dmitry Yarotsky. “Optimal approximation of continuous functions by very deep ReLU networks.” https://arxiv.org/pdf/1802.03620.pdf\n- possibly, the technique, used for the proof is a new interesting tool itself, so that it can be used to solve other similar problems. However, the authors did not mention anything about this in the main text of the paper, and for me it is difficult to assess the novelty as I am not the specialist in the control-theoretic problem of controllability, results of Agrachev et al., etc.\n\nTherefore, I think that in the current state the paper can not be accepted for a publication.\n\n=========\n\nAfter reading authors comments.\n\n- Well, still the Point 1 is generally not true in my opinion. The authors cited the work of Kidger & Lyons \"Universal approximation with deep narrow networks\", in which universal approximation is proved for a wide class of activation functions and long narrow networks. Moreover, if you do not strive to make the network as narrow as possible, then it is easy to convert a shallow wide network into a deep narrow one (but not vice versa). Point 2 seems quite possible. Point 3 coincides with what I mentioned --- the authors proved UAP for several examples of activations, although really quite important ones.\n\n- \"Results will be accessible without the need for the technicalities\". Well, I personally do not understand this statement in the conference article. From the main text, the reader only sees that a relatively basic property (universal approximation) is fulfilled for a couple of previously uncovered activation functions (or for a specialized class), and it is not clear why, since possibly new ideas are not in the main text. \n\n- This work clearly has strengths and weaknesses, which, in principle, are more or less clear. On the one hand, the universality property for approximations by dynamical systems and a specific class of activation functions is proved. How important, new and interesting is it? It doesn't seem to be super strong result, because this property is relatively weak, completely expected, and has already been proven many times before in various situations (wide shallow networks, narrow deep networks, dynamic systems with some classes of activation functions). In approximation theory, people have been studying more advanced questions like approximation rates for a long time. I also do not see any significant practical use of the results. On the other hand, it is possible that the authors have developed some new method when proving their results. But in order to check this fact the reviewer should check the whole appendix. If an article is submitted to a journal, the reviewer is expected to read full article, including the Appendix. But for this conference work, as far as I understand, this is not the case, the reviewer is not required to read the Appendix. Accordingly, if the authors could not convince the reviewer of the importance of the work within the main text, then this is their problem. As for the individual points that the authors wrote about - as I have already written, part of what they wrote is quite fair, no one disputes that they have some new results. At the same time, for example, I find it difficult to agree with point 1, as I have already written.\n\n- Well, in principle, there are two ideas that the authors mentioned (about just two values for all weights and memorising data), which are relatively interesting indeed. This is not something that would be mega-important, but it looks like non-obvious facts, which is a plus.\n\nIn principle, I can increase the grade for one point.", "extra_scores": {}, "extra_text": {}}, {"review_id": "OB7WpdJsOVp", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper gives a fresh perspective to a standard question in neural networks concerning their power to represent or approximately represent continuous functions. Here they focus on  deep residual neural networks where the activation units satisfy certain conditions defined via a differential equation (most common units in practice abide by this property) and present L_\\infinity approximation results which is the strongest possible sense of approximating a function.\n\nThe paper uses novel ideas and techniques stemming from dynamical systems and the crux of the argument relies on viewing residuals neural nets as a control system that one can influence via the weight assignments. Using the weights, one can try to obtain specific values as the network's output and hence the universal approximation property follows.\n\nSeveral ideas along those lines have been previously exploited in related work as the authors state. The reviewer believes there is technical novelty in the paper and also the result is quite elegant. Also, highlighting new connections to dynamical systems has recently been a fruitful direction. \n\nParts of the techniques establish some \"expansion\" property of the associated control system implying that for any given point in R^d, the dynamical system can be made to pass through it given the appropriate feedback loop. This holds true even for networks with simple\narchitectures and if the network's parameters take only two distinct values. \n\nThe paper overall offers nice theory for an important question although the practical implications seem limited.\n\nOther comments:\nStating the result for functions R^n to R^n isn't it a bit misleading? Isn't it true that once the result is established for f: R^n to R, then it can directly extend to higher dimension as well?\n \nIn terms of presentation, there are several ways to improve: it would be better to define separately what the activation units should be (satisfying the differential equations) and then use it in all places where it is needed. Currently, in most statements for Corollaries and Theorems the authors devote 2-3 lines of repeating information which waste space and make the theorems look ugly. \n\nAlso, the authors should clarify more what parts of the approach via Lie algebras is novel and how it is compared specifically with prior works. The reviewer (and potentially most people in ML) are not familiar with such techniques and currently it feels it may be an incremental result.\n\nMoreover, the paper uses ideas from dynamical systems to study approximation capabilities and related works missing from the discussion are https://arxiv.org/abs/2003.00777 and https://arxiv.org/abs/1912.04378 , where depth separations are provided. A similar property holds for those networks as the one described above where only two distinct parameters are used for the weights of the network and it would be interesting to see if your results also can be applied to obtain depth separations. \n\nThe paper in https://arxiv.org/abs/1602.04485 also offers representation benefits via general activations called semi-algebraic. What is the connection among those gates and your differential equations? A discussion would be appreciated.\n\nConcerning the Lie algebras presentation, some high-level discussion on how exactly it is used without delving into the technical details for non-experts would also be greatly appreciated and help others build on your work. Currently, several parts of the paper read off as a bit mysterious for non-experts.\n\nMinor Typo:\np1: to best of our knowledge", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper analyzes the universal approximation power of deep residual neural networks based on nonlinear control theory. Some concerns regarding the clarify, significance and connection to practice were raised, and partially addressed by the rebuttal. After discussions, all the reviewers feel positive about the contribution of the paper.
Decision: Accept (Poster)