Neural Network Architecture Beyond Width and Depth
Authors: Shijun Zhang
Zuowei Shen
Haizhao Yang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Neural Network Approximation
Nested Architecture
Parameter Sharing
Function Composition

Abstract: This paper proposes a new neural network architecture by introducing an additional dimension called height beyond width and depth. Neural network architectures with height, width, and depth as hyper-parameters are called three-dimensional architectures. It is shown that neural networks with three-dimensional architectures are significantly more expressive than the ones with two-dimensional architectures (those with only width and depth as hyper-parameters), e.g., standard fully connected networks. The new network architecture is constructed recursively via a nested structure, and hence we call a network with the new architecture nested network (NestNet). A NestNet of height $s$ is built with each hidden neuron activated by a NestNet of height $\le s-1$. When $s=1$, a NestNet degenerates to a standard network with a two-dimensional architecture. It is proved by construction that height-$s$ ReLU NestNets with $\mathcal{O}(n)$ parameters can approximate $1$-Lipschitz continuous functions on $[0,1]^d$ with an error $\mathcal{O}(n^{-(s+1)/d})$, while the optimal approximation error of standard ReLU networks with $\mathcal{O}(n)$ parameters is $\mathcal{O}(n^{-2/d})$. Furthermore, such a result is extended to generic continuous functions on $[0,1]^d$ with the approximation error characterized by the modulus of continuity. Finally, we use numerical experimentation to show the advantages of the super-approximation power of ReLU NestNets.

Neural Network Architecture Beyond Width and Depth
Zuowei Shen Department of Mathematics National University of Singapore matzuowsúnus. edu. ag
Haizhao Yang Department of Mathematics University of Maryland, College Park hzyang@umd. edu
Shijun Zhang"
Department of Mathematics National University of Singapore zhangshij junuu. nus edu
Abstract
This paper proposes a new neural network architecture by introducing an additional dimension called height beyond width and depth. Neural network architectures with height, width, and depth as hyper-parameters are called three-dimensional architectures. It is shown that neural networks with three-dimensional architectures are significantly more expressive than the ones with two-dimensional architectures (those with only width and depth as hyper-parameterrs), e.g., standard fully con- nected networks. The new network architecture is constructed recursively via a nested structure, and hence we call network with the new architecture nested net- work (NestNet). A NestNet of height is built with each hidden neuron activated by NestNet height < 1. When s 1, a NestNet degenerates to standard net- work with a two-dimensional architecture. It is proved by construction that height-s ReLU NestNets with (n) parameters can approximate 1-Lipschitz continuous functions on [0, 1]d with an error ((n-(*+1)44) while the optimal approximation error of standard ReLU networks with (n) parameters is O(n -2/4). Further- more, such result extended generic continuous functions on [o. 1] with the approximation eтror characterized by the modulus of continuity. Finally, we use numerical experimentation show the advantages of the super- super-approximatio power of ReLU NestNets.
Introduction
In this paper, design new neural network architecture by introducing one more dimension, called height, in addition to width and depth in the characterization of dimensions of neural networks. We call neural network architectures with height, width, and depth as hyper-parameter architectures. It is proved by construction that neural networks with three- -imeeeidimiiinall architectures improve the approximation power significantly, compared to standard networks with two-dimensional architectures (those with only width and depth as hyper-parameteen standard neural networks has been widely studied in recent years. The optimality of the approximation of standard fully-connected rectified linear unit (ReLU) networks (e.g., [35, 40, 49, 52]) implies limited room for further improvements. This motivates us to design new neural network architecture by introducing additional dimension of height beyond width and depth.
dimenssinlii
The approximation power of
Conesponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

We will focus on the ReLU (max{ 0, x)) activation function and use it to demonstrate our ideas. Our new network architecture is constructed recursively via a nested structure, and hence we call a neural network with the new architecture nested network (NestNet). A NestNet of height is built with each the case of 8 a NestNet degenerates to standard network with two-dimensional architecture. Let us use a simple example to explain the height of a NestNet. We say a network is activated by 01."'", 2r- if each hidden neuron of this network is activated by one of @i,"',, 2r. Here, 21,"",, 2r are trainable functions mapping IR to IR Then, a network of height & 2 can be regarded as a (ei,"., @r -activaaee network, where !i,'.,, er (realized by) networks of height & 1. See an example of a height-2 network in Figure 1. The network therein can be regarded as a (01: (2) -activated network, where @1 and @ are (realized by) networks of height 1 (i.e., standard networks). The number of parameters in the network of Figure 1 is the sum of the numbers of parameters in Li, L2 and @ 02-
hidden neuron activated by a NestNet of height 8- 1.
ss hidleen layer
2 nd hidden layer
output layer
Figure I: An example of a network of height 2, where and 0 are (realized by) networks of height 1 (i.c., standard networks). Here, Co. Li and L2 are affine linear maps.
We remark that NestNet can be regarded as a sufficiently large standard network by expanding all of its sub-network activation functions. We propose the nested network architecture since it shares the parameters via repetitions of sub-network activation functions. In other words, NestNet can provide special parameter-sharing scheme. This is the key reason why the NestNet has much better approximation power than the standard network. If we regard the network in Figure as a NestNet of height 2, then the number of parameters is the sum of the numbers of parameters in Co, L1, L2 and @1, Q2. However, if we expand the network in Figure to a large standard network, then the number of parameters in @1 and @ will be added many times for computing the total number of parameters.
Next, let us discuss our new network architecture from the perspective of dlype--paramet the network architecture with only width as a hyper-parameter one-dimensional architecture. Its depth and height are both equal to one. Neural networks with this type of architecture are generally called shallow networks. See an example in Figure 2(a). We call the network architecture with only width and depth as hyper-paue parerpaareetess wwo-dimensional architecture. Its height is equal to one. Neural networks with this type of architecture are generally called deep networks. See an example in Figure 2(b). We call the network architecture with height, width, and depth as soper-parameten three-dimensional architecture, which is proposed in this paper. Neural networks with this type of architecture are called NestNets. See an example in Figure 2(c). One may refer to Table 1 for the approximation power of networks with these three types of architectures discussed above.
We call
Table 1: Comparison for the approximation error of 1 1-Lipschitz continuous functions on [0, 1]d approximated by ReLU NestNets and standard ReLU networks.
dimension(s)
*panameters
approximation error
remark
reference
onc-tiideen-laye oetwork
varies (depth height= I)
O(v)
x=d d'
combination
deep network
width and depth vary (height= 1)
(m)
264
composition
[35, 40, 49,
NostNet of beight a
width, depth, and boight vary
((n) xy (y1
nested composition
this paper
Our main contributions are summarized as follows. We first propose a three-dimensional neural network architecture by introducing one more dimension called height beyond width and depth. We show that neural networks with three-dimensional architectures are significantly more expressive than standard networks. In particular, we prove that height-s ReLU NestNets with (n)) parameters can approximate 1-Lipschitz continuous functions on [0. 1] with error O(n ((==(d))) which is much better than the optimal error (n-2/d) of standard ReLU networks with ((n) parameters. In

---

(a)
(b)
000
(c)
(d)
Figure 2: Illustrations of neural networks with one-, two-, and three-dimensional architectures. (a) One-dimensional case (width 3, = heith= 1). (b) Two-dimensional case (width depth= height= 1). (c) thrce-dimensional case (width = depth=height=3)) 3).. (d) Zoom-in of an activation = function of the network in (c). The network (d) can also be regarded network of height 2.
the case of s + 2 d, the approximation error bounded by On (++1)//d))) s O(n which means we overcome the curse of dimensionality, Furthermore, we extend our result to generic continuous functions with the approximation error characterized by the modulus of continuity. See Theorem 2.1 and Corollary 2.2 for more details. Finally, conduct simple experiments to show the numerical advantages of the super- approximation power of ReLU NestNets.
The rest of this paper is organized as follows. In Section 2, we present the main results, provide the ideas of proving them, and discuss related work. The detailed proofs of the main results are placed in the appendix. Next, we conduct experiments to show the advantages of the super-approximation power of ReLU NestNets Section 3. Finally, Section concludes this paper with short discussion.
2 Main results and related work
In this section, we first present our main results and discuss the proof ideas. The detailed proofs of the main results are placed in the appendix. Next, we discuss related work from multiple perspectives.
2.1 Main results
We use AN&(() for n, E to denote the set of functions realized by height-s ReLU NestNets with most n parameters. We will give the mathematical definition of (n). We first discuss some notations regarding affine linear maps. We use L to denote the set of all affine linear maps, i.e.,
L C(x) Wx b. W E Rddd b Rd d1 (
Let #L denote the number of parameters in L. i.e.,
#C (d1 1)d2 if [(x)= Wx + for W Badd and b dd
We use (ei. " 2k) to denote an activation function vector, where @ IR - IR is an activation function for each i (1,
.., k). When ý= (01...., is applied to a vector input x (I,,).,))),
(ax) (#1), ", ex(2x) for any x (I1, xx) R.
Let set(ī) denote the function set containing all entries (functions) in g. For example, g (0r, L2, B3, C2 @1). ) then set(ğ) (ei, 02, 03).

---

n, s N recursively, we first consider the degenerate case. Define To define [n) n,
[N {idg, : :NV for n2, n
where idæ IR + IR is the identity map. That is, we regard the identity map and ReLU as height-0 ReLU NestNets with n parameters or as height-s ReLU NestNets with 0 parameters.
Next, let us present the recursive step. For n. E N", (vector-valued) function Φ <N) has the following form:
Φ= = m o Im o ... o i og Co. where Co. Lm L are affine linear maps. Moreover, Equation (1) satisfies the following two conditions:
(1)
Condition activation functions:
U (e.) (-1.-.,,r) er} and 2 e ANGt() for = 11,,,,...
(2)
Here, 0s is an activation function vector for each e {11. {1...,m)) All entries in ",. Fm form an activation function set (e1.-.,-3) For each j E (1.....)) 2j can realized by height-i NestNet with & nj parameters for some i = ig s 1. This condition means each hidden neuron activated by a NestNet lower height.
Condition the number of parameters:
m Σ 2=
#C
Σ
ny S n.
(3)
This condition means the total number of parameters is no more than n. The total number of parameters is calculated by adding two parts. The first is the number of parameters in affine linear maps Co, L. The other part is the number of parameters in the activation set (e1,"", 2r) formed by the entries in activation function vectors ",,"r
Then, with two conditions in Equations (2) and (3), we can define [N for n, s N* as follows:
ANs(n)= Φ: x= C. I.
L1 Φ1 o Co, Co. Fm L.
U
set((:) (e1."",r)
f=1
Ü t=0
ANN(nj) for j- 1,
Σ
#C; Σ n
ej
j-1
We remark that, in the definition above, m can be equal to 0. In this case, the function & degenerates an affine linear map.
In the NestNet example in Figure 1, the function & therein in Unar (N2(2n and the activation function vectors Φ1 and ÿ2 can be represented as
Ø1 (e1, @1, 21) and To (e2, @1 01 02 o2).
Moreover, the activation function set containing entries in 인 and da is a subset UneN NN1 (n). ie., {ei, e2) Unen A
Let C[[0, iJd) denote the set of continuous functions on [0, 1jd. By convention, the modulus of continuity of continuous function f C([0, is defined as
(r) sup (1f(z)) - [f))) : Ix 2l2 s x, y é [o, 1]4) for any r
Under these settings, we can find a function in .NN, (O))) to approximate f C(C,0,11 with an approximation error Ow (nwy(nn(e-1))))) (e=1)/d) shown in the main theorem below.
Theorem 2.1. Given a continuous function f C((O, 1] J for any n, s e N+ and p [1, o0], there exists e (N, 1)} such that
| 10-flLe((o.114)) $ 7 d (n (o+1)/d))
where C 10 if [1, 00) and C., 10 of 7)2 =0

---

We remark that the constant C..d in Theorem 2.1 is valid for all é N As we shall see later, C..d can be greatly reduced one only cares about large n N Generally, it challenging simplify the approximation error in Theorem 2.1 to make it explicitly depend wy()). However, the approximation error can be simplified to an explicit one depending on n in the case of special target function spaces like Hölder continuous function space. To be exact, if f is Hölder continuous function on [0, 1J order a (0, 1] with Hölder constant > 0, then
due to the complexity of
If(=) f(y) s A| 2l12 for any æ. y [00,]
This means can get an exponentially small approximation error 7Av dn -s+1)o/d as shown in Corollary 2.2 below.
implying wy(r) dr for any
Corollary 2.2. Suppose f is Hölder continuous function on [0,1] of order (0,1) with Hölder constant X> O. For any n, N+ and E [1, there exists & NNN 1)] such that
| fll 7A (s+1)a/d_
where
() ifp oo) and C. 104-0 (f 7)² ifp=
In Corollary 2.2, a i... f is a Lipschitz continuous function with a Lipschitz constant > O, then the approximation error can be further simplified to 7Av See Table for the comparison of the approximation error of 1-Lipschitz continuous functions on [0, 1]d approximated by ReLU NestNets and standard ReL.U networks.
Sketch of proving Theorem 2.1
We will discuss how to prove Theorem 2.1. Given target function f C([0, 1)"), the key point is to construct an almost piecewise constant function realized by a ReLU NestNet to approximate well except for a small region. Then we can get the desired result by dealing with the approximation in this small region. We divide the sketch of proving Theorem 2.1 into three main steps.
1. First, we divide [0, 1] into a union of cubes (Q))3((01..-K-11)4 and a small region Ω with K = (+) Each Qa is associated with representative 2g Qa for each vector index B. See Figure 3 for an illustration for K = 4 and d = 2.
Next, we design vector-valued function 1 (x) to map the whole cube Qa to its index & for each B. Here, 4 can be defined/construucted via
T (=)) [dl (21), (1(2), Φ (xa))"
where each one-dimensional function Φi step function outside small region. We can efficiently construct ReLU NestNets with the desired size to approximate such an almost step function Φe with sufficiently many "steps" by using the composition architecture of ReLU NestNets. See the appendix for the detailed construction.
Finally, we need to construct a function d2 realized by a ReLU NestNet to map 33 approximately to (xxz for each B {0, 1, K - 1)a. Then we have
Φ (x) (g) f(xg) f(x)
for any æ Qa and each G
implying
02 4
[0,1]4.
Then, we can get good approximation on [0. 1] by using Lemma 3.4 of our previous paper [24] to deal with the approximation inside Ω. We remark that, in the construction of Φ2 Rdd - R. we only need to care about the values of Φ2 at a set of Kd points 1, ..., K 1]d As we shall see later, this is the key point to ease the design of ReLU NestNet with the desired size realize Φ2.
See Figure 3 for illustration of the above steps. Observe that in Figure 3, we have
<(x) Φ2 11 (23
f(xg) 3 f(x)
for any x e Q and each 3 {0, 1,.,, K-13d. That means é- f is bounded by 6, + Eg on [0, 12. For any æ Q/ and each B, we have
| 2112 Vd/K
1f(xz) f(x) wy w/(V//K)
(V
5

---

11(x))
set of d-dimensional indices:
1,00
Q
2.3
for Qa
G e (0, 1.,,,.,, -)
0.75
| (2())= N f(x))
2
Q
>
2
0.30
Qhi1
?
Q1
Qu
set of function values at representatives:
0.25
Qu0
2.3
Qui
Q.p
{f(w)) g (0.1, 1)d)]
0.00
0.25
1.00
Figure 3: An illustration of the ideas of constructing é d2 D1 to approximate f for K = 4 and d = Note that @ outside Q since d(x) 22 d (+)))))))))))))))))) = (2 ( f(xg) f(x) for any æ Q/3 and each E {0, 1..,,...4 K 111
The upper bound of & is determined by the construction of d2 Rd RR As stated previously, we only need to care about the values of (22 at set of Kd points {0, 1..,,,K 1]d Rd which gives us much freedom to control 61. As we shall see later, 61 can be bounded by O (w (Vd/K)). Therefore, f is controlled by ((wr(Vd/K) outside Ω from which deduce the desired approximation error on [o, 1J since K = On (s+1)/d) Finally, by using Lemma 3.4 of our previous paper [24] to deal with the approximation inside Q we can get the desired approximation error on [0, 1]
2.3 Related work
We first compare our results with existing ones from approximation perspective. Next, discuss the parameter-sharing schemes of neural networks. Finally, we connect our NestNet architecture to existing trainable activation functions.
Discussion from an approximation perspective
The study of the approximation power of deep neural networks has become an active topic in recent years. This topic has been extensively studied from many perspectives, e.g., in terms of combinatories
[27]. topology [7], information theory [29]. fat-shattering dimension [1, 21]. Vapnik- Chinkkhhervoneniis
(VC) dimension [6, 14, 32], classical approximation theory [3, 4, 8, 9, 10, 11, 12, 13, 18, 22, 24, 25, 28, 34, 35, 38, 39, 42, 48, 49, 52, 53], etc. To the best of our knowledge, the study of neural network h-idden-layerr networks and deep networks.
approximation has two main stages: shallow
In the early works of neural network approximation, the approximation power of shallow networks is investigated. In particular, the universal approximation theorem [11, 17, 18], without approximation error estimate, showed that a sufficiently large neural network can approximate target function certain function space arbitrarily well. For one-hidden- layer neural networks of width n and sufficiently smooth functions, an asymptotic approximation error (nn-1/2 in the - is proved in [4, 5], leveraging an idea that is similar to Monte Carlo sampling for high- -igh--imensional integrals.
Recently, large number of works focus the study of deep neural networks. It is shown in [35, 49, 52] that the optimal approximation error is 2(dd by using ReLU networks with parameters approximate 1-Lipschitz continuous functions on 1] This optimal approximation error follows a natural question: How can we get a better approximation error? Generally, there two ideas to get better errors. The first one is to consider smaller function spaces, e.g., smooth functions [24, 50] and limited functions [26]. The other one to introduce new networks, e.g., Floor- ReLU networks [36]. Floor- Aioor-Exponentia--Stee (FLES) networks [37], and (Sin, ReLU, aa)-activteed networks [20].
This paper proposes three- dineeesinnnio neural network architecture by introducing one more dimension called height beyond width and depth. As shown in Theorem 2.1 and Corollary 2.2, neural networks with three-dimensional architectures are significantly more expressive than the ones with two-dimensional architectures. We will conduct experiments to explore the numerical properties of NestNets in Section 3.

---

Discussion from a parameter-sharing perspective
As discussed previously, our NestNet architecture can be regarded as sufficiently large standard network architecture with a specific parameter-sharing scheme. Parameter-sharing schemes are used in neural networks to control the overall number of parameters for reducing memory and communication costs. There are two common parameter-sharing schemes for a neural network. The first scheme is to share parameters in the same layer. A typical network example with this scheme is the convolutional neural network (CNN). In CNN architectures, filters in CNN layer are shared for all channels, which means the parameters in the filters are shared. The second scheme is share parameters across different layers of networks, c.g., recurrent neural networks.
In the NestNet architecture, share parameters via repetitions of sub-network activation functions. Both of aarameter-sharing schemes discussed just above are used in the NestNet architecture. The nested architecture of NestNets gives much freedom to determine how many parameters to share. Beyond parameter-sharing schemes for a neural network, there are also parameter-sharing schemes among different neural networks or models, especially for multi-task learning. One may refer to [30, 33, 44, 45, 46, 51] for more discussion on parameter sharing in neural networks.
Connection to trainable activation functions
The key idea of trainable activation functions is add a small number of trainable parameters to existing activation functions. Let us present several existing trainable activation functions as follows.
A ReLU-like function is introduced in [15] by modifying the negative part of ReLU using trainable
x if 2
parameter a, i.e., the parametric ReLU (PReLU) is defined as PRcLU(x) <xa if < O. A variant of ELU unit is introduced in [43] by adding two trainable parameters 3, > 0, i.e., the parametric if x: 2 ELU (PELU) is given by PELU(=)) = (3(exp(x/1) 1) if 0.
[3/y
Authors in [31] propose a type of
flexible ReLU (FReLU), which is detined via FReLU(z) == ReLU(x+a)+ B, where and are two trainable parameters. One may refer to [2] for a survey of modern trainable activation functions. To the best of our knowledge, most existing trainable activation functions can be regarded as parametric variants the original activation functions. That is, they are attained via parameterizing the original activation functions with small number of (typically 1 or 2) trainable parameters.
By contrast, activation functions in our NestNets are much more flexible. They can be (realized by) either complicated or simple sub-NestNets. That is, we can freely determine the number of parameters in the activation functions of NestNets. other words, in NestNets, we can randomly distribute the parameters in the affine linear maps and activation functions. In short, compared to the networks with existing trainable activation functions, our NestNets are more flexible and have much more freedom in the choice of activation functions.
Experimentation
In this section, we will conduct experiments as a proof of concept to explore the numerical properties of ReLU NestNets. It is challenging tune the hyper-parameters of large NestNets due to their nested architectures. Thus, our experimentation focuses on relatively small NestNets of height 2 and introduce simple sub-network activation function g. which is realized by trainable one-hidden-layer ReLU network of width 3. To be exact, is given by
e(x) = w w (xwo bo) + bj for any x E I.
(4)
where Wo W1. bo R23 and 61 E R are trainable parameters. There are 10 parameters in p. The initial settings for e in our experiments are wa (1, wi 11), by = (-0.2,-0.1, 0.0), and applications if proper
by = O. We believe that NestNets can achieve good results in some optimization algorithms are developed for NestNets. this paper, we only consider two classification problems: a synthetic classification problem based on the Archimedean spiral in Section 3.1 and an image classification problem corresponding to a standard benchmark dataset Rashion-MNIS"" [47] in Section 3.2. We remark that classification function can be continuously extended to IR if each class of samples are located in bounded closed subset of &Rd and these subsets are pairwise disjoint. That means we can apply our theory to classification problems.
7

---

3.1
Archimedean spiral
We will design a binary classification experiment by constructing two disjoint sets based on the Archimedean spiral, which can be described by the equation r a be in polar coordinates (r, 6) for given a, b E F. Let us first define two curves (Archimedean spirals) as follows:
ट, = (ax v) x r, cos 0, y T;
T; sin 0, T; = 0; + b,0, 0 [0, [0,
for = 0, where Ag = 0, 01 1, bn = b 1/x,, and & = 30. To simplify the discussion below, we normalize E₁ as C, S [0, 1] where C; is defined by
C, = (z. v) 2( y= 2(++2)
21 (T, E C.
for i= 0, Then, we can define the two desired sets as follows:
S == (u, v) V x) (u y)a s,, (I, E C:
for i 0, 1, where e 0.005 in our experiments. See an illustration for So and S in Figure 4.
Inpa
ArtFun
Norch
Bour
Art
FG!
05
FC
Art
Tlat Norm
Botrd
Art
FC!
0.0
FC
Fottch Narm
S-Stmaee
Output
Figure 4: An illustration for Sq and St.
Figure 5: A network architecture illustration.
To explore the numerical performance of NestNets, we design NestNets and standard networks to classify samples in So U.. We adopt four-hidden-layer fully connected network architecture of width 20, 35, or 50. To make the optimization more stable, we add the layers of batch normalization [19]. See Figure for illustration of the full network architecture. In Figure 5, FC and ActFun are short of fully connected layer and activation function, respectively. ActFun is ReLU for standard networks, while for NestNets, ActFun the learnable sub-network activation function e given in Equation (4).
Before presenting the experiment results, let us present the syper-parameters for training the networks mentioned above. For each E {0, we randomly choose 3 10t training samples and x 10 test samples in S; with label i. Then, we these 6 x 10 training samples to train the networks and use these 6 x 10* test samples to compute the test accuracy. We use the cross-entropy loss function to evaluate the loss between the networks and the target classification function. The number of epochs and the batch size are set to 500 and 512, respectively. We adopt RAdam [23] as the optimization method. In epochs 5(i 1) + 5i for 1, 2, ..., 100, the learning rate is 0.2 0.002 0.9*- for the parameters in e and 0.002 x 0.9i-1 for all other parameters. We remark that all training (test) samples are standardized before training, i.e., we rescale the samples to have a mean of and standard deviation of 1.
Finally, let us present the experiment results to compare the numerical performances of NestNets and standard networks. We adopt the average of test accuracies in the last 100 epochs as the target test accuracy. As we NestNets achieve much better test accuracies than standard networks though slightly more training time is required. In an "unfair" comparison, the test accuracy attained by the NestNet with 1.4 10 parameters is still better than that of the standard network with 7.9 x 10 parameters. This numerically verifies that the NestNet has much better approximation power than the standard network.
from Table 2 and Figure by adding 10 more parameters (stored in Ø).
3.2 FaSHion-MNIST
We will design convolutional neural network (CNN) architectures activated by ReLU or the sub- network activation function given in Equation (4) to classify image samples in FaSHi [47].

---

Table 2: Test accuracy comparison.
width
depth
#parameters
activation function
training time
test accuracy
standard network
20
1
1362
ReLU
2532 s
0.738290
NestNet
sub-network (g)
0.873631
standard network
35
4
3957
ReLU
2595
0.816048
NestNet
35
4
3957 +
sub network (g)
4104 s
0,995962
standard network NestNet
79022
ReLU
0,866118
50
4
7902 + 10
suh netwyork (g)
4218
0,999984
1.0
1:0
0.9
aeanney
0.5
accuraee
0.3
0.0
0.6-
0.6
0.4
Now
0.4
Nea Net
0,
Now
spach
epoch
(a) Width 20.
(b) Width 35.
(c) Width=50.
Figure 6: Test accuracy epochs.
This dataset consists of a training set of 6 10 samples and a test set of 10* samples. Each sample 28 28 grayscale image, associated with label from 10 classes. To compare the numerical performances of NestNets and standard networks, we design a standard CNN architecture and a NestNet architecture that is constructed by replacing few activation functions of standard CNN network by the sub-network activation function g. For simplicity, we denote the standard CNN and the NestNet as CNNI and CNN2. To make the optimization more stable, add the layers of dropout [16, 41 and batch normalization [19]. See illustrations of CNNI and CNN2 in Figure 7. We present more details them in Table 3.
Inpect
BatchNona
Class
BatcaNorma
Clear
Butltr
XGeePool
Dispost
Batcl-Nierr
FC
BarkNoon
Softroe
Outpor
Dospost
None
FC
BatcaNorta
Softmax
Output
(a) CNNI.
(b) CNN2.
Figure Illustrations of CNNI and CNN2. Conv and FC represent convolutional and fully connected layers, respectively. CNN2 is indeed a NestNet of height 2.
Table 3: Details of CNNI and CNN2.
layers
activation function
ouiput size of each layer dropout
batch normalization
CNNI
CNN2
28
Come::: 1: 1x (8x 3), 12
ReLU
SubNer
ReLU, 11 (20
(26 26)
12 (26 26)
yes
Com-2: 12x (3 33) 12
ReLU
SubNet (e).
ReLL.
1x4 24) 11x (24x 24)
1725 (MaxPPool & Flatten)
0.25
SubNet ReLU,
FC-1:
Rel.U
0.5
10 (Softmax)
yes
culput e 2.
Before presenting the numerical results, let us present the hyper parameters for training two CNN architectures above. We use the cross-entropy loss function to evaluate the loss between and the target classification function. The number of epochs and the batch size are set to 500 and 1 128, respectively. We adopt RAdam [23] as the optimization method and the weight decay of the optimizer is 0.0001. In epochs 5(i 1) 1 to 5i for 1,2, .., 100, the learning rate is 0.2 0.002 x 0.9
CNNs

---

for the parameters in and 0.002 x 0.9*1 for all other parameters. All training (test) samples in the Fashion- Fashion-MNIST to have mean of O and standard deviation of 1. In the settings above, we repeat the experiment 18 times and discard 3 top-pers nop-performing and bottom-perriiie settommperffrmiigg trials by using the average of test accuracy in the last 100 epochs as the performance criterion. For each epoch, we adopt the average of test accuracies in the rest 12 trials as the target test accuracy.
MNIST dataset are standardized in our experiment, i.c., we rescale all training (test) samples
Next, let us present the experiment results to compare the numerical performances of CNNI and CNN2. The test accuracy comparison of CNN1 and CNN2 is summarized in Table 4.
Table 4: Test accuracy comparison.
training
largest accurncy average largest accuracies average accuracy 100 epochs
CNNI
5802
0,925290
0.924796
0.924447
CNN2
7217
0,926620
0,926287
0.926032
For each of CNN1 and CNN2, we present the training time, the largest test accuracy, the average of the largest 100 test accuracies, and the average of test accuracies the last 100 epochs. For an intuitive comparison, we also provide illustrations of the test accuracy over epochs for CNNI and CNN2 in Figure 8. As we can see from Table and Figure 8, CNN2 performs better than CNNI though slightly more training time and 10 more parameters are required. This numerically shows that the NestNet significantly more expressive than the standard network.
0.9
0.970
w
aeruacy
mmmn
0.90
0.921
CNN1
CNN)
CNN2
0.80
160
200
300
10C
0.970
1.0
430
480
050
450
300
epoch
epoch
(a) Epochs 1-400.
(b) Epochs 401-500.
Figure 8: Test accuracy over epochs.
4
Conclusion
This paper proposes a three-dimensional neural network architecture by introducing one more dimension called height beyond width and depth. We show by construction that neural networks with faree-dimensiome architectures are significantly more expressive than the ones with two-dimensional architectures. We use simple numerical examples to show the advantages of the super-approxiimatin power of ReLU NestNets, which is regarded as proof of possibility. It would be of great interest to further explore the numerical performance of NestNets to bridge our theoretical results to applications. We believe that NestNets can be further developed and applied to real-world applications.
We remark that our analysis is limited the ReLU activation function and the (Hölder) continuous function space. It would be interesting to generalize our results to other activation functions (e.g., tanh and sigmoid functions) and other function spaces (e.g, Lebesgue and Sobolev spaces).
Acknowledgments
Z. Shen was supported by Distinguished Professorship of National University of Singapore. H. Yang was partially supported by the US National Science Foundation under award DMS-2244988, DMS- 2206333, and the Office of Naval Research Award N00014-23-1-2000
10

---

References
[1] Martin Anthony and Peter L. Bartlett. Neural Network Learning: Theoretical Foundations. Cambridge University Press, New York, NY, USA, 1st edition, 2009.
[2] Andrea Apicella, Francesco Donnarumma, Francesco Isgrò, and Roberto Prevete. A survey on modern trainable activation functions. Neural Networks, 138:14 32, 2021.
[3] Chenglong Bao, Qianxiao Li, Zuowci Shen, Cheng Tai, Lei Wu, and Xueshuang Xiang, Approximation analysis of convolutional neural networks. Semantic Scholar e-Preprint, page Corpus ID: 204762668, 2019.
[4] Andrew R. Barron. Universal approximation bounds for superpositions of sigmoidal function. IEEE Transactions on Information Theory, 39(3):33--94,, May 1993.
[5] Andrew R. Barron and Jason M. Klusowski. Approximation and estimation for high-dimensional deep learning networks. arXiv e-prints, page arXiv: 809.03090, September 2018.
[6] Peter Bartlett, Vitaly Maiorov, and Ron Meir. Almost linear VC-dimension bounds for piecewise polyno- mial networks. Neural Computation, 10(8):2159-2173, 1998.
[7] Monica Bianchini and Franco Scarselli. On the complexity of neural network classifiers: A comparison between shallow and deep architectures. IEEE Transactions on Neural Networks and Learning Systems, (000:15533-5555555 Aug 2014.
[8] Helmut. Böleskei, Philipp. Grohs, Gitta. Kutyniok, and Philipp. Petersen. Optimal approximation with sparsely connected deep neural networks. SIAM Journal on Mathematics of Data Science, 1(1):8 45, 2019.
[9] Minshuo Chen, Haoming Jiang. Wenjing Liao, and Tuo Zhao, Efficient approximation of deep ReLU networks for functions on low dimensional manifolds. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, F. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019.
[10] Charles K. Chui, Shao-Bo Lin, and Ding Xuan Zhou. Construction of neural networks for realization of localized deep learning, Frontiers in Applied Mathematics and Statistics, 4:14, 2018
[11] George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals, and Systems, 1:300-314, 1989.
[12] Rémi Gribonval, Gitta Kutyniok. Morten Nielsen, and Felix Voigtlaender. Approximation spaces of deep neural networks. Constructive Approximation, 55:259-366 36799-3774260 2022.
[13] Ingo Gühring, Gitta Kutyniok, and Philipp Petersen.
Error bounds for approximations with deep ReLU neural networks in W norms. Analysis and Applications, 18(05):803-859, 2020.
[14] Nick Harvey, Christopher Liaw, and Abbas Mehrabian. Nearly-tight VC-dimension bounds for piecewise linear neural networks. In Satyen Kale and Ohad Shamir, editors, Proceedings of the 2017 Conference on Learning Theory, volume 65 of Proceedings of Machine Learning Research, pages (064-1068. Amsterdam, Netherlands, 07-10 Jul 2017. PMLR.
[15] Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 1026-1034, 2015.
[16] Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever. and Ruslan Salakhutdinov. Im- proving neural networks by preventing adaptatittnnnnno of feature detectors. CoRR, abs/1201 00580000,.000 2012.
[17] Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural Networks, 4(2):511- 257, 1991.
[18] Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural Networks, 2(5):359-36 -66, 1989.
[19] Sergey loffe and Christian Szegedy. Batch normalization: Accelcrating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on International Conference on Machine Learning Volume 37, ICML'15 15, page 448-456 JMgR.org, 2015.
[20] Yuling Jiao, Yanming Lai, Xiliang Lu, Fengru Wang. Jerry Zhijian Yang, and Yuanyuan Yang. Deep neural networks with ReLU-Sine Le-----in--Epppeeee Exponential activations break curse of dimensionality on hölder class. arXiv e-prints, page arXiv:2103, 100054, February 2021.
11

---

[21] Michael J. Kearns and Robert E. Schapire. Efficient distribution-free learning of probabilistti concepts. J. Comput. Syst. Sci., 48(3):464-497, June 1994.
[22] Qianxiao Li, Ting Lin, and Zuowei Shen. Deep learning via dynamical systems: An approximation perspective. Journal of the European Mathematical Society, to appear.
[23] Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. In International Conference on Learning Representations, 2020.
[24) Jianfeng Lu, Zuowei Shen, Haizhao Yang, and Shijun Zhang. Deep network approximation for smooth functions. SIAM Journal on Mathematical Analyvis, 5365):565-5556, 2021.
[25] Hadrien Montanelli and Haizhao Yang. Error bounds for deep ReLU networks using the Kolmogorov- Arnold superposition theorem. Neural Networks, 129:1-6, 2020.
[26] Hadrien Montanelli, Haizhao Yang, and Qiang Du. Deep ReLUU networks overcome the curse of dimen- sionality for bandlimited functions. Journal of Computational Mathematics, 39(6):801 815, 2021.
[27] Guido Montufar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q Weinberger, editors, Advances in Neural Information Processing Syxtems 27, pages 2924-2932. Curran Associates, Inc., 2014.
[28] Ryumei Nakada and Masaaki Imaizumi. Adaptive approximation and generalization of deep neural network with intrinsic dimensionality. Journal of Machine Learning Research, 21(174):1-33, 2020.
[29] Philipp Petersen and Felix Voigtlaender. Optimal approximation of piecewise smooth functions using deep ReLU neural networks. Neural Networks, 108:296-330, 2018.
[30] Bryan A. Plummer, Nikoli Dryden, Julius Frost, Torsten Hoefler, and Kate Saenko. Neural parameter allocation search. In International Conference on Learning Representations, 2022.
[31] Suo Qiu, Xiangmin Xu, and Bolun Cai. Frelu: Flexible rectified linear units for improving convolutional neural networks. In 2018 24th International Conference on Puttern Recognition (ICPR), pages 1223-1228, Los Alamitos, CA, USA, aug 2018. IEEE Computer Society.
[32] Akito Sakurai. Tight bounds for the VC-dimension of piecewise polynomial networks. In Advances Neural Information Processing Syxtems, pages 323-329. Neural information processing systems foundation. 1999.
[33] Pedro Savarese and Michael Maire. Learning implicitly recurrent CNNs through parameter sharing. In International Conference on Learning Representations, 2019.
[34] Zuowei Shen, Haizhao Yang, and Shijun Zhang. Nonlinear approximation via compositions. Neural Networks, 119:74 84, 2019.
[35] Zuowei Shen, Haizhao Yang. and Shijun Zhang. Deep network approximation characterized by number of neurons. Communications in Computational Physics, 28(5):1768-1888 2020.
[36] Zuowei Shen. Haizhao Yang, and Shijun Zhang. Deep network with approximation error being reciprocal of width power of square root of depth. Neural Compulation, 1005):0005- 03 2021.
[37] Zuowei Shen, Haizhao Yang, and Shijun Zhang. Neural network approximation: Three hidden layers are enough. Neural Networks, 141:160-17: 10-166,, 173, 2021.
[38] Zuowei Shen, Haizhao Yang, and Shijun Zhang. Deep network approximation: Achieving arbitrary accuracy with fixed number of neurons. Journal of Machine Learning Research, 23(276): 60, 2022.
[39] Zuowei Shen, Haizhao Yang, and Shijun Zhang. Deep network approximation in terms intrinsic parameters. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 19934.19934. PMLR, 17-23 Jul 2022.
[40] Zuowei Shen, Haizhao Yang, and Shijun Zhang. Optimal approximation rate of ReIUU networks in terms of width and depth. Journal de Mathématiques Pures et Appliquées, 157:101-135, 2022.
[41] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overtitting. Journal of Machine Learning Research, 15(5):19929 1958, 2014.
12

---

[42] Taiji Suzuki. Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality. In International Conference on Learning Representations. 2019.
[43] Ludovic Trottier, Philippe Giguère, and Brahim Chaib-draa. Parametric exponential linear unit for deep convolutional neural networks. 2017 16th IEEE International Conference Machine Learning and Applications (ICMLA), pages 207-214, 2017.
[44] Matthew Wallingford, Hao Li, Alessandro Achille, Avinash Ravichandran, Charless Fowlkes, Rahul Bhotika, and Stefano Soatto. Task adaptive parameter sharing multi-task learning. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7551-7560, 2022.
[45] Jiaxing Wang, Haoli Bai. Jiaxiang Wu, Xupeng Shi, Junzhou Huang. Irwin King, Michael Lyu, and Jian Cheng. Revisiting parameter sharing for automatic neural channel number search. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 5991-6002. Curran Associates, Inc., 2020.
[46] Ze Wang, Xiuyuan Cheng, Guillermo Sapiro, and Qiang Qiu. ACDC: Weight sharing in atom-coefficient decomposed convolution. arXiv e-prints, page arXiv:2009 Ttt.20009003886, September 2020.
[47] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNISTE novel image dataset for benchmarking machine learning algorithms. arXiv e-prints, page atXiv:1708.07744 August 2017.
[48] Dmitry Yarotsky. Error bounds for approximations with deep ReLU networks. Neural Networks, 94; 103 114, 2017.
[49] Dmitry Yarotsky. Optimal approximation of continuous functions by very deep ReLU networks. In Sébastien Bubeck. Vianney Perchet, and Philippe Rigollet, editors, Proceedings of the 31st Conference Learning Theory, volume 75 Proceedings of Machine Learning Research, pages 639-649, 649. PMLR. 06-09 2018
[50] Dmitry Yarotsky and Anton Zhevnerchuk. The phase diagram of approximation rates for deep neural networks. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 13005-13015. Curran Associates, Inc., 2020.
[51] Lijun Zhang, Qizheng Yang, Xiao Liu, and Hui Guan. Rethinking hard-parameter sharing in multi-domain lcarning. In 2022 IEEE International Conference on Multimedia and Expo (ICME), pages 01 01-06, 2022.
[52] Shijun Zhang. Deep neural network approximation via function compositions. PhD Thesis, National University of Singapore, 2020. URI.: http::ttc///cchhaaabaiii bank. nus edu. atg/handle/10663 /86604
[53] Ding- Xuan Zhou, Universality of deep convolutional neural networks. Applied and Computational Harmonic Analysis, 48(2):787-799 2020.
13


Reviews:
[{"review_id": "pBjei96nLSJ", "reviewer": "Reviewer_wtW7", "summary": "The paper presents a novel three-dimensional NN architecture. An additional dimension called height is introduced to empower the capacity of neural networks. A simple 257 experiment demonstrates the numerical advantages of the proposed method. The proof seems to be solid. However, I am not an expert in theory so I cannot judge the contribution of the proof. Overall, I think the idea of the paper is novel.", "questions": "See Weakness", "limitations": "There is no computational cost analysis and comparison, or experiments on real-world datasets.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The three-dimensional NN architecture is a novel concept. The authors present theoretical proof to show their advantage with ReLu activations. The proof seems to be solid.\n\nWeaknesses\n- Although the method does not add extra parameters, the proposed three-dimensional NN significantly introduces an extra computational burden to iteratively activate the neurons (e.g., FLOPs during inference). It is good to present the computational burden of training and inference, and compare the performance against the computational burden.\n- The results of commonly-used benchmarks are preferred such as mnist, cifar, and ImageNet.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "efkq0obhlCt", "reviewer": "Reviewer_zWh2", "summary": "The paper proposes a new family of neural architectures which is given not just by width and depth but also by 'height', a newly introduced dimension. Such models are allowed to have activations that are themselves realized by other networks, resulting in an implicit parameter sharing scheme. Results on approximation theory are shown for such family of models, where they have better approximation properties than standard 'width/depth' networks.", "questions": "Suggestions are listed in the previous section (Strengths And Weaknesses).\n\nMy main question is regarding the practical aspects of NestNets. The construction used to prove Theorem 2.1 seems to have an effective depth of approximately $3 n^{s+1}$, where by 'effective depth' I mean the diameter of the unrolled computational graph of the network. For n=10 and s=5, this results in a model that requires 3 million sequential operations to compute its forward pass (i.e. evaluate the model on a batch of samples), which is on a completely different scale than the depth of 'standard' networks used in practice and simply impractical. Is it the case that the approximation power of NestNets indeed rely on networks with an effective depth that outscales the number of parameters (as n and s increase) and even a single forward pass becomes unfeasible, or is the construction used for Theorem 2.1 extremely inefficient in terms of effective depth? I understand if a conclusive answer is not possible at this time but some I'd like to know the authors' thoughts on this.", "limitations": "A discussion on the practical limitations would be valuable (see 'Questions' comments regarding effective depth and evaluation cost).", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is quite rigorous and the definitions are precise and formal. There is some minor repetition in the text (abstract / introduction), which could be removed to open up some space.\n\nTwo weaknesses in my opinion:\n\n1) There is no discussion / intuition / sketch of the proof of Theorem 2.1 in the main text. The construction given in the Appendix is important as it plays a major role in the theoretical results given in the paper, and a brief description of it would be valuable to have in the main text (and I believe enough space could be opened up by removing repeated text, wrapping Fig 3, etc). The organization of the Appendix could also be improved, since a reader looking for details on the construction used to prove Theorem 2.1 would go to Appendix A, then to B.1 and it is only in B.2 that the construction is given. This is a minor weakness as it is mostly regarding the organization of the manuscript, and had no impact on my final rating.\n\n2) The discussion on how NestNets relate to models that have been previously proposed and adopted could be greatly expanded and improved. As the paper states, NestNets can be seen as 'standard' networks but with a specific parameter sharing scheme (this point could be given more significance in the text, as parameter sharing seems to be the main ingredient of NestNets and the cause of their expressive power). The scheme is given by activation layers consisting of element-wise operations on the input, and a repetition of these activation layers throughout the network.\n\nAs for the first property (element-wise maps), there is a connection to convolutions which doesn't seem to be mentioned: a stack of 1d convolutions with a kernel size of 1 also results in a module that maps an input x = (x1, x2, ..., xk) to u = (f(x1,w), f(x2,w), ..., f(xk,w)) where f will be a deep network that maps R -> R. For image data, we have a similar relationship with stacks of 2d convolutions with 1x1 kernels and with stacks of depthwise separable 2d convolutions. Note that stacks of 1x1 or depthwise convolutions are used in some CNN models and, to some extent, also in transformers (the ReLU MLP blocks following attention layers can be framed as convolutions). To summarize, this type of parameter sharing has been widely explored in the literature and a more rigorous and complete discussion on it is required.\n\nFor the second property (layer repetition throughout the network), there is also a missing connection to models that re-use parameters across different layers of the network, most notably models that can be seen as hybrids between recurrent and non-recurrent networks. Some methods aim to train the recurrence scheme itself, learning how and when to re-use layer-wise parameters across a deep network. Some examples that should be discussed are:\n\n[1] - Learning Implicitly Recurrent CNNs through Parameter Sharing\n[2] - ACDC: Weight Sharing in Atom-Coefficient Decomposed Convolution\n[3] - Neural Parameter Allocation Search\n\nLastly, a discussion with Maxout would be valuable, as it can be seen as a learnable activation function but considerably more expressive than the mentioned ReLU variants.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "cb_lpt_xZiB", "reviewer": "Reviewer_7MPy", "summary": "The authors propose a novel neural network architecture that adds a new \"height\" dimension through a recursive construction. The authors show that their proposed network has better asymptotic error than standard ReLU networks when accounting for similar $O(n)$ number of parameters, and on a class on Lipschitz continuous function in $[0,1]^d$. ", "questions": "To be clear I come from an applied ML perspective, which colors my questions. My main question is understanding the value of this work. Is the value: \\\nA) Primarily theoretical in understanding a new class of NNs which can achieve better approximation error? \\\nB) Also applied in that the authors expect their proposed class of NNs to be used on modern ML tasks?\n\nIf A), then I think the work is valuable. If B), then I am concerned about the addition of hyper-parameters and the lack of experiments on any real task. ", "limitations": "The authors describe theoretical limitations in their analysis, which I think are reasonable and am unconcerned about. \n", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality:\\\n(+) To the best of my knowledge this recursive \"height\" network architecture is novel. \n\nQuality: \\\nI looked at the proof but am unfamiliar field.\n\nClarity:\\\n(+) The paper is well written and easy to understand, even for someone not up to date on theory of NN.\n\nSignificance: \\\n(+) To my limited understanding the approximation error that these proposed networks achieve is meaningfully better than standard NNs.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept