Title: THE POWER OF DEEPER NETWORKS FOR EXPRESSING NATURAL FUNCTIONS

Abstract: It is well-known that neural networks are universal approximators, but that deeper networks tend in practice to be more powerful than shallower ones. We shed light on this by proving that the total number of neurons m required to approximate natural classes of multivariate polynomials of n variables grows only linearly with n for deep neural networks, but grows exponentially when merely a single hidden layer is allowed. We also provide evidence that when the number of hidden layers is increased from 1 to k, the neuron requirement grows exponentially not with n but with n 1/k , suggesting that the minimum number of layers required for practical expressibility grows only logarithmically with n.

1. INTRODUCTION:

Deep learning has lately been shown to be a very powerful tool for a wide range of problems, from image segmentation to machine translation. Despite its success, many of the techniques developed by practitioners of artificial neural networks (ANNs) are heuristics without theoretical guarantees. Perhaps most notably, the power of feedforward networks with many layers (deep networks) has not been fully explained. The goal of this paper is to shed more light on this question and to suggest heuristics for how deep is deep enough.

It is well-known (Cybenko, 1989; Funahashi, 1989; Hornik et al., 1989; Barron, 1994; Pinkus, 1999) that neural networks with a single hidden layer can approximate any function under reasonable assumptions, but it is possible that the networks required will be extremely large. Recent authors have shown that some functions can be approximated by deeper networks much more efficiently (i.e. with fewer neurons) than by shallower ones. Often, these results admit one or more of the following limitations: "existence proofs" without explicit constructions of the functions in question; explicit constructions, but relatively complicated functions; or applicability only to types of network rarely used in practice.

It is important and timely to extend this work to make it more concrete and actionable, by deriving resource requirements for approximating natural classes of functions using today's most common neural network architectures. Lin et al. (2017) recently proved that it is exponentially more efficient to use a deep network than a shallow network when Taylor-approximating the product of input variables. In the present paper, we move far beyond this result in the following ways: (i) we use standard uniform approximation instead of Taylor approximation, (ii) we show that the exponential advantage of depth extends to all general sparse multivariate polynomials, and (iii) we address the question of how the number of neurons scales with the number of layers. Our results apply to standard feedforward neural networks and are borne out by empirical tests.

Our primary contributions are as follows:

• It is possible to achieve arbitrarily close approximations of simple multivariate and univariate polynomials with neural networks having a bounded number of neurons (see §3).

• Such polynomials are exponentially easier to approximate with deep networks than with shallow networks (see §4).

• The power of networks improves rapidly with depth; for natural polynomials, the number of layers required is at most logarithmic in the number of input variables, where the base of the logarithm depends upon the layer width (see §5).

2. RELATED WORK:

Deeper networks have been shown to have greater representational power with respect to various notions of complexity, including piecewise linear decision boundaries (Montufar et al., 2014) and topological invariants (Bianchini & Scarselli, 2014) . Recently, Poole et al. (2016) and Raghu et al. (2016) showed that the trajectories of input variables attain exponentially greater length and curvature with greater network depth.

Work including Daniely (2017) ; Eldan & Shamir (2016) ; Pinkus (1999) ; Poggio et al. (2017) ; Telgarsky (2016) shows that there exist functions that require exponential width to be approximated by a shallow network. Barron (1994) provides bounds on the error in approximating general functions by shallow networks. Mhaskar et al. (2016) and Poggio et al. (2017) show that for compositional functions (those that can be expressed by recursive function composition), the number of neurons required for approximation by a deep network is exponentially smaller than the best known upper bounds for a shallow network. Mhaskar et al. (2016) ask whether functions with tight lower bounds must be pathologically complicated, a question which we answer here in the negative.

Various authors have also considered the power of deeper networks of types other than the standard feedforward model. The problem has also been posed for sum-product networks (Delalleau & Bengio, 2011) and restricted Boltzmann machines (Martens et al., 2013) . Cohen et al. (2016) showed, using tools from tensor decomposition, that shallow arithmetic circuits can express only a measure-zero set of the functions expressible by deep circuits. A weak generalization of this result to convolutional neural networks was shown in Cohen & Shashua (2016) .

3. THE POWER OF APPROXIMATION:

In this paper, we will consider the standard model of feedforward neural networks (also called multilayer perceptrons). Formally, the network may be considered as a multivariate function

N (x) = A k σ(• • • σ(A 1 σ(A 0 x)) • • • )

, where A 0 , A 1 , . . . , A k are constant matrices and σ denotes a scalar nonlinear function applied element-wise to vectors. The constant k is referred to as the depth of the network. The neurons of the network are the entries of the vectors σ(A

• • • σ(A 1 σ(A 0 x)) • • • ), for = 1, . . . , k -1.

These vectors are referred to as the hidden layers of the network.

Two notions of approximation will be relevant in our results: -approximation, also known as uniform approximation, and Taylor approximation.

Definition 3.1. For constant > 0, we say that a network N (x) -approximates a multivariate function f (x) (for x in a specified domain

(-R, R) n ) if sup x |N (x) -f (x)| < .

Definition 3.2. We say that a network N (x) Taylor-approximates a multivariate polynomial p(x) of degree d if p(x) is the dth order Taylor polynomial (about the origin) of N (x).

The following proposition shows that Taylor approximation implies -approximation for homogeneous polynomials. The reverse implication does not hold. Proposition 3.3. Suppose that the network N (x) Taylor-approximates the homogeneous multivariate polynomial p(x). Then, for every , there exists a network N (x) that -approximates p(x), such that N (x) and N (x) have the same number of neurons in each layer. (This statement holds for x ∈ (-R, R) n for any specified R.) 

Proof. Suppose that N (x) = A k σ(• • • σ(A 1 σ(A 0 x)

(x) = ∞ i=d+1 E i (x

) is a Taylor series with each E i (x) homogeneous of degree i. Since N (x) is the function defined by a neural network, it converges for every x ∈ R n . Thus, E(x) converges, as does E(δx)/δ d = ∞ i=d+1 δ i-d E i (x). By picking δ sufficiently small, we can make each term δ i-d E i (x) arbitrarily small. Let δ be small enough that

|E(δx)/δ d | < holds for all x in (-R, R) n . Let A 0 = δA 0 , A k = A k /δ d , and A = A for = 1, 2, . . . , k -1. Then, for N (x) = A k σ(• • • σ(A 1 σ(A 0 x)) • • • ), we observe that N (x) = N (δx)/δ d ,

and therefore:

|N (x) -p(x)| = |N (δx)/δ d -p(x)| = |p(δx)/δ d + E(δx)/δ d -p(x)| = |E(δx)/δ d | < .

We conclude that N (x) is an -approximation of p(x), as desired.

For a fixed nonlinear function σ, we consider the total number of neurons (excluding input and output neurons) needed for a network to approximate a given function. Remarkably, it is possible to attain arbitrarily good approximations of a (not necessarily homogeneous) multivariate polynomial by a feedforward neural network, even with a single hidden layer, without increasing the number of neurons past a certain bound. (See also Corollary 1 in Poggio et al. (2017) .) Theorem 3.4. Suppose that p(x) is a degree-d multivariate polynomial and that the nonlinearity σ has nonzero Taylor coefficients up to degree d. Let m k (p) be the minimum number of neurons in a depth-k network that -approximates p. Then, the limit lim →0 m k (p) exists (and is finite). (Once again, this statement holds for x ∈ (-R, R) n for any specified R.)

Proof. We show that lim →0 m 1 (p) exists; it follows immediately that lim →0 m k (p) exists for every k, since an -approximation to p with depth k can be constructed from one with depth 1.

Let p 1 (x), p 2 (x), . . . , p s (x) be the monomials of p(x), so that p(x) = i p i (x). We claim that each p i (x) can be Taylor-approximated by a network N i (x) with one hidden layer. This follows, for example, from the proof in Lin et al. ( 2017) that products can be Taylor-approximated by networks with one hidden layer, since each monomial is the product of several inputs (with multiplicity); we prove a far stronger result about N i (x) later in this paper (see Theorem 4.1).

Suppose now that N i (x) has m i hidden neurons. By Proposition 3.3, we conclude that since p i (x) is homogeneous, it may be δ-approximated by a network N i δ (x) with m i hidden neurons, where δ = /s. By combining the networks N i δ (x) for each i, we can define a network N (x) = i N i δ (x) with i m i neurons. Then, we have:

|N (x) -p(x)| ≤ i |N i δ (x) -p i (x)| ≤ i δ = sδ = .

Hence, N (x) is an -approximation of p(x), implying that m 1 (p) ≤ i m i for every . Thus, lim →0 m 1 (p) exists, as desired.

This theorem is perhaps surprising, since it is common for -approximations to functions to require ever-greater complexity, approaching infinity as → 0. For example, the function exp(| -x|) may be approximated on the domain (-π, π) by Fourier sums of the form m k=0 a m cos(kx). However, in order to achieve -approximation, we need to take m ∼ 1/ √ terms. By contrast, we have shown that a finite neural network architecture can achieve arbitrarily good approximations merely by altering its weights.

Note also that the assumption of nonzero Taylor coefficients cannot be dropped from Theorem 3.4. For example, the theorem is false for rectified linear units (ReLUs), which are piecewise linear and do not admit a Taylor series. This is because -approximating a non-linear polynomial with a piecewise linear function requires an ever-increasing number of pieces as → 0. In the next section, we will show that there is an exponential gap between m uniform 1 (p) and m uniform (p) and between m Taylor 1 (p) and m Taylor (p) for various classes of polynomials p.

4. THE INEFFICIENCY OF SHALLOW NETWORKS:

In this section, we compare the efficiency of shallow networks (those with a single hidden layer) and deep networks at approximating multivariate polynomials. Proofs of our main results are included in the Appendix.

4.1. MULTIVARIATE POLYNOMIALS:

Our first result shows that uniform approximation of monomials requires exponentially more neurons in a shallow than a deep network. Theorem 4.1. Let p(x) denote the monomial

x r1 1 x r2 2 • • • x rn n , with d = n i=1 r i .

Suppose that the nonlinearity σ has nonzero Taylor coefficients up to degree 2d. Then, we have:

(i) m uniform 1 (p) = n i=1 (r i + 1), (ii) m uniform (p) ≤ n i=1 (7 log 2 (r i ) + 4),

where x denotes the smallest integer that is at least x.

We can prove a comparable result for m Taylor under slightly weaker assumptions on σ. Note that by setting r 1 = r 2 = . . . = r n = 1, we recover the result of Lin et al. ( 2017) that the product of n numbers requires 2 n neurons in a shallow network but can be Taylor-approximated with linearly many neurons in a deep network. Theorem 4.2. Let p(x) denote the monomial

x r1 1 x r2 2 • • • x rn n , with d = n i=1 r i .

Suppose that σ has nonzero Taylor coefficients up to degree d. Then, we have:

(i) m Taylor 1 (p) = n i=1 (r i + 1), (ii) m Taylor (p) ≤ n i=1 (7 log 2 (r i ) + 4).

It is worth noting that neither of Theorems 4.1 and 4.2 implies the other. This is because it is possible for a polynomial to admit a compact uniform approximation without admitting a compact Taylor approximation.

It is natural now to consider the cost of approximating general polynomials. However, without further constraint, this is relatively uninstructive because polynomials of degree d in n variables live within a space of dimension n+d d , and therefore most require exponentially many neurons for any depth of network. We therefore consider polynomials of sparsity c: that is, those that can be represented as the sum of c monomials. This includes many natural functions.

The following theorem, when combined with Theorems 4.1 and 4.2, shows that general polynomials p with subexponential sparsity have exponentially large m uniform 1 (p) and m Taylor 1 (p), but subexponential m uniform (p) and m Taylor (p). Theorem 4.3. Let p(x) be a multivariate polynomial of degree d and sparsity c, having monomials q 1 (x), q 2 (x), . . . , q c (x). Suppose that the nonlinearity σ has nonzero Taylor coefficients up to degree 2d. Then, we have:

(i) m uniform 1 (p) ≥ 1 c max j m uniform 1 (q j ).

(ii) m uniform (p) ≤ j m uniform (q j ).

These statements also hold if m uniform is replaced with m Taylor .

As mentioned above with respect to ReLUs, some assumptions on the Taylor coefficients of the activation function are necessary for the results we present. However, it is possible to loosen the assumptions of Theorem 4.1 and 4.2 while still obtaining exponential lower bounds on m uniform (r i + 1). (An even better lower bound is the maximum coefficient in the polynomial i (1 + y + . . . + y ri ).)

4.2. UNIVARIATE POLYNOMIALS:

As with multivariate polynomials, depth can offer an exponential savings when approximating univariate polynomials. We show below (Proposition 4.5) that a shallow network can approximate any degree-d univariate polynomial with a number of neurons at most linear in d. The monomial x d requires d + 1 neurons in a shallow network (Proposition 4.6), but can be approximated with only logarithmically many neurons in a deep network. Thus, depth allows us to reduce networks from linear to logarithmic size, while for multivariate polynomials the gap was between exponential and linear. The difference here arises because the dimensionality of the space of univariate degree-d polynomials is linear in d, which the dimensionality of the space of multivariate degree-d polynomials is exponential in d. Proposition 4.5. Suppose that the nonlinearity σ has nonzero Taylor coefficients up to degree d. Then, m Taylor 1 (p) ≤ d + 1 for every univariate polynomial p of degree d.

Proof. Pick a 0 , a 1 , . . . , a d to be arbitrary, distinct real numbers. Consider the Vandermonde matrix

A with entries A ij = a j i . It is well-known that det(A) = i<i (a i -a i ) = 0.

Hence, A is invertible, which means that multiplying its columns by nonzero values gives another invertible matrix. Suppose that we multiply the jth column of A by σ j to get A , where σ(x) = j σ j x j is the Taylor expansion of σ(x). Now, observe that the ith row of A is exactly the coefficients of σ(a i x), up to the degree-d term. Since A is invertible, the rows must be linearly independent, so the polynomials σ(a i x), restricted to terms of degree at most d, must themselves be linearly independent. Since the space of degree-d univariate polynomials is (d + 1)-dimensional, these d + 1 linearly independent polynomials must span the space. Hence, m Taylor 1 (p) ≤ d + 1 for any univariate degree-d polynomial p. In fact, we can fix the weights from the input neuron to the hidden layer (to be a 0 , a 1 , . . . , a d , respectively) and still represent any polynomial p with d + 1 hidden neurons.

Proposition 4.6. Let p(x) = x d , and suppose that the nonlinearity σ(x) has nonzero Taylor coefficients up to degree 2d. Then, we have:

(i) m uniform 1 (p) = d + 1. (ii) m uniform (p) ≤ 7 log 2 (d) .

These statements also hold if m uniform is replaced with m Taylor .

Proof. Part (i) follows from part (i) of Theorems 4.1 and 4.2 by setting n = 1 and r 1 = d.

For part (ii), observe that we can Taylor-approximate the square x 2 of an input x with three neurons in a single layer:

1 2σ (0) (σ(x) + σ(-x) -2σ(0)) = x 2 + O(x 4 + x 5 + . . .).

We refer to this construction as a square gate, and the construction of Lin et al. ( 2017) as a product gate. We also use identity gate to refer to a neuron that simply preserves the input of a neuron from the preceding layer (this is equivalent to the skip connections in residual nets (He et al., 2016) ).

Consider a network in which each layer contains a square gate (3 neurons) and either a product gate or an identity gate (4 or 1 neurons, respectively), according to the following construction: The square gate squares the output of the preceding square gate, yielding inductively a result of the form x 2 k , where k is the depth of the layer. Writing d in binary, we use a product gate if there is a 1 in the 2 k-1 -place; if so, the product gate multiplies the output of the preceding product gate by the output of the preceding square gate. If there is a 0 in the 2 k-1 -place, we use an identity gate instead of a product gate. Thus, each layer computes x 2 k and multiplies x 2 k-1 to the computation if the 2 k-1 -place in d is 1. The process stops when the product gate outputs x d . This network clearly uses at most 7 log 2 (d) neurons, with a worst case scenario where d + 1 is a power of 2. Hence m Taylor (p) ≤ 7 log 2 (d) , with m uniform (p) ≤ m Taylor (p) by Proposition 3.3 since p is homogeneous.

5. HOW EFFICIENCY IMPROVES WITH DEPTH:

We now consider how m uniform k (p) scales with k, interpolating between exponential in n (for k = 1) and linear in n (for k = log n). In practice, networks with modest k > 1 are effective at representing natural functions. We explain this theoretically by showing that the cost of approximating the product polynomial drops off rapidly as k increases.

By repeated application of the shallow network construction in Lin et al. ( 2017), we obtain the following upper bound on m uniform k (p), which we conjecture to be essentially tight. Our approach leverages the compositionality of polynomials, as discussed e.g. in Mhaskar et al. (2016) and Poggio et al. (2017) , using a tree-like neural network architecture.

Theorem 5.1. Let p(x) equal the product x 1 x 2 • • • x n , and suppose σ has nonzero Taylor coefficients up to degree n. Then, we have:

EQUATION

)

Proof. We construct a network in which groups of the n inputs are recursively multiplied up to Taylor approximation. The n inputs are first divided into groups of size b 1 , and each group is multiplied in the first hidden layer using 2 b1 neurons (as described in Lin et al. ( 2017)). Thus, the first hidden layer includes a total of 2 b1 n/b 1 neurons. This gives us n/b 1 values to multiply, which are in turn divided into groups of size b 2 . Each group is multiplied in the second hidden layer using 2 b2 neurons. Thus, the second hidden layer includes a total of 2 b2 n/(b 1 b 2 ) neurons.

We continue in this fashion for b

1 , b 2 , . . . , b k such that b 1 b 2 • • • b k = n,

giving us one neuron which is the product of all of our inputs. By considering the total number of neurons used, we conclude 

EQUATION

L(b i , λ) := n - k i=1 b i λ + k i=1   k j=i+1 b j   2 bi .

Figure 1 : The optimal settings for {b i } k i=1 as n varies are shown for k = 1, 2, 3. Observe that the b i converge to n 1/k for large n, as witnessed by a linear fit in the log-log plot. The exact values are given by equations ( 4) and (5).

Figure 2 : Performance of trained networks in approximating the product of 20 input variables, ranging from red (high error) to blue (low error). The error shown here is the expected absolute difference between the predicted and actual product. The curve w = n (k-1)/k • 2 n 1/k for n = 20 is shown in black. In the region above and to the right of the curve, it is possible to effectively approximate the product function (Theorem 5.1).

Differentiating L with respect to b i , we obtain the conditions 

EQUATION

EQUATION

Thus, the optimal b i are not exactly equal but very slowly increasing with i (see Figure 1 ).

The following conjecture states that the bound given in Theorem 5.1 is (approximately) optimal. Conjecture 5.2. Let p(x) equal to the product x 1 x 2 • • • x n , and suppose that σ has all nonzero Taylor coefficients. Then, we have:

EQUATION

i.e., the exponent grows as n 1/k for n → ∞.

We empirically tested Conjecture 5.2 by training ANNs to predict the product of input values x 1 , . . . , x n with n = 20 (see Figure 2 ). The rapid interpolation from exponential to linear width aligns with our predictions.

In our experiments, we used feedforward networks with dense connections between successive layers. In the figure, we show results for σ(x) = tanh(x) (note that this behavior is even better than expected, since this function actually has numerous zero Taylor coefficients). Similar results were also obtained for rectified linear units (ReLUs) as the nonlinearity, despite the fact that this function does not even admit a Taylor series. The number of layers was varied, as was the number of neurons within a single layer. The networks were trained using the AdaDelta optimizer (Zeiler, 2012) to minimize the absolute value of the difference between the predicted and actual values. Input variables x i were drawn uniformly at random from the interval [0, 2], so that the expected value of the output would be of manageable size.

Eq. ( 6) provides a helpful rule of thumb for how deep is deep enough. Suppose, for instance, that we wish to keep typical layers no wider than about a thousand (∼ 2 10 ) neurons. Eq. ( 6) then implies n 1/k ∼ < 10, i.e., the number of layers should be at least k ∼ > log 10 n.

It would be very interesting if one could show that general polynomials p in n variables require a superpolynomial number of neurons to approximate for any constant number of hidden layers.

The analogous statement for Boolean circuits -whether the complexity classes T C 0 and T C 1 are equal -remains unresolved and is assumed to be quite hard. Note that the formulations for Boolean circuits and deep neural networks are independent statements (neither would imply the other) due to the differences between computation on binary and real values. Indeed, gaps in expressivity have already been proven to exist for real-valued neural networks of different depths, for which the analogous results remain unknown in Boolean circuits (see e.g. Mhaskar (1993) ; Chui et al. (1994; 1996) ; Montufar et al. ( 2014); Cohen et al. (2016) ; Telgarsky ( 2016)).

6. CONCLUSION:

We have shown how the power of deeper ANNs can be quantified even for simple polynomials. We have proved that arbitrarily good approximations of polynomials are possible even with a fixed number of neurons and that there is an exponential gap between the width of shallow and deep networks required for approximating a given sparse polynomial. For n variables, a shallow network requires size exponential in n, while a deep network requires at most linearly many neurons. Networks with a constant number k > 1 of hidden layers appear to interpolate between these extremes, following a curve exponential in n 1/k . This suggests a rough heuristic for the number of layers required for approximating simple functions with neural networks. For example, if we want no layers to have more than 2 10 neurons, say, then the minimum number of layers required grows only as log 10 n. To further improve efficiency using the O(n) constructions we have presented, it suffices to increase the number of layers by a factor of log 2 10 ≈ 3, to log 2 n.

The key property we use in our constructions is compositionality, as detailed in Poggio et al. (2017) .

It is worth noting that as a consequence our networks enjoy the property of locality mentioned in Cohen et al. (2016) , which is also a feature of convolutional neural nets. That is, each neuron in a layer is assumed to be connected only to a small subset of neurons from the previous layer, rather than the entirety (or some large fraction). In fact, we showed (e.g. Prop. 4.6) that there exist natural functions computable with linearly many neurons, with each neuron is connected to at most two neurons in the preceding layer, which nonetheless cannot be computed with fewer than exponentially many neurons in a single layer, no matter how may connections are used. Our construction can also be framed with reference to the other properties mentioned in Cohen et al. (2016) : those of sharing (in which weights are shared between neural connections) and pooling (in which layers are gradually collapsed, as our construction essentially does with recursive combination of inputs).

This paper has focused exclusively on the resources (neurons and synapses) required to compute a given function for fixed network depth. (Note also results of Lu et al. (2017) ; Hanin & Sellke (2017) ; Hanin (2017) for networks of fixed width.) An important complementary challenge is to quantify the resources (e.g. training steps) required to learn the computation, i.e., to converge to appropriate weights using training data -possibly a fixed amount thereof, as suggested in Zhang et al. (2017) .

There are simple functions that can be computed with polynomial resources but require exponential resources to learn (Shalev-Shwartz et al., 2017) . It is quite possible that architectures we have not considered increase the feasibility of learning. For example, residual networks (ResNets) (He et al., 2016) and unitary nets (see e.g. Arjovsky et al. (2016) ; Jing et al. (2017) ) are no more powerful in representational ability than conventional networks of the same size, but by being less susceptible to the "vanishing/exploding gradient" problem, it is far easier to optimize them in practice. We look forward to future work that will help us understand the power of neural networks to learn.

• • • ± y N ), proving that m Taylor (p) ≤ n i=1 (r i + 1). Proposition 3.3 implies that for homogeneous polynomials p, we have m uniform

1 (p) ≤ m Taylor 1 (p).

We now show that this number of neurons is also necessary for approximating p(x). Suppose that N (x) is an -approximation to p(x) with depth 1, and let the Taylor series of N (x) be p(x)+E(x). Let E k (x) be the degree-k homogeneous component of E(x), for 0 ≤ k ≤ 2d. By the definition of -approximation, sup x E(x) goes to 0 as does, so by picking small enough, we can ensure that the coefficients of each E k (x) go to 0.

Let m = m uniform 1 (p) and suppose that σ(x) has the Taylor expansion ∞ k=0 σ k x k . Then, by grouping terms of each order, we conclude that there exist constants a ij and w j such that

σ d m j=1 w j n i=1 a ij x i d = p(x) + E d (x) σ k m j=1 w j n i=1 a ij x i k = E k (x) for k = d.

For each S ⊆ X, let us take the derivative of this equation by every variable that occurs in S, where we take multiple derivatives of variables that occur multiple times. This gives

EQUATION

EQUATION

Observe that there are r ≡ n i=1 (r i + 1) choices for S, since each variable x i can be included anywhere from 0 to r i times. Define A to be the r × m matrix with entries A S,j = h∈S a hj . We claim that A has full row rank. This would show that the number of columns m is at least the number of rows r = n i=1 (r i + 1), proving the desired lower bound on m. Suppose towards contradiction that the rows A S ,• admit a linear dependence:

r =1 c A S ,• = 0,

where the coefficients c are all nonzero and the S denote distinct subsets of X. Let S * be such that |c * | is maximized. Then, take the dot product of each side of the above equation by the vector with entries (indexed by j) equal to w j ( 

EQUATION

+ |(|S | =|S * |) c • |S |! σ d+|S |-|S * | • (d + |S | -|S * |)! • ∂ ∂S E d+|S |-|S * | (x)

Consider the coefficient of the monomial 9) tells us that this coefficient should be zero -a contradiction. We conclude that A has full row rank, and therefore that m uniform 1 (p) = m ≥ n i=1 (r i + 1). This completes the proof of part (i).

We now consider part (ii) of the theorem. It follows from Proposition 4.6, part (ii) that, for each i, we can Taylor-approximate x ri i using 7 log 2 (r i ) neurons arranged in a deep network. Therefore, we can Taylor-approximate all of the x ri i using a total of i 7 log 2 (r i ) neurons. From Lin et al. ( 2017), we know that these n terms can be multiplied using 4n additional neurons, giving us a total of i (7 log 2 (r i ) +4). Proposition 3.3 implies again that m uniform 1 (p) ≤ m Taylor 1 (p). This completes the proof.

7.2 PROOF OF THEOREM 4.2.

As above, suppose that r i > 0 for i = 1, . . . , n, and let X be the multiset in which x i occurs with multiplicity r i .

It is shown in the proof of Theorem 4.1 that n i=1 (r i + 1) neurons are sufficient to Taylorapproximate p(x). We now show that this number of neurons is also necessary for approximating p(x). Let m = m Taylor 1 (p) and suppose that σ(x) has the Taylor expansion ∞ k=0 σ k x k . Then, by grouping terms of each order, we conclude that there exist constants a ij and w j such that

EQUATION

For each S ⊆ X, let us take the derivative of equations ( 10) and ( 11) by every variable that occurs in S, where we take multiple derivatives of variables that occur multiple times. This gives

EQUATION

σ k • k! |S|! m j=1 w j h∈S a hj n i=1 a ij x i k-|S| = 0 (13) for |S| ≤ k ≤ d -1.

Observe that there are r = n i=1 (r i + 1) choices for S, since each variable x i can be included anywhere from 0 to r i times. Define A to be the r × m matrix with entries A S,j = h∈S a hj . We claim that A has full row rank. This would show that the number of columns m is at least the number of rows r = n i=1 (r i + 1), proving the desired lower bound on m. Suppose towards contradiction that the rows A S ,• admit a linear dependence: 

0 = |(|S |=s) c • |S |! σ d • d! • ∂ ∂S p(x) + |(|S |<s) c • |S |! σ d+|S |-s • (d + |S | -s)! • 0 = |(|S |=s) c • |S |! σ d • d! • ∂ ∂S p(x).

Since the distinct monomials ∂ ∂S p(x) are linearly independent, this contradicts our assumption that the c are nonzero. We conclude that A has full row rank, and therefore that m Taylor 1 (p) = m ≥ n i=1 (r i + 1). This completes the proof of part (i). Part (ii) of the theorem was demonstrated in the proof of Theorem 4.1. This completes the proof. 7.3 PROOF OF THEOREM 4.3.

Our proof in Theorem 4.1 relied upon the fact that all nonzero partial derivatives of a monomial are linearly independent. This fact is not true for general polynomials p; however, an exactly similar argument shows that m uniform 1 (p) is at least the number of linearly independent partial derivatives of p, taken with respect to multisets of the input variables.

Consider the monomial q of p such that m uniform 1 (q) is maximized, and suppose that q(x) = x r1 1 x r2 2 • • • x rn n . By Theorem 4.1, m uniform 1 (q) is equal to the number n i=1 (r i + 1) of distinct monomials that can be obtained by taking partial derivatives of q. Let Q be the set of such monomials, and let D be the set of (iterated) partial derivatives corresponding to them, so that for d ∈ D, we have d(q) ∈ Q.

Consider the set of polynomials P = {d(p) | d ∈ D}. We claim that there exists a linearly independent subset of P with size at least |D|/c. Suppose to the contrary that P is a maximal linearly independent subset of P with |P | < |D|/c. Since p has c monomials, every element of P has at most c monomials. Therefore, the total number of distinct monomials in elements of P is less than |D|. However, there are at least |D| distinct monomials contained in elements of P , since for d ∈ D, the polynomial d(p) contains the monomial d(q), and by definition all d(q) are distinct as d varies. We conclude that there is some polynomial p ∈ P \P containing a monomial that does not appear in any element of P . But then p is linearly independent of P , a contradiction since we assumed that P was maximal.

We conclude that some linearly independent subset of P has size at least |D|/c, and therefore that the space of partial derivatives of p has rank at least |D|/c = m uniform 1 (q)/c. This proves part (i) of the theorem. Part (ii) follows immediately from the definition of m uniform (p).

Similar logic holds for m Taylor . 7.4 PROOF OF THEOREM 4.4.

We will prove the desired lower bounds for m uniform 1 (p); a very similar argument holds for m Taylor 1 (p). As above, suppose that r i > 0 for i = 1, . . . , n. Let X be the multiset in which x i occurs with multiplicity r i . Suppose that N (x) is an -approximation to p(x) with depth 1, and let the degree-d Taylor polynomial of N (x) be p(x) + E(x). Let E d (x) be the degree-d homogeneous component of E(x). Observe that the coefficients of the error polynomial E d (x) can be made arbitrarily small by setting sufficiently small.

Let m = m uniform 1 (p) and suppose that σ(x) has the Taylor expansion ∞ k=0 σ k x k . Then, by grouping terms of each order, we conclude that there exist constants a ij and w j such that For each S ⊆ X, let us take the derivative of this equation by every variable that occurs in S, where we take multiple derivatives of variables that occur multiple times. This gives Consider this equation as S ⊆ X varies over all C s multisets of fixed size s. The left-hand side represents a linear combination of the m terms ( n i=1 a ij x i ) d-s . The polynomials ∂ ∂S p(x)+ ∂ ∂S E d (x) on the right-hand side must be linearly independent as S varies, since the distinct monomials ∂ ∂S p(x) are linearly independent and the coefficients of ∂ ∂S E d (x) can be made arbitrarily small. This means that the number m of linearly combined terms on the left-hand side must be at least the number C s of choices for S. Observe that C s is the coefficient of the term y s in the polynomial g(y) = i (1 + y + . . . + y ri ). A simple (and not very good) lower bound for C s is 1 d n i=1 (r i + 1), since there are n i=1 (r i + 1) distinct sub-multisets of X, and their cardinalities range from 0 to d.

:



Figures and Tables:

Theorem 3.4 allows us to make the following definition: Definition 3.5. Suppose that a nonlinear function σ is given. For p a multivariate polynomial, let m uniform k (p) be the minimum number of neurons in a depth-k network that -approximates p for all arbitrarily small. Set m uniform (p) = min k m uniform k (p). Likewise, let m Taylor k (p) be the minimum number of neurons in a depth-k network that Taylor-approximates p, and set m Taylor (p) = min k m Taylor k (p).

Theorem 4.4. Let p(x) denote the monomial x r1 1 x r2 2 • • • x rn n , with d = n i=1 r i . Suppose that the nonlinearity σ has nonzero dth Taylor coefficient (other Taylor coefficients are allowed to be zero). Then, m uniform 1 (p) and m Taylor 1 (p) are at least 1 d n i=1

since p is homogeneous. Setting b i = n 1/k , for each i, gives us the desired bound (1).In fact, we can solve for the choice of b i such that the upper bound in (2) is minimized, under the condition b1 b 2 • • • b k = n.Using the technique of Lagrange multipliers, we know that the optimum occurs at a minimum of the function

and rearranging gives us the recursion b

We can use (7) to simplify the first term and (8) (with k = d + |S | -|S * |) to simplify the second term, giving us:

r =1 c A S ,• = 0, where the coefficients c are nonzero and the S denote distinct subsets of X. Set s = max |S |. Then, take the dot product of each side of the above equation by the vector with entries (indexed by j) equal to w j ( n i=1 a ij x i ) We can use (12) to simplify the first term and (13) (with k = d + |S | -s) to simplify the second term, giving us:

x) + E d (x)

∂ ∂S * p(x), which appears in the first summand with coefficient c * • |S * |! σ d •d! . Since the S are distinct, this monomial does not appear in any other term ∂ ∂S p(x), but it could appear in some of the terms ∂ ∂S E k (x). By definition, |c * | is the largest of the values |c |, and by setting small enough, all coefficients of ∂ ∂S E k (x) can be made negligibly small for every k. This implies that the coefficient of the monomial ∂ ∂S * p(x) can be made arbitrarily close to c * • |S * |! σ d •d! , which is nonzero since c * is nonzero. However, the left-hand side of equation (

Table content: <table/>

Peer review (iclr2018):
 Summary and significance: The authors prove that for expressing simple multivariate monomials over n variables, networks of depth 1 require exp(n) many neurons, whereas networks of depth n can represent these monomials using only O(n) neurons. The paper provides a simple and clear explanation for the important problem of theoretically explaining the power of deep networks, and quantifying the improvement provided by depth. +ves: Explaining the power of depth in NNs is fundamental to an understanding of deep learning. The paper is very easy to follow. and the proofs are clearly written. The theorems provide exponential gaps for very simple polynomial functions. -ves: 1. My main concern with the paper is the novelty of the contribution to the techniques. The results in the paper are more general than that of Lin et al., but the proofs are basically the same, and it*s difficult to see the contribution of this paper in terms of the contributing fundamentally new ideas. 2. The second concern is that the results apply only to non-linear activation functions with sufficiently many non-zero derivatives (same requirements as for the results of Lin et al.). 3. Finally, in prop 3.3, reducing from uniform approximations to Taylor approximations, the inequality |E(δx)| <= δ^(d+1) |N(x) - p(x)| does not follow from the definition of a Taylor approximation. Despite these criticisms, I contend that the significance of the problem, and the clean and understandable results in the paper make it a decent paper for ICLR.