Depth separation and weight-width trade-offs for sigmoidal neural networks

Abstract: Some recent work has shown separation between the expressive power of depth-2 and depth-3 neural networks. These separation results are shown by constructing functions and input distributions, so that the function is well-approximable by a depth-3 neural network of polynomial size but it cannot be well-approximated under the chosen input distribution by any depth-2 neural network of polynomial size. These results are not robust and require carefully chosen functions as well as input distributions.
      
      We show a similar separation between the expressive power of depth-2 and depth-3 sigmoidal neural networks over a large class of input distributions, as long as the weights are polynomially bounded. While doing so, we also show that depth-2 sigmoidal neural networks with small width and small weights can be well-approximated by low-degree multivariate polynomials.

Reviews and author responses:

This paper contributes to the growing literature on depth separations in neural network, showing cases where depth is provably needed to express certain functions. Specifically, the paper shows that there are functions on R^d that can be approximated well by a depth-3 sigmoidal network with poly(d) weights, that cannot be approximated by a depth-2 sigmoidal network with poly(d) weights, and with respect to any input distributions with sufficiently large density in some part of the domain. The proof builds on ideas in Daniely (2017) and Shalev-Shwartz et al. (2011). 

Compared to previous works, the main novelty of the result is that it applies to a very large family of input distributions, as opposed to some specific distributions. On the flip side, it applies only to networks with sigmoids as activation functions, and the weights need to be polynomially bounded. Moreover, although the result is robust to the choice of input distribution, the function used to get the lower bound is still rather artificial ( x -> sin(N||x||^2) for some large N). In a sense, this is complementary to the separation result in Safran and Shamir (2017), mentioned by the authors, where the function is arguably "natural", but the distribution is not. Finally, the proof ideas appear to be not too different than those of Daniely (2017).

Overall, I think this is a decent contribution to this topic, and would recommend accepting it given enough room. It's a bit incremental in light of existing work, but does contribute to the important question of whether we can prove depth separations which are also robust.

This paper proves a new separation results from 3-layer neural networks to 2-layer neural networks. The core of the analysis is a proof that any 2-layer neural networks can be well approximated by a polynomial function with reasonably low degrees. Then the authors constructs a highly non-smooth function can be represented by a 3-layer network, but impossible to approximate by any polynomial-degree polynomial function.

Similar results about polynomial approximation can be found in [1] (Theorem 4). To me, the result proved in [1] is spiritually very similar to propositions 3-4. The authors need to justify the difference.

The main strength of the new separation result is that it holds for a larger class of input distributions. Comparing to Daniely (2017) which requires the input distribution to be spherically uniform, the new result only needs the distribution to be lower bounded by 1/poly(d) in a small ball of radius 1/poly(d). Conceptually I don't think this is a much weaker condition. For a "truly" non-uniform distribution, one should allow its density function to be very close to zero at certain regions of the ball. Nevertheless, the result is a step forward from Daniely (2017) and the paper is well written.

I am still in doubt of the practical value of such kind of separation results. The paper proves the separation by constructing a very specific function that cannot be approximated by 2-layer networks. This function has a super large Lipschitz constant, which we don't expect to see in practice. Consider the function f(x)=cos(Nx). When N is chosen large enough, the function f can not be well approximated by any 2-layer network with polynomial size. Does it imply that the family of cosine functions is rich enough so that it is a better family to learn than 2-layer neural networks? I guess the answer would be negative. In addition, the paper doesn't show that any 2-layer network can be well approximated by a 3-layer network, which is a missing piece in justifying the richness of 3-layer nets.

Finally, the constructed "hard" function has order d^5 Lipschitz constant, but Theorem 7 assumes that the 2-layer networks' weight must be bounded by O(d^2). This assumption is crucial to the proof but not well justified (especially considering the d^5 factor in the function definition).

[1] On the Computational Efficiency of Training Neural Networks, Livni et al., NIPS'14

The paper shows that there are functions that can be represented by depth 3 sigmoidal neural networks (with polynomial weights and polynomially many units), but sigmoidal networks of depth 2 with polynomially bounded weights require exponentially many units. There is nothing new technically in the paper and I find the results uninteresting given the spate of results of this kind. I don't share the authors enthusiasm about much more general distributions etc. The approximations the authors are shooting for are much stronger the kind that has been used by Eldan and Shamir (2016) or other such papers. The approximation used here is $\ell_\infty$ rather than $\ell_2$. So a negative result for depth 2 is weaker; the earlier work (and almost trivially by using the work of Cybenko, Hornik, etc.) already shows that the depth -3 approximations are uniform approximators. 

The fact that sigmoidal neural networks with bounded weights can be expressed as "low" degree polynomials is not new. Much stronger results including bounds on the weights of the polynomial (sum of squares of coefficients) appear implicitly in Zhang et al. (2014) and Goel et al. (2017). In fact, these last two papers go further and show that this has implications for learnability not just for representation as the current paper shows. 

Additionally, I think the paper is a bit sloppy in the maths. For example, Theorem 7 does not specifiy what delta is. I'm sure they mean that there is a "small enough \delta" (with possible dependence on d, B, etc.). But surely this statement is not true for all values of $\delta$. For e.g. when $\delta = 1$, sin(\pi d^5 \Vert x \Vert^2) can rather trivially be expressed as a sigmoidal neural network of depth 2. 

Overall, I think this paper has a collection of results that are well-known to experts in the field and add little novelty. It's unlikely that having yet another paper separating depth 2 from depth 3 with some other set of conditions will move us towards further progress in the very important question of depth separation. 

1. Added a new section (Section 5 on separation under L2).
2. Moved some of the proofs from Section 3 to the appendix.

Thank you for a careful and detailed review. 

The general idea of using low degree polynomials for depth separation is well-used in the previous work. We would like to make one remark about the connection with Daniely (2017). While we have to place the extra restriction of upper bounds on weights and type of nonlinearity, our proof is much simpler than Daniely’s (for example, we don’t use spherical harmonics) and this simplicity lends it flexibility which allows us to prove lower bounds for a general class of distributions. We agree that our proofs are simple modifications of previous techniques but we think that simplicity ought to be valued over complexity if it produces interesting results. Please see our discussion of points raised by AnonReviewer 2 and 3. 


Thank you for a careful and detailed review. 

Our result does allow non-uniform distributions that are close to zero in certain regions. For example, Theorem 7 only requires the density function to be lower bounded by 1/poly(d) in a small ball of radius 1/poly(d); the density can be zero or close to zero anywhere outside this small ball. In response to AnonReviewer 2, we have pointed out that our proof also works for a stronger depth-2-vs-3 separation for L_2 approximation (instead of L_\infty) under a general class of distributions. 

Thank you for providing the Livni et al. (2014) reference. The low-degree polynomial approximation in Section 3 are known and follow easily from the previous work of Shalev-Shwartz et al. (2011), which is clearly cited by us as well as by Livni et al. (2014). Section 3 is only for completeness as previous work we cited did not have the precise statements of lemmas used in our proof of Theorem 7. As you pointed out, our Proposition 4 is essentially Theorem 4 in Livni et al. (2014), so we will correct that. Our Proposition 5 is its straightforward extension to higher depths.

All depth-2-vs-3 separation results that we are aware of use poly(d)-Lipschitz functions. In dimension 1, any function with poly(d)-Lipschitz constant can be well-approximated by a depth-2 networks of size poly(d), ref. Debao (1993). We do not use sine in any crucial way; in fact, any function that is far from low-degree polynomials would do (as in Daniely). Thus, the class of functions for which the separation works is more general than what we get by using just sine. The recent progression of results by Eldan-Shamir (COLT’16), Safran-Shamir (ICML’17), Daniely (COLT’17) points to depth-width trade-offs for approximating “natural” functions under “natural” distributions as an important open problem. Safran-Shamir consider approximations of “natural” functions under carefully chosen distributions. Daniely considers uniform distribution on S^{d-1} x S^{d-1} as an instance of “natural” distribution. The definition of “natural” is debatable, so one would ideally like to prove such results for distributions as general as possible. To the best of our knowledge, our work is the first attempt in this direction.  

We do not understand your point about “richness” of cosines vs. sigmoid neural networks. The fact that cos(Nx) cannot be well-approximated by 2-layer networks does not mean that the family of cosines is “richer”, if richness is taken to mean the ability to approximate a larger set of functions. For example, the class of single cosine functions (as opposed to, say, their linear combinations) cannot approximate step functions in a bounded interval, but 2-layer networks can come arbitrarily close to step functions. 

We only claimed that there are functions which are well-approximable by depth-3 networks but not by depth-2 networks for a wide class of distributions. However, here is a construction to show that any sigmoid depth-2 network N (of size |N|) can be approximated by a sigmoid depth-3 network N’ of size |N|+d (d is the number of input coordinates). We do this by adding a layer of sigmoids between the inputs and N. For convenience, we will describe the construction using the closely related tanh gates instead (tanh(u) := 2\sigma(u) -1). Using sigmoids requires some minor changes in the construction. Each new tanh acts as (approximate) identity. In a little more detail, for each input coordinate x_i, we add a new sigmoid (2/C) \tanh(C x_i) where C is a small constant. It can be seen that (2/C)\tanh(C x_i) \approx x_i for all bounded x_i (by choosing constant C small enough one can make the approximation as close as one wishes for a given range of x_i). The output of this new layer is passed on to N. Since the new layer acts as approximate identity, the new network N’ approximates N. In the above construction, the only property of sigmoids (or tanh) that we used was that it is able to represent the identity function after applying a linear transformation (at least approximately). Thus the construction applies to networks that use different nonlinearities with this property.

The bound of d^2 on weights is not special. In general, given any bound B on the weights of the 2-layer network we can construct an L-Lipschitz function with L = d^3 B such that this function can be well-approximated by a small 3-layer network but any 2-layer network requires a large size for the type of distributions mentioned in the paper. 


Thank you for a careful and detailed review. 

We completely agree with you that a negative result for L_2 approximation is stronger than for L_\infty. Our technique indeed works for L_2 as mentioned in the remark after Theorem 7, which is our main result. We have updated our paper by adding Section 5 containing separation under L2 under a large class of distributions. 

We strongly disagree with you that depth-2-vs-3 separation for L_2 approximation under general distributions is uninteresting or known. If you believe this is already known, please provide a reference. The recent progression of results by Eldan-Shamir (COLT’16), Safran-Shamir (ICML’17), Daniely (COLT’17) points to depth-width trade-offs for approximating “natural” functions under “natural” distributions as an important open problem. Safran-Shamir consider approximations of “natural” functions under carefully chosen distributions. Daniely considers uniform distribution on S^{d-1} x S^{d-1} as an instance of “natural” distribution. The definition of “natural” is debatable, so one would ideally like to prove such results for distributions as general as possible. To the best of our knowledge, our work is the first attempt in this direction.   

We agree with you that our techniques are simple modifications of existing ideas, however, simplicity ought to be valued over complicated proofs if it leads to interesting results. We find it interesting that a simple proof yields depth-2-vs-3 separation of sigmoid networks for L_2 approximation under a general class of distributions. We cite Shalev-Shwartz et al. (2011) and others clearly, since the low-degree polynomial approximations in Section 3 are known and follow easily from their work. We will add similar results from Goel et al. (2017) too. Section 3 exists only for completeness as the previous work does not contain the precise statements of lemmas required in our proof. However, our main result on depth-width trade-off (Theorem 7) is in Section 4 (and in the updated version, also in Section 5). Depth separation and learnability are related but different problems, and the results in Zhang et al. (2014) and Goel et al. (2017) have no bearing on our Section 4 as far as we can see.

Thank you for pointing out other minor errors such as being specific about “small enough \delta” in Theorem 7. We will re-write them with precise bounds for \delta, d, n, B etc. In particular, \delta < 1/3 works. 


Meta-review: The reviewers point out that most of the results are already known and are not novel. There are also issues with the presentation. Studying only depth 2 and depth 3 networks is very limiting.

Decision: rejected-papers