Title: Why Deep Neural Networks for Function Approximation?
Authors: Shiyu Liang
R. Srikant
Venue: ICLR 2017

Abstract: Recently there has been much interest in understanding why deep neural networks are preferred to shallow networks. We show that, for a large class of piecewise smooth functions, the number of neurons needed by a shallow network to approximate a function is exponentially larger than the corresponding number of neurons needed by a deep network for a given degree of function approximation. First, we consider univariate functions on a bounded interval and require a neural network to achieve an approximation error of $\varepsilon$ uniformly over the interval. We show that shallow networks (i.e., networks whose depth does not depend on $\varepsilon$) require $\Omega(\text{poly}(1/\varepsilon))$ neurons while deep networks (i.e., networks whose depth grows with $1/\varepsilon$) require $\mathcal{O}(\text{polylog}(1/\varepsilon))$ neurons. We then extend these results to certain classes of important multivariate functions. Our results are derived for neural networks which use a combination of rectifier linear units (ReLUs) and binary step units, two of the most popular type of activation functions. Our analysis builds on a simple observation: the multiplication of two bits can be represented by a ReLU.



Review discussion:
Section 5 claims that exponentially more units are needed when using more shallow networks, which seems to be referring to Corollary 12, but precisely this statement is given without a proof and only pointing to Theorem 11.
Is the conclusion of the paper referring to some statement other than Corollary 12?
What is the proof of Corollary 12 and how is the lower bound from Theorem 11, which increases with L, resolved?
Thanks for the comments.
Regarding Corollary 12, we have added a short proof now in the appendix, which shows why we can use far fewer neurons in deep networks compared to shallow networks.
We have also added a short discussion at the end of the proof of Theorem 11, which shows that the lower bound is Omega((1/epsilon)^(1/L)) for shallow networks with a fixed depth L, and Omega(log(1/\epsilon)) for deep networks. Note that L is a parameter we can choose. For shallow networks, by definition, L is fixed independent of epsilon, whereas by optimally choosing L to be a function of epsilon (specifically L is Theta(log (1/epsilon)), we get the Omega(log(1/\epsilon)) result.
In particular, one should not interpret the lower bound as increasing in L since L is not optimized in the statement of the theorem. Without optimizing L, the number of neurons required will indeed increase in L since we need at least one neuron in each layer by the definition of an L-layer network.
Again, thanks for the comments.
This results are interesting. How do you connect this with the results in "Norm-Based Capacity Control in Neural Networks" proving that the Rademacher complexity of neural nets with ReLU activations grows exponentially with the depth?
Also, this shows that deep networks with a few hidden units can present an extremely large class of functions which is preferable for approximation but not preferable for learning. Why does the fact that complexity is hidden in the magnitude of weights rather than number of parameters show that deep networks are preferred to shallow networks with many hidden units?
Thanks for your comments. We have not directly tried to relate our results to the VC dimension or Rademacher complexity of neural networks.  We were aware of the work on the VC dimension of neural networks, but we were not aware of the paper that you have cited. Thank you again for bringing this paper to our attention.
It is indeed true that the VC dimension and Rademacher complexity increase with the depth of a neural network. Our results do not advocate the use of neural networks with arbitrarily large depth. We show that using O(polylog(1/epsilon)) layers minimizes the number of neurons required to achieve an approximation error of epsilon. Your question on how this relates to complexity measures is very interesting. The generalization error is upper bounded by the sum of the empirical risk and a term that depends on the complexity measure.
We believe our results are more relevant to the first term, but it is ongoing work to make this more precise. The results in our paper (and earlier papers we have cited) indicate that the piecewise linear approximation that a neural network provides depends on the number of neurons per layer and the number of layers. Increasing the number of layers increases the number of linear pieces that we can work with, although we haven't characterized precisely how this affects the empirical risk.
We are not sure if this response addresses your comment on the magnitude of the weights, we may not be fully understanding this part of your comment. Thanks.
This paper shows:
1. Easy, constructive proofs to derive e-error upper-bounds on neural networks with O(log 1/e) layers and O(log 1/e) ReLU units.
2. Extensions of the previous results to more general function classes, such as smooth or vector-valued functions.
3. Lower bounds on the neural network size, as a function of its number of layers. The lower bound reveals the need of exponentially many more units to approximate functions using shallow architectures.
The paper is well written and easy to follow. The technical content, including the proofs in the Appendix, look correct.
Although the proof techniques are simple (and are sometimes modifications of arguments by Gil, Telgarsky, or Dasgupta), they are brought together in a coherent manner to produce sharp results.
Therefore, I am leaning toward acceptance.
Thanks for your comments. Thanks.
The main contribution of this paper is a construction to eps-approximate a piecewise smooth function with a multilayer neural network that uses O(log(1/eps)) layers and O(poly log(1/eps)) hidden units where the activation functions can be either ReLU or binary step or any combination of them. The paper is well written and clear. The arguments and proofs are easy to follow. I only have two questions:
1- It would be great to have similar results without binary step units. To what extent do you find the binary step unit central to the proof?
2- Is there an example of piecewise smooth function that requires at least poly(1/eps) hidden units with a shallow network?
Thanks for your comments and suggestions.
1- It would be great to have similar results without binary step units. To what extent do you find the binary step unit central to the proof?
(1) The result can be extended to use only ReLUs, but the approximation error will then have to be measured in the L_1 sense rather than the L_\infty sense that we have used in the paper.
2- Is there an example of piecewise smooth function that requires at least poly(1/eps) hidden units with a shallow network?
(2) Indeed our lower bound provides such an example: the function f(x)=x^2 is one such example.
Again, thanks for your comments and suggestions.
SUMMARY
This paper contributes to the description and comparison of the representational power of deep vs shallow neural networks with ReLU and threshold units. The main contribution of the paper is to show that approximating a strongly convex differentiable function is possible with much less units when using a network with one more hidden layer.
PROS
The paper presents an interesting combination of tools and arrives at a nice result on the exponential superiority of depth.
CONS
The main result appears to address only strongly convex univariate functions.
SPECIFIC COMMENTS
- Thanks for the comments on L.
Still it would be a good idea to clarify this point as far as possible in the main part.
Also, I would suggest to advertise the main result more prominently.
I still have not read the revision and maybe you have already addressed some of these points there.
- The problem statement is close to that from [Montufar, Pascanu, Cho, Bengio NIPS 2014], which specifically arrives at exponential gaps between deep and shallow ReLU networks, albeit from a different angle. I would suggest to include that paper it in the overview.
- In Lemma 3, there is an i that should be x
- In Theorem 4, ``\tilde f'' is missing the (x).
- Theorem 11, the lower bound always increases with L ?
- In Theorem 11, \bf x\in [0,1]^d?
Thanks for your comments.  We would also like to point out that the upper bound in our paper holds for a larger class of functions than strongly convex functions.
For example, piecewise linear functions are one class of examples which are not strongly convex, but for which our results hold. The lower bound is indeed for strongly convex functions.
We have now cited Montufar et al (2014) in our paper. Thank you very much for bringing this paper to our attention. We have also addressed your other comments in the revision.

Meta-review:
role: PC
data: comment: The paper makes a solid technical contribution in proving that the deep networks are exponentially more efficient in function approximation compared to the shallow networks. They take the case of piecewise smooth networks, which is practically motivated (e.g. images have edges with smooth regions), and analyze the size of both the deep and shallow networks required to approximate it to the same degree.
 
 The reviewers recommend acceptance of the paper and I am happy to go with their recommendation.

Decision: Accept (Poster)