Shallow and Deep Networks are Near-Optimal Approximators of Korobov Functions

1 INTRODUCTION . Neural networks have known tremendous success in the past years in many applications such as computer vision and pattern detection ( Krizhevsky et al. , 2017 ; Silver et al. , 2016 ) . A natural question that arises is how to explain their practical success theoretically . Neural networks are shown to be universal ( Hornik et al. , 1989 ) : any Borel-measurable function can be approximated arbitrarily well by a neural network provided the network has a sufficient number of neurons . Furthermore , universality holds for as low as 1-hidden-layer neural network with reasonable activation functions . However , these results do not specify the number of neurons and parameters to train needed for this approximation . If these numbers are unreasonably high , the universality of neural networks would not explain their practical success . We are interested in evaluating the number of neurons and training parameters needed to approximate a given function within with a neural network . An interesting question is how do these numbers scale with and the dimensionality of the problem , that is , the number of variables of the function . Mhaskar ( 1996 ) showed that any function of the Sobolev space of order r and dimension d can be approximated within with a 1-layer neural network with O ( − dr ) neurons and an infinitely differentiable activation function . This bound exhibits the curse of dimensionality : the number of neurons needed for an −approximation scales exponentially in the dimension of the problem d. Thus , Mhaskar ’ s bound raises the question of whether this curse is inherent to neural networks . DeVore et al . ( 1989 ) showed an important result towards answering this question . They proved that any continuous function approximator ( a notion we formalize in Section 5 ) that approximates all Sobolev functions of order r and dimension dwithin , needs at least Θ ( − d r ) parameters . This result bridges the gap with Mhaskar ’ s bound and confirms that neural networks can not escape the curse of dimensionality for the Sobolev space . A main question is therefore for which set of functions can neural networks break this curse of dimensionality . One way to circumvent the curse of dimensionality is to restrict considerably the considered space of functions and focus on functions with specific structures adapted to neural networks . For example , Mhaskar et al . ( 2016 ) showed that compositional functions with regularity r can be approximated within with deep neural networks with O ( d · − 2r ) neurons . Other structural constraints have been considered for compositions of functions ( Kohler & Krzyżak , 2016 ) , piecewise smooth functions ( Petersen & Voigtlaender , 2018 ; Imaizumi & Fukumizu , 2019 ) , or structures on the data space e.g . lying on a manifold ( Mhaskar , 2010 ; Nakada & Imaizumi , 2019 ; Schmidt-Hieber , 2019 ) . Approximation bounds have also been obtained for the function approximation from data under smoothness constraints ( Kohler & Krzyżak , 2005 ; Kohler & Mehnert , 2011 ) and specifically on mixed smooth Besov spaces which are known to circumvent the curse of dimensionality ( Suzuki , 2018 ) . An- other example is the class of Sobolev functions of order d/α and dimension d for which Mhaskar ’ s bound becomes O ( −α ) . Recently , Montanelli et al . ( 2019 ) considered bandlimited functions and showed that they can be approximated within by deep networks with depth O ( ( log 1 ) 2 ) and O ( −2 ( log 1 ) 2 ) neurons . Weinan et al . ( 2019 ) showed that the closure of the space of 2-layer neural networks with a particular regularity ( namely a restriction on the size of the network ’ s weights ) is the Barron space . They further show that Barron functions can be approximated within with 2-layer networks with O ( −2 ) neurons . Similar line of work restrict the function space with spectral conditions that allows functions to be written as limits of shallow networks ( Barron , 1994 ; Klusowski & Barron , 2016 ; 2018 ) . In this work , we are interested in more general and generic spaces of functions . Our space of interest is the space of multivariate functions of bounded second mixed derivatives , namely the Korobov space . This space is included in the Sobolev space but is reasonably large and general . The Korobov space presents two motivations . The first motivation is theoretical : it is a natural candidate for a large and general space included in the Sobolev space where numerical approximation methods can overcome the curse of dimensionality to some extent ( see Section 2.1 ) . The second motivation is practical : Korobov spaces are useful for solving partial differential equations ( Korobov , 1959 ) and have been used in the context of high-dimensional function approximation ( Zenger & Hackbusch , 1991 ; Zenger , 1991 ) . Recently , Montanelli & Du ( 2019 ) showed that deep neural networks with depth O ( log 1 ) and O ( − 12 ( log 1 ) 3 ( d−1 ) 2 +1 ) neurons can approximate Korobov functions within , lessening the curse of dimensionality for deep neural networks asymptotically in . While they used deep structures to prove their result , the question of whether shallow neural networks also break the curse of dimensionality for the Korobov space remains open . In this paper , we study deep and shallow neural network ’ s approximation power for the Korobov space and make the following contributions : • Representation power of shallow neural networks . We prove that any Korobov function can be approximated within with a 2-layer neural network with ReLU activation function with O ( −1 ( log 1 ) 3 ( d−1 ) 2 +1 ) neurons and O ( − 12 ( log 1 ) 3 ( d−1 ) 2 ) training parameters ( Theorem 3.1 ) . We further extend this result to a large class of commonly used activation functions ( Theorem 3.4 ) . Asymptotically in , our bound can be written as O ( −1−δ ) for all δ > 0 , and in that sense breaks the curse of dimensionality for shallow neural networks . • Representation power of deep neural networks . We prove that any function of the Korobov space can be approximated within with a deep neural network of depth dlog2 ( d ) e+1 independent of , with non-linear C2 activation function , O ( − 12 ( log 1 ) 3 ( d−1 ) 2 ) neurons and O ( − 12 ( log 1 ) 3 ( d−1 ) 2 ) training parameters ( Theorem 4.1 ) . This result improves over the result in Montanelli & Du ( 2019 ) that constructed an approximating neural network with larger depthO ( log 1 log d ) –increasing with the precision – and larger number of neurons O ( − 12 ( log 1 ) 3 ( d−1 ) 2 +1 ) . We note however that they used a different activation function , namely ReLU . • Near-optimality of neural networks as function approximators . Under continuous function approximators model introduced by DeVore et al . ( 1989 ) , we prove that any continuous function approximator needs Θ ( − 1 2 ( log 1 ) d−1 2 ) parameters to approximate functions of the Korobov space within ( Theorem 5.2 ) . This lower bound nearly matches our established upper bounds on the number of training parameters needed by deep and shallow neural networks to approximate functions of the Korobov space , proving that deep and shallow neural networks are near-optimal function approximators of the Korobov space . Table 1 summarizes our new bounds and some of the existing bounds on shallow and deep neural network approximation power for the Korobov space , Sobolev space and bandlimited functions . Our proofs are constructive and give explicit structures to construct such neural networks with ReLU and general activation functions . Our constructions rely on sparse grid approximation introduced by Zenger ( 1991 ) , and studied in detail in Bungartz ( 1992 ) ; Bungartz & Griebel ( 2004 ) . Specifically , we use the sparse grid approach to approximate smooth functions with sums of products then construct neural networks which approximate this structure . A key difficulty is to approximate the product function . In particular in the case of shallow neural networks , we propose , to the best of our knowledge , the first architecture approximating the product function with polynomial number of neurons . To derive our lower bound on the number of parameters needed to approximate the Korobov space , we construct a linear subspace of the Korobov space , with large Bernstein width . This subspace is then used to apply a general lower bound on nonlinear approximation derived by DeVore et al . ( 1989 ) . The rest of the paper is structured as follows . In Section 2 , we define the notions of interest , formalize our objective and introduce the sparse grids approach which will be key in our constructions . In Section 3 and 4 , we prove our bounds on the number of neurons and training parameters needed by a neural network to approximate functions of the Korobov space . Section 3 provides bounds for shallow networks and Section 4 for deep networks . Finally , we formalize in Section 5 the notion of optimal continuous function approximators and prove our novel near-optimality result . 2 PRELIMINARIES . We begin by presenting the basic definitions that will be used in the paper . In this work , we consider feed-forward neural networks , using a linear output neuron and a non-linear activation function σ : R→ R for the other neurons , such as the commonly-used rectified unit ( ReLU ) σ ( x ) = max ( x , 0 ) , the sigmoid σ ( x ) = ( 1 + e−x ) −1 or the Heaviside function σ ( x ) = 1 { x≥0 } . Let d ≥ 1 be the dimension of the input . We define a 1-hidden-layer network with N neurons as x 7→ N∑ k=1 ukσ ( w > k x+ bk ) , wherewk ∈ Rd , bk ∈ R for i = 1 , . . . , N are the parameters of the network . A neural network with several hidden layers is obtained by feeding the outputs of a given layer as inputs to the next layer . We are interested in the expressive power of neural networks i.e . the ability to approximate a target function f : Rd → R with as few neurons as possible , on the unit hyper-cube Ω : = [ 0 , 1 ] d. Another relevant metric is the number of parameters that needs to be trained to approximate the function , that is , the number of parameters of the approximating network ( uk , wk and bk ) depending on the function to approximate . We will adopt L∞ norm as a measure of approximation error . Let us now define some notations that are necessary to present the space of functions that we will approximate . For an integer r , we denote Cr the space of one dimensional functions differentiable r times and with continuous derivatives . In our analysis , we consider functions f with bounded mixed derivatives . For a multi-index α ∈ Nd , we define the derivative of order α , Dαf : = ∂ |α|1f ∂x α1 1 ... ∂x αd d , where |α|1 = ∑d i=1 |αi| . Two common function spaces in a compact Ω ⊂ Rd are the Sobolev spacesW r , p ( Ω ) of functions having weak partial derivatives up to order r in Lp ( Ω ) and the Korobov spaces Xr , p ( Ω ) of functions vanishing at the boundary and having weak mixed second derivatives up to order r in Lp ( Ω ) . W r , p ( Ω ) = { f ∈ Lp ( Ω ) : Dαf ∈ Lp ( Ω ) , |α|1 ≤ r } , Xr , p ( Ω ) = { f ∈ Lp ( Ω ) : f |∂Ω = 0 , Dαf ∈ Lp ( Ω ) , |α|∞ ≤ r } . where ∂Ω denotes the boundary of Ω , |α|1 = ∑d i=1 |αi| and |α|∞ = supi=1 , ... , d |αi| are respectively the L1 and infinity norm . Note that Korobov spaces Xr , p ( Ω ) are subsets of Sobolev spaces W r , p ( Ω ) . For p =∞ , the usual norms on these spaces are given by |f |W r , p ( Ω ) : = max |α|1≤r ‖Dαf‖∞ , |f |Xr , p ( Ω ) : = max|α|∞≤r ‖Dαf‖∞ , For simplicity we will write | · |2 , ∞ for | · |X2 , ∞ . We focus our analysis on approximating functions on the Korobov space X2 , ∞ ( Ω ) for which the curse of dimensionality is drastically lessened and we show that neural networks are near-optimal . Intuitively , one of the key differences compared to the Sobolev space is that Korobov functions do not have high frequency oscillations in all directions at a time . Such functions are known to require an exponential number of neurons Telgarsky ( 2016 ) and are one of the main difficulties to approximating the Sobolev space which therefore exhibits the curse of dimensionality DeVore et al . ( 1989 ) . On the contrary , the Korobov space prohibits such behaviour by ensuring that functions can be differentiable on all dimensions together . Further discussions on Korobov functions and concrete examples are provided in Appendix A .

Summary: This paper studies the ability of shallow and deep neural networks to approximate Korobov functions, and analyses their representation power in terms of the number of used parameters. The authors first show that 2 layers neural networks using common activation functions can approximate any Korobov function within eps error in infinity norm using O(eps^{-1/2} (log 1/eps)^{3(d-1)/2}) parameters. This result improves the existing upper bound of O(eps^{-d/r}) parameters for approximating Sobolev functions in W^{r,p} using 1-hidden layer neural networks, hence reducing the curse of dimensionality.  For deep neural network, this paper provides a new result, showing that neural networks with depth O(log d) and O(eps^{-1/2} (log 1/eps)^{3(d-1)/2}) approximate Korobov functions in X^{2, infinity} within eps error in L-infinity norm. This improves the result of [1], by requiring a depth independent of the required accuracy. However, the current work requires a C^2 and non-linear activation function, and is hence not applicable to ReLU.  Finally, the authors show that any continuous function approximator for the Korobov space requires O(eps^{-1/2} (log 1/eps)^{(d-1)/2}) parameters for achieving error eps, hence matching the previous upper bound for NNs up to factor (log 1/eps)^{d-1}.  [1] Hadrien Montanelli, Haizhao Yang, and Qiang Du. Deep ReLU networks overcome the curse of dimensionality for bandlimited functions