Shallow and Deep Networks are Near-Optimal Approximators of Korobov Functions

1 INTRODUCTION . Neural networks have known tremendous success in the past years in many applications such as computer vision and pattern detection ( Krizhevsky et al. , 2017 ; Silver et al. , 2016 ) . A natural question that arises is how to explain their practical success theoretically . Neural networks are shown to be universal ( Hornik et al. , 1989 ) : any Borel-measurable function can be approximated arbitrarily well by a neural network provided the network has a sufficient number of neurons . Furthermore , universality holds for as low as 1-hidden-layer neural network with reasonable activation functions . However , these results do not specify the number of neurons and parameters to train needed for this approximation . If these numbers are unreasonably high , the universality of neural networks would not explain their practical success . We are interested in evaluating the number of neurons and training parameters needed to approximate a given function within with a neural network . An interesting question is how do these numbers scale with and the dimensionality of the problem , that is , the number of variables of the function . Mhaskar ( 1996 ) showed that any function of the Sobolev space of order r and dimension d can be approximated within with a 1-layer neural network with O ( − dr ) neurons and an infinitely differentiable activation function . This bound exhibits the curse of dimensionality : the number of neurons needed for an −approximation scales exponentially in the dimension of the problem d. Thus , Mhaskar ’ s bound raises the question of whether this curse is inherent to neural networks . DeVore et al . ( 1989 ) showed an important result towards answering this question . They proved that any continuous function approximator ( a notion we formalize in Section 5 ) that approximates all Sobolev functions of order r and dimension dwithin , needs at least Θ ( − d r ) parameters . This result bridges the gap with Mhaskar ’ s bound and confirms that neural networks can not escape the curse of dimensionality for the Sobolev space . A main question is therefore for which set of functions can neural networks break this curse of dimensionality . One way to circumvent the curse of dimensionality is to restrict considerably the considered space of functions and focus on functions with specific structures adapted to neural networks . For example , Mhaskar et al . ( 2016 ) showed that compositional functions with regularity r can be approximated within with deep neural networks with O ( d · − 2r ) neurons . Other structural constraints have been considered for compositions of functions ( Kohler & Krzyżak , 2016 ) , piecewise smooth functions ( Petersen & Voigtlaender , 2018 ; Imaizumi & Fukumizu , 2019 ) , or structures on the data space e.g . lying on a manifold ( Mhaskar , 2010 ; Nakada & Imaizumi , 2019 ; Schmidt-Hieber , 2019 ) . Approximation bounds have also been obtained for the function approximation from data under smoothness constraints ( Kohler & Krzyżak , 2005 ; Kohler & Mehnert , 2011 ) and specifically on mixed smooth Besov spaces which are known to circumvent the curse of dimensionality ( Suzuki , 2018 ) . An- other example is the class of Sobolev functions of order d/α and dimension d for which Mhaskar ’ s bound becomes O ( −α ) . Recently , Montanelli et al . ( 2019 ) considered bandlimited functions and showed that they can be approximated within by deep networks with depth O ( ( log 1 ) 2 ) and O ( −2 ( log 1 ) 2 ) neurons . Weinan et al . ( 2019 ) showed that the closure of the space of 2-layer neural networks with a particular regularity ( namely a restriction on the size of the network ’ s weights ) is the Barron space . They further show that Barron functions can be approximated within with 2-layer networks with O ( −2 ) neurons . Similar line of work restrict the function space with spectral conditions that allows functions to be written as limits of shallow networks ( Barron , 1994 ; Klusowski & Barron , 2016 ; 2018 ) . In this work , we are interested in more general and generic spaces of functions . Our space of interest is the space of multivariate functions of bounded second mixed derivatives , namely the Korobov space . This space is included in the Sobolev space but is reasonably large and general . The Korobov space presents two motivations . The first motivation is theoretical : it is a natural candidate for a large and general space included in the Sobolev space where numerical approximation methods can overcome the curse of dimensionality to some extent ( see Section 2.1 ) . The second motivation is practical : Korobov spaces are useful for solving partial differential equations ( Korobov , 1959 ) and have been used in the context of high-dimensional function approximation ( Zenger & Hackbusch , 1991 ; Zenger , 1991 ) . Recently , Montanelli & Du ( 2019 ) showed that deep neural networks with depth O ( log 1 ) and O ( − 12 ( log 1 ) 3 ( d−1 ) 2 +1 ) neurons can approximate Korobov functions within , lessening the curse of dimensionality for deep neural networks asymptotically in . While they used deep structures to prove their result , the question of whether shallow neural networks also break the curse of dimensionality for the Korobov space remains open . In this paper , we study deep and shallow neural network ’ s approximation power for the Korobov space and make the following contributions : • Representation power of shallow neural networks . We prove that any Korobov function can be approximated within with a 2-layer neural network with ReLU activation function with O ( −1 ( log 1 ) 3 ( d−1 ) 2 +1 ) neurons and O ( − 12 ( log 1 ) 3 ( d−1 ) 2 ) training parameters ( Theorem 3.1 ) . We further extend this result to a large class of commonly used activation functions ( Theorem 3.4 ) . Asymptotically in , our bound can be written as O ( −1−δ ) for all δ > 0 , and in that sense breaks the curse of dimensionality for shallow neural networks . • Representation power of deep neural networks . We prove that any function of the Korobov space can be approximated within with a deep neural network of depth dlog2 ( d ) e+1 independent of , with non-linear C2 activation function , O ( − 12 ( log 1 ) 3 ( d−1 ) 2 ) neurons and O ( − 12 ( log 1 ) 3 ( d−1 ) 2 ) training parameters ( Theorem 4.1 ) . This result improves over the result in Montanelli & Du ( 2019 ) that constructed an approximating neural network with larger depthO ( log 1 log d ) –increasing with the precision – and larger number of neurons O ( − 12 ( log 1 ) 3 ( d−1 ) 2 +1 ) . We note however that they used a different activation function , namely ReLU . • Near-optimality of neural networks as function approximators . Under continuous function approximators model introduced by DeVore et al . ( 1989 ) , we prove that any continuous function approximator needs Θ ( − 1 2 ( log 1 ) d−1 2 ) parameters to approximate functions of the Korobov space within ( Theorem 5.2 ) . This lower bound nearly matches our established upper bounds on the number of training parameters needed by deep and shallow neural networks to approximate functions of the Korobov space , proving that deep and shallow neural networks are near-optimal function approximators of the Korobov space . Table 1 summarizes our new bounds and some of the existing bounds on shallow and deep neural network approximation power for the Korobov space , Sobolev space and bandlimited functions . Our proofs are constructive and give explicit structures to construct such neural networks with ReLU and general activation functions . Our constructions rely on sparse grid approximation introduced by Zenger ( 1991 ) , and studied in detail in Bungartz ( 1992 ) ; Bungartz & Griebel ( 2004 ) . Specifically , we use the sparse grid approach to approximate smooth functions with sums of products then construct neural networks which approximate this structure . A key difficulty is to approximate the product function . In particular in the case of shallow neural networks , we propose , to the best of our knowledge , the first architecture approximating the product function with polynomial number of neurons . To derive our lower bound on the number of parameters needed to approximate the Korobov space , we construct a linear subspace of the Korobov space , with large Bernstein width . This subspace is then used to apply a general lower bound on nonlinear approximation derived by DeVore et al . ( 1989 ) . The rest of the paper is structured as follows . In Section 2 , we define the notions of interest , formalize our objective and introduce the sparse grids approach which will be key in our constructions . In Section 3 and 4 , we prove our bounds on the number of neurons and training parameters needed by a neural network to approximate functions of the Korobov space . Section 3 provides bounds for shallow networks and Section 4 for deep networks . Finally , we formalize in Section 5 the notion of optimal continuous function approximators and prove our novel near-optimality result . 2 PRELIMINARIES . We begin by presenting the basic definitions that will be used in the paper . In this work , we consider feed-forward neural networks , using a linear output neuron and a non-linear activation function σ : R→ R for the other neurons , such as the commonly-used rectified unit ( ReLU ) σ ( x ) = max ( x , 0 ) , the sigmoid σ ( x ) = ( 1 + e−x ) −1 or the Heaviside function σ ( x ) = 1 { x≥0 } . Let d ≥ 1 be the dimension of the input . We define a 1-hidden-layer network with N neurons as x 7→ N∑ k=1 ukσ ( w > k x+ bk ) , wherewk ∈ Rd , bk ∈ R for i = 1 , . . . , N are the parameters of the network . A neural network with several hidden layers is obtained by feeding the outputs of a given layer as inputs to the next layer . We are interested in the expressive power of neural networks i.e . the ability to approximate a target function f : Rd → R with as few neurons as possible , on the unit hyper-cube Ω : = [ 0 , 1 ] d. Another relevant metric is the number of parameters that needs to be trained to approximate the function , that is , the number of parameters of the approximating network ( uk , wk and bk ) depending on the function to approximate . We will adopt L∞ norm as a measure of approximation error . Let us now define some notations that are necessary to present the space of functions that we will approximate . For an integer r , we denote Cr the space of one dimensional functions differentiable r times and with continuous derivatives . In our analysis , we consider functions f with bounded mixed derivatives . For a multi-index α ∈ Nd , we define the derivative of order α , Dαf : = ∂ |α|1f ∂x α1 1 ... ∂x αd d , where |α|1 = ∑d i=1 |αi| . Two common function spaces in a compact Ω ⊂ Rd are the Sobolev spacesW r , p ( Ω ) of functions having weak partial derivatives up to order r in Lp ( Ω ) and the Korobov spaces Xr , p ( Ω ) of functions vanishing at the boundary and having weak mixed second derivatives up to order r in Lp ( Ω ) . W r , p ( Ω ) = { f ∈ Lp ( Ω ) : Dαf ∈ Lp ( Ω ) , |α|1 ≤ r } , Xr , p ( Ω ) = { f ∈ Lp ( Ω ) : f |∂Ω = 0 , Dαf ∈ Lp ( Ω ) , |α|∞ ≤ r } . where ∂Ω denotes the boundary of Ω , |α|1 = ∑d i=1 |αi| and |α|∞ = supi=1 , ... , d |αi| are respectively the L1 and infinity norm . Note that Korobov spaces Xr , p ( Ω ) are subsets of Sobolev spaces W r , p ( Ω ) . For p =∞ , the usual norms on these spaces are given by |f |W r , p ( Ω ) : = max |α|1≤r ‖Dαf‖∞ , |f |Xr , p ( Ω ) : = max|α|∞≤r ‖Dαf‖∞ , For simplicity we will write | · |2 , ∞ for | · |X2 , ∞ . We focus our analysis on approximating functions on the Korobov space X2 , ∞ ( Ω ) for which the curse of dimensionality is drastically lessened and we show that neural networks are near-optimal . Intuitively , one of the key differences compared to the Sobolev space is that Korobov functions do not have high frequency oscillations in all directions at a time . Such functions are known to require an exponential number of neurons Telgarsky ( 2016 ) and are one of the main difficulties to approximating the Sobolev space which therefore exhibits the curse of dimensionality DeVore et al . ( 1989 ) . On the contrary , the Korobov space prohibits such behaviour by ensuring that functions can be differentiable on all dimensions together . Further discussions on Korobov functions and concrete examples are provided in Appendix A .

Summary: This paper studies approximation capabilities of neural networks for the purpose of approximating Korobov functions which are multivariate functions of bounded second mixed derivatives.  The paper presents a complete study for approximating such functions with NNs: they study shallow nets and show that 2 layers with ReLUs and total #neurons of O(1/eps log^{1.5d}(1/eps))  where d is the dimension, can \eps-approximate  Korobov functions. Moreover, by allowing larger depths, close to logd, they can get a better depedence on \eps. Finally, they prove that any continuous function approximator requires a #params close to their upper bound in order to approximation Korobov functions.  This gives a complete picture for how Korobov functions behave wrt to function approximation with shallow or deep nets.