Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality

1 INTRODUCTION . Deep learning has shown great success in several applications such as computer vision and natural language processing . As its application range is getting wider , theoretical analysis to reveal the reason why deep learning works so well is also gathering much attention . To understand deep learning theoretically , several studies have been developed from several aspects such as approximation theory and statistical learning theory . A remarkable property of neural network is that it has universal approximation capability even if there is only one hidden layer ( Cybenko , 1989 ; Hornik , 1991 ; Sonoda & Murata , 2017 ) . Thanks to this property , deep and shallow neural networks can approximate any function with any precision ( of course , the meaning of the terminology “ any ” must be rigorously defined like “ any function in L1 ( R ) ” ) . A natural question coming next is its expressive power . It is shown that the expressive power of deep neural network grows exponentially against the number of layers ( Montufar et al. , 2014 ; Bianchini & Scarselli , 2014 ; Cohen et al. , 2016 ; Cohen & Shashua , 2016 ; Poole et al. , 2016 ) where the “ expressive power ” is defined by several ways . The expressive power of neural network can be analyzed more precisely by specifying the target function ’ s property such as smoothness . Barron ( 1993 ; 1994 ) developed an approximation theory for functions having limited “ capacity ” that is measured by integrability of their Fourier transform . An interesting point of the analysis is that the approximation error is not affected by the dimensionality of the input . This observation matches the experimental observations that deep learning is quite effective also in high dimensional situations . Another typical approach is to analyze function spaces with smoothness conditions such as the Hölder space . In particular , deep neural network with the ReLU activation ( Nair & Hinton , 2010 ; Glorot et al. , 2011 ) has been extensively studied recently from the view point of its expressive power and its generalization error . For example , Yarotsky ( 2017 ) derived the approximation error of the deep network with the ReLU activation for functions in the Hölder space . Schmidt-Hieber ( 2018 ) evaluated the estimation error of regularized least squares estimator performed by deep ReLU network based on this approximation error analysis in a nonparametric regression setting . Petersen & Voigtlaender ( 2017 ) generalized the analysis of Yarotsky ( 2017 ) to the class of piece-wise smooth functions . Imaizumi & Fukumizu ( 2018 ) utilized this analysis to derive the estimation error to estimate the piece-wise smooth function and concluded that deep leaning can outperform linear estimators in that setting . Although these error analyses are standard from a nonparametric statistics view point and the derived rates are known to be ( near ) minimax optimal , the analysis is rather limited because they are given mainly based on the Hölder space . However , there are several other function spaces such as the Sobolev space and the space of finite total variations . A comprehensive analysis to deal with such function classes from a unified view point is required . In this paper , we give generalization error bounds of deep ReLU networks for a Besov space and its variant with mixed smoothness , which includes the Hölder space , the Sobolev space , and the function class with total variation as special cases . By doing so , ( i ) we show that deep learning1 achieves the minimax optimal rate on the Besov space and notably it outperforms any linear estimator such as the kernel ridge regression , and ( ii ) we show that deep learning can avoid the curse of dimensionality on the mixed smooth Besov space and achieves the minimax optimal rate . As related work , Mhaskar & Micchelli ( 1992 ) ; Mhaskar ( 1993 ) ; Chui et al . ( 1994 ) ; Mhaskar ( 1996 ) ; Pinkus ( 1999 ) also developed an approximation error analysis which essentially leads to analyses for Besov spaces . However , the ReLU activation is basically excluded and comprehensive analyses for the Besov space have not been given . As a summary , the contribution of this paper is listed as follows : ( i ) To investigate adaptivity of deep learning , we give an explicit form of approximation and estimation error bounds for deep learning with the ReLU activation where the target functions are in the Besov spaces ( Bsp , q ) for s > 0 and 0 < p , q ≤ ∞ with s > d ( 1/p− 1/r ) + where Lr-norm is used for error evaluation . In particular , deep learning outperforms any linear estimator such as kernel ridge regression if the target function has highly spatial inhomogeneity of its smoothness . See Table 1 for the overview . ( ii ) To investigate the effect of dimensionality , we analyze approximation and estimation problems in so-called the mixed smooth Besov space by ReLU neural network . It is shown that deep learning with the ReLU activation can ease the curse of dimensionality and achieve the near minimax optimal rate . The theory is developed on the basis of the sparse grid technique ( Smolyak , 1963 ) . See Table 2 for the overview . 2 SET UP OF FUNCTION SPACES . In this section , we define the function classes for which we develop error bounds . In particular , we define the Besov space and its variant with mixed smoothness . The typical settings in statistical learning theory is to estimate a function with a smoothness condition . There are several ways to characterize “ smoothness. ” Here , we summarize the definitions of representative functional spaces that are appropriate to define the smoothness assumption . Let Ω ⊂ Rd be a domain of the functions . Throughout this paper , we employ Ω = [ 0 , 1 ] d. For a function f : Ω → R , let ‖f‖p : = ‖f‖Lp ( Ω ) : = ( ∫ Ω |f |pdx ) 1/p for 0 < p < ∞ . For p = 1In this paper , we mainly consider a regularized empirical risk minimization given in Eq . ( 7 ) as an estimation procedure , and say “ deep learning ” to indicate it . ∞ , we define ‖f‖∞ : = ‖f‖L∞ ( Ω ) : = supx∈Ω |f ( x ) | . For α ∈ Rd , let |α| = ∑d j=1 |αj | . Let C0 ( Ω ) be the set of continuous functions equipped with L∞-norm : C0 ( Ω ) : = { f : Ω → R | f is continuous and ‖f‖∞ < ∞ } 2 . For α ∈ Zd+ , we denote by Dαf ( x ) = ∂|α|f ∂α1x1 ... ∂ αdxd ( x ) 3 . Definition 1 ( Hölder space ( Cβ ( Ω ) ) ) . Let β > 0 with β 6∈ N be the smoothness parameter . For an m times differentiable function f : Rd → R , let the norm of the Hölder space Cβ ( Ω ) be ‖f‖Cβ : = max|α|≤m ∥∥Dαf‖∞ + max|α|=m supx , y∈Ω |Dαf ( x ) −Dαf ( y ) ||x−y|β−m , where m = bβc ( the largest integer less than β ) . Then , ( β- ) Hölder space Cβ ( Ω ) is defined as Cβ ( Ω ) = { f | ‖f‖Cβ < ∞ } . The parameter β > 0 controls the “ smoothness ” of the function . Along with the Hölder space , the Sobolev space is also important . Definition 2 ( Sobolev space ( W kp ( Ω ) ) ) . Sobolev space ( W kp ( Ω ) ) with a regularity parameter k ∈ N and a parameter 1 ≤ p ≤ ∞ is a set of functions such that the Sobolev norm ‖f‖Wkp : = ( ∑ |α|≤k ‖Dαf‖pp ) 1 p is finite ( where the derivative D is taken in the weak sense ) . There are some ways to define a Sobolev space with fractional order , one of which will be defined by using the notion of interpolation space ( DeVore , 1998 ; Adams & Fournier , 2003 ) , but we don ’ t pursue this direction here . Finally , we introduce Besov space which further generalizes the definition of the Sobolev space . To define the Besov space , we introduce the modulus of smoothness . Definition 3 . For a function f ∈ Lp ( Ω ) for some p ∈ ( 0 , ∞ ] , the r-th modulus of smoothness of f is defined by wr , p ( f , t ) = sup h∈Rd : ‖h‖2≤t ‖∆rh ( f ) ‖p , where ∆rh ( f ) ( x ) = { ∑r j=0 ( r j ) ( −1 ) r−jf ( x+ jh ) ( x ∈ Ω , x+ rh ∈ Ω ) , 0 ( otherwise ) . Based on the modulus of smoothness , the Besov space is defined as in the following definition . Definition 4 ( Besov space ( Bαp , q ( Ω ) ) ) . For 0 < p , q ≤ ∞ , α > 0 , r : = bαc + 1 , let the seminorm | · |Bαp , q be |f |Bαp , q : = { ( ∫∞ 0 ( t−αwr , p ( f , t ) ) q dt t ) 1 q ( q < ∞ ) , supt > 0 t −αwr , p ( f , t ) ( q =∞ ) . The norm of the Besov space Bαp , q ( Ω ) can be defined by ‖f‖Bαp , q : = ‖f‖p+ |f |Bαp , q , and B α p , q ( Ω ) = { f ∈ Lp ( Ω ) | ‖f‖Bαp , q < ∞ } . 2Since Ω = [ 0 , 1 ] d in our setting , the boundedness automatically follows from the continuity . 3 We let N : = { 1 , 2 , 3 , . . . } , Z+ : = { 0 , 1 , 2 , 3 , . . . } , Zd+ : = { ( z1 , . . . , zd ) | zi ∈ Z+ } , R+ : = { x ≥ 0 | x ∈ R } , and R++ : = { x > 0 | x ∈ R } . Note that p , q < 1 is also allowed . In that setting , the Besov space is no longer a Banach space but a quasi-Banach space . The Besov space plays an important role in several fields such as nonparametric statistical inference ( Kerkyacharian & Picard , 1992 ; Donoho et al. , 1996 ; Donoho & Johnstone , 1998 ; Giné & Nickl , 2015 ) and approximation theory ( Temlyakov , 1993a ) . These spaces are closely related to each other as follows ( Triebel , 1983 ) : • For m ∈ N , Bmp,1 ( Ω ) ↪→Wmp ( Ω ) ↪→ Bmp , ∞ ( Ω ) , and Bm2,2 ( Ω ) = Wm2 ( Ω ) 4 . • For 0 < s < ∞ and s 6∈ N , Cs ( Ω ) = Bs∞ , ∞ ( Ω ) . • For 0 < s , p , q , r ≤ ∞ with s > δ : = d ( 1/p− 1/r ) + , it holds that Bsp , q ( Ω ) ↪→ Bs−δr , q ( Ω ) . In particular , under the same condition , from the definition of ‖ · ‖Bsp , q , it holds that Bsp , q ( Ω ) ↪→ Lr ( Ω ) . ( 1 ) • For 0 < s , p , q ≤ ∞ , if s > d/p , then Bsp , q ( Ω ) ↪→ C0 ( Ω ) . ( 2 ) Hence , if the smoothness parameter satisfies s > d/p , then it is continuously embedded in the set of the continuous functions . However , if s < d/p , then the elements in the space are no longer continuous . Moreover , it is known that B11,1 ( [ 0 , 1 ] ) is included in the space of bounded total variation and B11 , ∞ ( [ 0 , 1 ] ) includes it ( Peetre & Dept , 1976 ) . Hence , the Besov space also allows spatially inhomogeneous smoothness with spikes and jumps ; which makes difference between linear estimators and deep learning ( see Sec . 4.1 ) . It is known that the minimax rate to estimate fo is lower bounded by n−2s/ ( 2s+d ) , ( Kerkyacharian & Picard , 1992 ; Donoho et al. , 1996 ; Donoho & Johnstone , 1998 ; Giné & Nickl , 2015 ) . We see that the curse of dimensionality is unavoidable as long as we consider the Besov space . This is an undesirable property because we easily encounter high dimensional data in several machine learning problems . Hence , we need another condition to derive approximation and estimation error bounds that are not heavily affected by the dimensionality . To do so , we introduce the notion of mixed smoothness . The Besov space with mixed smoothness is defined as follows ( Schmeisser , 1987 ; Sickel & Ullrich , 2009 ) . To define the space , we define the coordinate difference operator as ∆r , ih ( f ) ( x ) = ∆ r h ( f ( x1 , . . . , xi−1 , · , xi+1 , . . . , xd ) ) ( xi ) for f : Rd → R , h ∈ R+ , i ∈ [ d ] , and r ≥ 1 . By applying this difference operator to each coordinate in a subset e ⊂ { 1 , . . . , d } of coordinates , the mixed differential operator for a step length h ∈ Rd is defined as ∆r , eh ( f ) = ( ∏ i∈e ∆ r , i hi ) ( f ) , ∆r , ∅h ( f ) = f. Then , the mixed modulus of smoothness is defined as wer , p ( f , t ) : = sup|hi|≤ti , i∈e ‖∆ r , e h ( f ) ‖p for t ∈ Rd+ and 0 < p ≤ ∞ . This quantity measures how the function f is “ rough ” in a coordinatewise manner . In contrast to the Besov space ( Def . 3 ) , the differentiation is taken over all coordinates in e , which imposes a coordinate-wise smoothness . Letting 0 < p , q ≤ ∞ , α ∈ Rd++ and ri : = bαic+ 1 , the semi-norm | · |MBα , ep , q based on the mixed smoothness is defined by |f |MBα , ep , q : = { ∫ Ω [ ( ∏ i∈e t −αi i ) w e r , p ( f , t ) ] q dt∏ i∈e ti } 1/q ( 0 < q < ∞ ) , supt∈Ω ( ∏ i∈e t −αi i ) w e r , p ( f , t ) ( q =∞ ) . By summing up the semi-norm over the choice of e , the ( quasi- ) norm of the mixed smooth Besov space ( abbreviated to m-Besov space ) is defined by ‖f‖MBαp , q : = ‖f‖p + ∑ e⊂ { 1 , ... , d } |f |MBα , ep , q , 4The notation ↪→ means a continuous embedding ; that is , X ↪→ Y for two normed spaces X , Y means X is continuously embedded in Y . and thus MBαp , q ( Ω ) : = { f ∈ Lp ( Ω ) | ‖f‖MBαp , q < ∞ } where 0 < p , q ≤ ∞ and α ∈ R d ++ . In this paper , we assume that α1 = · · · = αd . With a slight abuse of notation , we also use the notation MBαp , q for α > 0 to indicate MB ( α , ... , α ) p , q . It is known that , when p = q , the m-Besov space is characterized as a tensor product space of Bsp , q ( [ 0 , 1 ] ) ( Sickel & Ullrich , 2009 ) . The m-Besov space includes several important models considered in the literature of statistical learning , e.g. , the additive model ( Meier et al. , 2009 ) and the tensor model ( Signoretto et al. , 2010 ) . It is known that an appropriate estimator in these models can avoid curse of dimensionality ( Meier et al. , 2009 ; Raskutti et al. , 2012 ; Kanagawa et al. , 2016 ; Suzuki et al. , 2016 ) . What we will show in this paper supports that this fact is also applied to deep learning from a unifying viewpoint . The difference between the ( normal ) Besov space and the m-Besov space can be informally explained as follows . For regularity condition αi ≤ 2 ( i = 1 , 2 ) , the m-Besov space consists of functions for which the following derivatives are “ bounded ” : ∂f ∂x1 , ∂f ∂x2 , ∂2f ∂x21 , ∂2f ∂x22 , ∂2f ∂x1∂x2 , ∂3f ∂x1∂x22 , ∂3f ∂x21∂x2 , ∂4f ∂x21∂x 2 2 . That is , the “ max ” of the orders of derivatives over coordinates needs to be bounded by 2 . On the other hand , the Besov space only ensures the boundedness of the following derivatives : ∂f ∂x1 , ∂f ∂x2 , ∂2f ∂x21 , ∂2f ∂x22 , ∂2f ∂x1∂x2 , where the “ sum ” of the orders needs to be bounded by 2 . This difference directly affects the rate of convergence of approximation accuracy . Further details about this space and related topics can be found in a comprehensive survey ( Dũng et al. , 2016 ) . Relation to Barron class . Barron ( 1991 ; 1993 ; 1994 ) showed that , if the Fourier transform of a function f satisfies some integrability condition , then we may avoid curse of dimensionality for estimating neural networks with sigmoidal activation functions . The integrability condition is given by ∫ Cd ‖ω‖|f̂ ( ω ) |dω < ∞ , where f̂ is the Fourier transform of a function f . We call the class of functions satisfying this condition Barron class . A similar function class is analyzed by Klusowski & Barron ( 2016 ) too . We can not compare directly the m-Besov space with the Barron class , but they are closely related . Indeed , if p = q = 2 and s = α1 = · · · = αd , then m-Besov space MBs2,2 ( Ω ) is equivalent to the tensor product of Sobolev space ( Sickel & Ullrich , 2011 ) which consists of functions f : Ω → R satisfying ∫ Cd ∏d i=1 ( 1 + |ωi|2 ) s|f̂ ( ω ) |2dω < ∞ . Therefore , our analysis gives a ( similar but ) different characterization of conditions to avoid curse of dimensionality .

Summary: This paper describes approximation and estimation error bounds for functions in Besov spaces using estimators corresponding to deep ReLU networks. The general idea of connecting network parameters such as depth, width, and sparsity to classical function spaces is interesting and could lead to novel insights into how and why these networks work and under what settings. The authors carefully define Besov spaces and related literature, and overall the paper is clearly written.