Deep learning is adaptive to intrinsic dimensionality of model smoothness in anisotropic Besov space

1 Introduction . Based on the recent literature pertaining to machine learning , deep learning has exhibited superior performance in several tasks such as image recognition ( Krizhevsky et al. , 2012 ) , natural language processing ( Devlin et al. , 2018 ) , and image synthesis ( Radford et al. , 2015 ) . In particular , its superiority is remarkable for complicated and high-dimensional data like images . This is mainly due to its high flexibility and superior feature-extraction ability for effectively extracting the intrinsic structure of data . Its theoretical analysis also has been extensively developed considering several aspects such as expressive ability , optimization , and generalization error . Amongst representation ability analysis of deep neural networks such as universal approximation ability ( Cybenko , 1989 ; Hornik , 1991 ; Sonoda & Murata , 2017 ) , approximation theory of deep neural networks on typical function classes such as Hölder , Sobolev , and Besov spaces have been extensively studied . In particular , analyses of deep neural networks with the ReLU activation ( Nair & Hinton , 2010 ; Glorot et al. , 2011 ) have been recently developed . Schmidt-Hieber ( 2020 ) showed that the deep learning with ReLU activations can achieve the minimax optimal estimation accuracy to estimate composite functions in Hölder spaces by using the approximation theory of Yarotsky ( 2017 ) . Suzuki ( 2019 ) generalized this analysis to those on the Besov space and the mixed smooth 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Besov space by utilizing the techniques developed in approximation theories ( Temlyakov , 1993 ; DeVore , 1998 ) . It was shown that deep learning can achieve an adaptive approximation error rate that is faster than that of ( non-adaptive ) linear approximation methods ( DeVore & Popov , 1988 ; DeVore et al. , 1993 ; Dũng , 2011 ) , and it outperforms any linear estimators ( including kernel ridge regression ) in terms of the minimax optimal rate . From these analyses , one can see that the approximation errors and estimation errors are strongly influenced by two factors , i.e. , the smoothness of the target function and the dimensionality of the input ( see Table 1 ) . In particular , they suffer from the curse of dimensionality , which is unavoidable . However , these analyses are about the worst case errors and do not exploit specific intrinsic properties of the true distributions . For example , practically encountered data usually possess low intrinsic dimensionality , i.e. , data are distributed on a low dimensional sub-manifold of the input space ( Tenenbaum et al. , 2000 ; Belkin & Niyogi , 2003 ) . Recently , Nakada & Imaizumi ( 2020 ) ; Schmidt-Hieber ( 2019 ) ; Chen et al . ( 2019 ) ; Chen et al . ( 2019 ) have shown that deep ReLU network has adaptivity to the intrinsic dimensionality of data and can avoid curse of dimensionality if the intrinsic dimensionality is small . However , one drawback is that they assumed exact low dimensionality of the input data . This could be a strong assumption because practically observed data are always noisy , and injecting noise immediately destroys the low-dimensional structure . Therefore , we consider another direction in this paper . In terms of curse of dimensionality , Suzuki ( 2019 ) showed that deep learning can alleviate the curse of dimensionality to estimate functions in a so called mixed smooth Besov space ( m-Besov ) . However , m-Besov space assumes strong smoothness toward all directions uniformly and does not include the ordinary Besov space as a special case . Moreover , the convergence rate includes heavy poly-log term which is not negligible ( see Table 1 ) . In practice , one of the typically expected properties of a true function on high-dimensional data is that it is invariant against perturbations of an input in some specific directions ( Figure 1 ) . For example , in image-recognition tasks , the target function must be invariant against the spatial shift of an input image , which is utilized by data-augmentation techniques ( Simard et al. , 2003 ; Krizhevsky et al. , 2012 ) . In this paper , we investigate the approximation and estimation abilities of deep learning on anisotropic Besov spaces ( Nikol ’ skii , 1975 ; Vybiral , 2006 ; Triebel , 2011 ) ( also called dominated mixed-smooth Besov spaces ) . An anisotropic Besov space is a set of functions that have “ direction-dependent ” smoothness , whereas ordinary function spaces such as Hölder , Sobolev , and Besov spaces assume isotropic smoothness that is uniform in all directions . We consider a composition of functions included in an anisotropic Besov space , including several existing settings as special cases ; it includes analyses of the Hölder space Schmidt-Hieber ( 2020 ) and Besov space Suzuki ( 2019 ) , as well as the low-dimensional sub-manifold setting ( Nakada & Imaizumi , 2020 ; Schmidt-Hieber , 2019 ; Chen et al. , 2019 ; Chen et al. , 2019 ) 1 . By considering such a space , we can show that deep learning can alleviate curse of dimensionality if the smoothness in each direction is highly anisotropic . Interestingly , any linear estimator ( including kernel ridge regression ) has worse dependence on the dimensionality than deep learning . Our contributions can be summarized as follows : • We consider a situation in which the target function is included in a class of anisotropic Besov spaces and show that deep learning can avoid the curse of dimensionality even if the input data 1We would like to remark that the analysis of Nakada & Imaizumi ( 2020 ) does not require smoothness of the embedded manifold that is not covered in this paper . do not lie on a low-dimensional manifold . Moreover , deep learning can achieve the optimal adaptive approximation error rate and minimax optimal estimation error rate . • We compare deep learning with general linear estimators ( including kernel methods ) and show that deep learning has better dependence on the input dimensionality than linear estimators . 2 Problem setting and the model . In this section , we describe the problem setting considered in this work . We consider the following nonparametric regression model : yi = f o ( xi ) + ξi ( i = 1 , . . . , n ) , where xi is generated from a probability distribution PX on [ 0 , 1 ] d , ξi ∼ N ( 0 , σ2 ) , and the data Dn = ( xi , yi ) ni=1 are independently identically distributed . f o is the true function that we want to estimate . We are interested in the mean squared estimation error of an estimator f̂ : EDn [ ‖f̂ − fo‖2L2 ( PX ) ] , where EDn [ · ] indicates the expectation with respect to the training data Dn . We consider a least-squares estimator in the deep neural network model as f̂ ( see Eq . ( 5 ) ) and discuss its optimality . More specifically , we investigate how the “ intrinsic dimensionality ” of data affects the estimation accuracy of deep learning . For this purpose , we consider an anisotropic Besov space as a model of the target function . 2.1 Anisotropic Besov space . In this section , we introduce the anisotropic Besov which was investigated as the model of the true function in this paper . Throughout this paper , we set the domain of the input to Ω = [ 0 , 1 ] d. For a function f : Ω → R , let ‖f‖p : = ‖f‖Lp ( Ω ) : = ( ∫ Ω |f |pdx ) 1/p for 0 < p < ∞ . For p = ∞ , we define ‖f‖∞ : = ‖f‖L∞ ( Ω ) : = supx∈Ω |f ( x ) | . For β ∈ Rd++ , let |β| = ∑d j=1 |βj |2 . For a function f : Rd → R , we define the rth difference of f in the direction h ∈ Rd as ∆rh ( f ) ( x ) : = ∆ r−1 h ( f ) ( x+ h ) −∆ r−1 h ( f ) ( x ) , ∆ 0 h ( f ) ( x ) : = f ( x ) , for x ∈ Ω with x+ rh ∈ Ω , otherwise , let ∆rh ( f ) ( x ) = 0 . Definition 1 . For a function f ∈ Lp ( Ω ) where p ∈ ( 0 , ∞ ] , the r-th modulus of smoothness of f is defined by wr , p ( f , t ) = suph∈Rd : |hi|≤ti ‖∆ r h ( f ) ‖p , for t = ( t1 , . . . , td ) , ti > 0 . With this modulus of smoothness , we define the anisotropic Besov space Bβp , q ( Ω ) for β = ( β1 , . . . , βd ) > ∈ Rd++ as follows . Definition 2 ( Anisotropic Besov space ( Bβp , q ( Ω ) ) ) . For 0 < p , q ≤ ∞ , β = ( β1 , . . . , βd ) > ∈ Rd++ , r : = maxibβic+ 1 , let the seminorm | · |Bαp , q be |f |Bβp , q : = ( ∞∑ k=0 [ 2kwr , p ( f , ( 2 −k/β1 , . . . , 2−k/βd ) ) ] q ) 1/q ( q < ∞ ) , supk≥0 2 kwr , p ( f , ( 2 −k/β1 , . . . , 2−k/βd ) ) ( q =∞ ) . The norm of the anisotropic Besov space Bβp , q ( Ω ) is defined by ‖f‖Bβp , q : = ‖f‖p + |f |Bβp , q , and Bβp , q ( Ω ) = { f ∈ Lp ( Ω ) | ‖f‖Bβp , q < ∞ } . Roughly speaking β represents the smoothness in each direction . If βi is large , then a function in Bβp , q is smooth to the ith coordinate direction , otherwise , it is non-smooth to that direction . p is also an important quantity that controls the spatial inhomogeneity of the smoothness . If β1 = β2 = · · · = βd , then the definition is equivalent to the usual Besov space ( DeVore & Popov , 1988 ; DeVore et al. , 1993 ) . Suzuki ( 2019 ) analyzed curse of dimensionality of deep learning through a so-called mixed smooth Besov ( m-Besov ) space which imposes a stronger condition toward all directions uniformly . 2We let N : = { 1 , 2 , 3 , . . . } , Z+ : = { 0 , 1 , 2 , 3 , . . . } , Zd+ : = { ( z1 , . . . , zd ) | zi ∈ Z+ } , R+ : = { x ≥ 0 | x ∈ R } , and R++ : = { x > 0 | x ∈ R } . We let [ N ] : = { 1 , . . . , N } for N ∈ N. Particularly , it imposes stronger smoothness toward non-coordinate axis directions . Moreover , mBesov space does not include the vanilla Besov space as a special case and thus can not capture the situation that we consider in this paper . Throughout this paper , for given β = ( β1 , . . . , βd ) > ∈ Rd++ , we write β : = mini βi ( smallest smoothness ) and β : = maxi βi ( largest smoothness ) . The approximation error of a function in anisotropic Besov spaces is characterized by the harmonic mean of ( βj ) dj=1 , which corresponds to the average smoothness , and thus we define β̃ : = ( ∑d j=1 1/βj ) −1 . ( 1 ) The Besov space is closely related to other function spaces such as Hölder space . Let ∂αf ( x ) = ∂|α|f ∂α1x1 ... ∂ αdxd ( x ) . Definition 3 ( Hölder space ( Cβ ( Ω ) ) ) . For a smoothness paraemter β ∈ R++ with β 6∈ N , consider an m times differentiable function f : Rd → R where m = bβc ( the largest integer less than β ) , and let the norm of the Hölder space Cβ ( Ω ) be ‖f‖Cβ : = max|α|≤m ∥∥∂αf‖∞ + max|α|=m supx , y∈Ω |∂αf ( x ) −∂αf ( y ) | ‖x−y‖β−m . Then , ( β- ) Hölder space C β ( Ω ) is defined as Cβ ( Ω ) = { f | ‖f‖Cβ < ∞ } . Let C0 ( Ω ) be the set of continuous functions equipped with L∞-norm : C0 ( Ω ) : = { f : Ω → R | f is continuous and ‖f‖∞ < ∞ } . These function spaces are closely related to each other . Proposition 1 ( Triebel ( 2011 ) ) . There exist the following relations between the spaces : 1 . For β = ( β0 , . . . , β0 ) > ∈ Rd with β0 6∈ N , it holds that Cβ0 ( Ω ) = Bβ∞ , ∞ ( Ω ) . 2 . For 0 < p1 , p2 , q ≤ ∞ , p1 ≤ p2 and β ∈ Rd++ with β̃ > ( 1/p1 − 1/p2 ) +3 , it holds that4 Bβp1 , q ( Ω ) ↪→ B γβ p2 , q ( Ω ) for γ = 1− ( 1/p1 − 1/p2 ) +/β̃ . 3 . For 0 < p , q1 , q2 ≤ ∞ , q1 < q2 , and β ∈ Rd++ , it holds that Bβp , q1 ↪→ B β p , q2 . In particular , with properties 1 and 2 , if β̃ > 1/p , it holds that Bβp , q ( Ω ) ↪→ Cγβ ( Ω ) where γ = 1− 1/ ( β̃p ) . 4 . For 0 < p , q ≤ ∞ and β ∈ Rd++ , if β̃ > 1/p , then Bβp , q ( Ω ) ↪→ C0 ( Ω ) . This result is basically proven by Triebel ( 2011 ) . For completeness , we provide its derivation in Appendix D. If the average smoothness β̃ is sufficiently large ( β̃ > 1/p ) , then the functions in Bβp , q are continuous ; however , if it is small ( β̃ < 1/p ) , then they are no longer continuous . Small p indicates spatially inhomogeneous smoothness ; thus , spikes and jumps appear ( see Donoho & Johnstone ( 1998 ) for this perspective , from the viewpoint of wavelet analysis ) .

Summary: In this paper, the authors investigate the approximation and estimation error of deep neural networks on anisotropic Besov spaces. It has been shown that Holder and Sobolev function spaces are plagued by an unavoidable curse of dimensionality, and a line of research has focused on showing how deep neural network can adaptively beat this curse on restricted classes of functions, such as compositorial functions or functions that depend on a low dimension subspace. This work focuses on Besov spaces, but with anisotropic smoothness requirements (which do not require for the data to lie exactly on a low dimensional subspace) and consider deep composition model similarly to Schmidt-Hieber (2018), but with functions in the anisotropic Besov spaces (they don’t need to be low-dimensional). They show that the rate of approximation/estimation depends on the harmonic mean of the directional smoothness parameters, instead of the worst case. In particular, this harmonic mean is independent of the dimension if the function is non smooth in only a few directions. Their results require bounds on the width, depth, sparsity, and infinity norm of the weights. They further show that the estimation error rates are minmax optimal. Finally they consider linear estimators and show a lower bound on their minimax rate, that is larger than the minimax rate of neural networks, which illustrates the poor adaptivity of linear estimators to the intrinsic dimensionality of the data.