Learnability of convolutional neural networks for infinite dimensional input via mixed and anisotropic smoothness

1 INTRODUCTION . Deep learning has shown high performance in several tasks such as image recognition , speech recognition , and natural language processing . In particular , convolutional neural networks ( CNNs ) and dilated CNNs have been quite effective in tasks involving high-dimensional data ( van den Oord et al. , 2016 ; He et al. , 2016 ; Simonyan & Zisserman , 2015 ; Yoon , 2014 ) . However , many aspects of its theoretical nature are still unclear while related theoretical studies have attracted much attention . Aside from the analysis of CNNs , one of the most fundamental issues in deep learning theories is its function approximation and estimation capabilities . For example , it is well known that any continuous function with compact support can be approximated with arbitrary accuracy by a twolayer fully connected neural network ( Cybenko , 1989 ; Hornik , 1991 ) . Moreover , the representation ability of deep learning to approximate a function in some function classes such as Hölder classes has also been extensively analyzed ( Mhaskar & Micchelli , 1992 ; Mhaskar , 1993 ; Chui et al. , 1994 ; Mhaskar , 1996 ; Pinkus , 1999 ; Yarotsky , 2017 ; Petersen & Voigtlaender , 2017 ) . In addition to the approximation ability , the estimation ability of deep learning for estimating a function by a finite sample has also been extensively studied . For example , Schmidt-Hieber ( 2020 ) derived the estimation error bound of deep learning with ReLU activation ( Nair & Hinton , 2010 ; Glorot et al. , 2011 ) to estimate functions in the Hölder space and showed the rate of convergence achieves the ( near ) minimax optimal rate . Suzuki ( 2019 ) derived approximation and estimation error rates of deep learning with ReLU activation for the Besov spaces , which were also shown to be ( near ) minimax optimal . Although the derived rates of convergence are near optimal , these studies assumed that the dimensionality of inputs is fixed and much smaller than the sample size . Indeed , the derived rates suffer from the curse of dimensionality . However , in practice , we often encounter settings where the input dimensionality is larger than the sample size or even infinite . For example , in image recognition and natural language processing , the dimensionality of inputs ( images or texts ) is very large , and they could be seen as almost infinite dimensional . To address this issue , some researches considered a setting where the dimensionality of the support of the data distribution is low dimensional . Chen et al . ( 2019b ; a ) considered a setting where data can be embedded in a low dimensional sub-manifold and derived the approximation error of functions that depends merely on the dimensionality of the sub-manifold instead of that of the entire space . Nakada & Imaizumi ( 2020 ) also considered a similar setting , and showed that the estimation error is characterized by the Minkowski dimension of the support of the data distribution . Suzuki ( 2019 ) showed that , even if the data can not be embedded in a low dimensional manifold , anisotropic smoothness of the target function can mitigate the curse of dimensionality . Although these studies revealed that deep learning can avoid curse of dimensionality by utilizing some low dimensional structures of data and the target functions , it still remains unclear how deep learning performs for very high dimensional settings including an infinite dimensional setting . See Table 1 for a summarized comparison to existing studies . In terms of infinite dimensional inputs , there have been already several studies on approximation and estimation errors for non-deep-learning methods . For example , so called hyperbolic cross approximation has been considered to approximate a function in a tensor product space with support on [ 0 , 1 ] ∞ ( Dũng & Griebel , 2016 ) and a polynomial order approximation is possible for functions with mixed smoothness , that is , specific summability properties of the smoothness indices are fulfilled . Ingster & Stepanova ( 2011 ) analyzed a Gaussian white noise model with an infinite dimensional input and showed that the estimation accuracy for signals on infinite dimensional anisotropic Sobolev spaces depends on the reciprocal sum of the smoothness per axis ( see also Ingster & Stepanova ( 2006 ) ; Ingster & Suslina ( 2007 ) ; Ingster & Stepanova ( 2009 ) ) . Oliva et al . ( 2013 ; 2015 ) proposed methods to estimate a map where the input and output are functions or distributions , and derive the rate of convergence . Ferraty et al . ( 2007 ) analyzed the Nadaraya-Watson estimator when the inputs are functions , derived the convergence rate of the estimator , and gave the asymptotic confidence band in the context of functional data analysis ( see Ling & Vieu ( 2018 ) as a comprehensive survey of the nonlinear functional data analysis literature ) . However , these researches are not for the deep learning and the benefit of deep learning for such situation has not been well characterized in the literature . In this study , we analyze the approximation and estimation accuracy in a setting where the input is infinite dimensional , and derive their convergence rates . We assume that the true function has mixed and anisotropic smoothness , that is , the function has different smoothness toward different coordinate similarly to Dũng & Griebel ( 2016 ) ; Ingster & Stepanova ( 2011 ) . The intuition behind this setting is as follows : Considering a function which takes an image as an input , an image can be decomposed into different frequency components and usually a function of images has less sensitivity on the high frequency components and more dependent on the low frequency components , which can be formulated as non-uniform smoothness toward each coordinate direction . By considering such a setting , we can show that the rate of convergence can avoid the curse of dimensionality and be of polynomial order . Our contribution can be summarized as follows : 1 . We consider a learning problem in which the target function to be approximated or estimated can take an infinite dimensional input and has mixed or anisotropic smoothness . Then , we show that deep learning by fully connected neural networks can achieve approximation and estimation errors that depend only on smoothness of the target function and are independent of the dimensionality . 2 . We also consider a setting where the smoothness of the target function has a sparse structure , and then we show that dilated CNNs can find appropriate variables and improve the rate of convergence . This indicates that CNNs can capture a long range dependence among the input . These results show that even when the dimension d of the data is very large compared to the number of observations n , or even when the input is infinite dimensional , it is possible to derive a polynomial order estimation error bound that depends only on the smoothness of the function class . This analysis partially explains the great success of CNNs in various applications with high dimensional inputs . 2 PROBLEM SETTING AND NOTATIONS . In this section , we prepare the notations and introduce the problem setting . Throughout this paper , we use the following notations . Let R > 0 : = { s ∈ R : s > 0 } , and for a set D , let D∞ : = { ( s1 , . . . , si , . . . ) : si ∈ D } ( for example , R∞ : = { ( si ) ∞i=1 : si ∈ R ( ∀i = 1 , 2 , . . . ) } ) . For s ∈ R∞ , let supp ( s ) = { i ∈ N : si 6= 0 } . Let N∞0 : = { l ∈ ( N ∪ { 0 } ) ∞ : supp ( l ) < ∞ } and define Z∞0 and R∞0 in the same way . Furthermore , for s ∈ R∞0 , let 2s : = 2 ∑∞ i=1 si . For L ∈ N , let [ L ] = { 1 , . . . , L } . For a ∈ R , let bac be the largest integer less than or equal to a . 2.1 REGRESSION PROBLEM WITH INFINITE DIMENSIONAL PREDICTOR . In this paper , we consider a regression problem where the predictor ( input ) is infinite dimensional . Let λ be the uniform probability measure on ( [ 0 , 1 ] , B ( [ 0 , 1 ] ) ) where B ( [ 0 , 1 ] ) is the Borel σ-field on [ 0 , 1 ] , and let λ∞ be the product measure of λ on ( [ 0 , 1 ] ∞ , B ( [ 0 , 1 ] ∞ ) ) where B ( [ 0 , 1 ] ∞ ) is the product σ-algebra generated by the cylindric sets ∩j≤d { x ∈ [ 0 , 1 ] ∞ : xj ∈ Bj } for d = 1 , 2 , . . . and Bj ∈ B ( [ 0 , 1 ] ) . Let PX be a probability measure defined on the measurable space ( [ 0 , 1 ] ∞ , B ( [ 0 , 1 ] ∞ ) ) that is absolutely continuous to λ∞ and its Radon-Nikodym derivative satisfies ‖ dPXdλ∞ ‖L∞ ( [ 0,1 ] ∞ ) < ∞ 1 . Then , suppose that there exists a true function fo : [ 0 , 1 ] ∞ → R , and consider the following nonparametric regression problem with an infinite dimensional input : Y = fo ( X ) + ξ , ( 1 ) where X is a random variable taking its value on [ 0 , 1 ] ∞ and obeys the distribution PX introduced above , and ξ is a observation noise generated from N ( 0 , σ2 ) ( a normal distribution with mean 0 and variance σ2 > 0 ) . Let P be the joint distribution of X and Y obeying the regression model . What we investigate in the following is ( i ) how efficiently we can approximate the true function fo by a neural network , and ( ii ) how accurately deep learning can estimate the true function fo from n observations Dn = ( Xi , yi ) ni=1 where ( Xi , yi ) n i=1 are i.i.d . observations from the model . As a performance measure , we employ the mean squared error ‖f − fo‖2PX : = EP [ ( f ( X ) − f o ( X ) ) 2 ] , which can be seen as the excess risk of the predictive error E ( X , Y ) ∼P [ ( f ( X ) −Y ) 2 ] associated with the squared loss ( i.e. , ‖f − fo‖2PX = E ( X , Y ) ∼P [ ( f ( X ) − Y ) 2 ] − E ( X , Y ) ∼P [ ( fo ( X ) − Y ) 2 ] = E ( X , Y ) ∼P [ ( f ( X ) − Y ) 2 ] − inff : measurable E ( X , Y ) ∼P [ ( f ( X ) − Y ) 2 ] ) . 2.2 MIXED AND ANISOTROPIC SMOOTHNESS ON INFINITE DIMENSIONAL VARIABLES . Here , we introduce a function class in which we suppose the true function fo is included . For a given l ∈ Z∞0 , define ψli : [ 0 , 1 ] → R as ψli ( x ) = √ 2 cos ( 2π|li|x ) ( li < 0 ) , √ 2 sin ( 2π|li|x ) ( li > 0 ) , 1 ( li = 0 ) , for x ∈ [ 0 , 1 ] , and define ψl ( X ) : = ∏∞ i=1 ψli ( xi ) for X = ( xi ) ∞ i=1 ∈ [ 0 , 1 ] ∞ . Let L2 ( [ 0 , 1 ] ∞ ) : = { f : [ 0 , 1 ] ∞ → R : ∫ [ 0,1 ] ∞ f2 ( x ) dλ∞ ( x ) < ∞ } equipped with the inner product 〈f , g〉 : = ∫ [ 0,1 ] ∞ f ( x ) g ( x ) dλ∞ ( x ) for f , g ∈ L2 ( [ 0 , 1 ] ∞ ) . Then , ( ψl ) l∈Z∞0 forms a complete orthonormal system of L 2 ( [ 0 , 1 ] ∞ ) , that is , f ∈ L2 ( [ 0 , 1 ] ∞ ) can be expanded as f ( X ) = ∑ l∈Z∞0 〈f , ψl〉ψl ( X ) ( see Ingster & Stepanova ( 2011 ) for example ) . For s ∈ N∞0 , let δs ( f ) : R∞ → R be δs ( f ) ( · ) = ∑ l∈Z∞0 : b2si−1c≤|li| < 2si 〈f , ψl〉ψl ( · ) , 1This is a rather strong assumption . We can omit this if we take θ = 1 and p = ∞ for Fγp , θ . However , we don ’ t pursue this direction in this study . which can be seen as the frequency component of f of frequency |li| ' 2si toward each coordinate . We also define ‖f‖p : = ( ∫ [ 0,1 ] ∞ |f |pdλ∞ ) 1/p for p ≥ 1 . Then , we define a function space with a general smoothness configuration as follows . Definition 1 ( Function class with γ-smoothness ) . For a given γ : N∞0 → R > 0 which is monotonically non-decreasing with respect to each coordinate . For p ≥ 1 , θ ≥ 1 , we define the γ-smooth space as Fγp , θ ( [ 0 , 1 ] ∞ ) : = { f = ∑ l∈Z∞0 〈f , ψl〉ψl : ( ∑ s∈N∞0 2θγ ( s ) ‖δs ( f ) ‖θp ) 1/θ < ∞ } , equipped with the norm ‖f‖Fγp , θ : = ( ∑ s∈N∞0 2θγ ( s ) ‖δs ( f ) ‖θp ) 1/θ . In the following , Fγp , θ ( [ 0 , 1 ] ∞ ) is abbreviated to F γ p , θ , and its unit ball is denoted by U ( F γ p , θ ) . Remind that δs ( f ) represents the frequency component associated with the frequency ( 2si ) ∞i=1 , and then the norm of the γ-smooth space imposes weight 2θγ ( s ) on each frequency component associated with s. In that sense , γ ( s ) controls the weight of each frequency component and accordingly a function in the space can have different smoothness toward different coordinates . As a special case of γ ( s ) , we investigate the following ones in this paper . We can see that a finite dimensional analysis can be easily reduced to a special case of the infinite dimensional analysis ( see Appendix A ) . In that sense , our analysis generalizes existing finite dimensional analyses . Definition 2 ( Mixed smoothness and anisotropic smoothness ) . Given a monotonically nondecreasing sequence a = ( ai ) ∞i=1 ∈ R∞ > 0 , we define the mixed smoothness as ( mixed smoothness ) γ ( s ) = 〈a , s〉 , where 〈a , s〉 : = ∑∞ i=1 aisi 2 , and define the anisotropic smoothness as ( anisotropic smoothness ) γ ( s ) = max { aisi : i ∈ N } . Each component ai of a = ( ai ) ∞i=1 represents the smoothness of the function with respect to the variable xi . Since we assumed ( ai ) ∞i=1 is monotonically non-decreasing , a function in the space has higher smoothness toward the coordinate xi with higher index i . In other words , the function f in the space is less sensitive to the variable xi with a larger index i . For example , in computer vision tasks , we may suppose xi with a large index i corresponds to a higher frequency component of the input image , and then the function is less sensitive to such high frequency components and more sensitive to a low-frequency “ global ” information . This can be seen as an infinite dimensional variant of the mixed smooth Besov space ( Schmeisser , 1987 ; Sickel & Ullrich , 2009 ) and the anisotropic Besov space ( Nikol ’ skii , 1975 ; Vybiral , 2006 ; Triebel , 2011 ) ( see Appendix C for detailed discussions ) . In our theoretical analysis , we will assume that the true target function fo is included in the γ-smooth function space . Assumption 3 . The target function satisfies fo ∈ U ( Fγp , θ ) with p ≥ 1 and θ ≥ 1 , and ‖fo‖∞ ≤ Bf for a fixed constant Bf > 0 , where the smoothness γ is either the mixed smoothness or the anisotropic smoothness .

Summary: The paper studies non-parametric regression for functions defined on infinite-dimensional input data (such as signals in $\ell^2$), using fully-connected networks or dilated convolutional networks (in the CNN case, convolutional layers are followed by a fully-connected network). The authors consider certain smoothness classes similar to mixed or anisotropic smoothness but extended to infinite-dimensional input data, which requires per-coordinate smoothness orders ($a_i$ in Definition 2) that are non-decreasing or increasing with some rate. For such classes, the authors obtain rates that only depend on $a_1$ for the mixed smoothness case, or $\sum_i a_i^{-1}$ for the anisotropic case, for the ERM under certain classes of FNNs or CNNs. Notably, CNNs avoid the need to selecting specific finite subsets of variables as inputs (needed by FNNs), and dilated CNNs additionally allow some adaptivity to sparsity in the $a_i$, in particular, by avoiding dependence on the specific order of the $a_i$ (the growth condition is on the sorted values instead of the unsorted ones for the non-dilated case with a single convolutional layer).