Approximation and non-parametric estimation of ResNet-type convolutional neural networks via block-sparse fully-connected neural networks

1 INTRODUCTION . Convolutional Neural Network ( CNN ) is one of the most popular architectures in deep learning research , with various applications such as computer vision ( Krizhevsky et al . ( 2012 ) ) , natural language processing ( Wu et al . ( 2016 ) ) , and sequence analysis in bioinformatics ( Alipanahi et al . ( 2015 ) , Zhou & Troyanskaya ( 2015 ) ) . Despite practical popularity , theoretical justification for the power of CNNs is still scarce from the viewpoint of statistical learning theory . For fully-connected neural networks ( FNNs ) , there is a lot of existing work , dating back to the 80 ’ s , for theoretical explanation regarding their approximation ability ( Cybenko ( 1989 ) , Barron ( 1993 ) , Lu et al . ( 2017 ) , Yarotsky ( 2017 ) , and Petersen & Voigtlaender ( 2017 ) ) and generalization power ( Barron ( 1994 ) , Arora et al . ( 2018 ) , and Suzuki ( 2018 ) ) . See also Pinkus ( 2005 ) and Kainen et al . ( 2013 ) for surveys of earlier works . Although less common compared to FNNs , recently , statistical learning theory for CNNs has been studied , both about approximation ability ( Zhou ( 2018 ) , Yarotsky ( 2018 ) , Petersen & Voigtlaender ( 2018 ) ) and about generalization power ( Zhou & Feng ( 2018 ) ) . One of the standard approaches is to relate the approximation ability of CNNs with that of FNNs , either deep or shallow . For example , Zhou ( 2018 ) proved that CNNs are a universal approximator of the Barron class ( Barron ( 1993 ) , Klusowski & Barron ( 2016 ) ) , which is a historically important function class in the approximation theory . Their approach is to approximate the function using a 2-layered FNN ( i.e. , an FNN with a single hidden layer ) with the ReLU activation function ( Krizhevsky et al . ( 2012 ) ) and transform the FNN into a CNN . Very recently independent of ours , Petersen & Voigtlaender ( 2018 ) showed any function realizable with an FNN can extend to an equivariant function realizable by a CNN that has the same order of parameters . However , to the best of our knowledge , no CNNs that achieves the minimax optimal rate ( Tsybakov ( 2008 ) , Giné & Nickl ( 2015 ) ) in important function classes , including the Hölder class , can keep the number of units in each layer constant with respect to the sample size . Architectures that have extremely large depth , while moderate channel size and width have become feasible , thanks to recent methods such as identity mappings ( He et al . ( 2016 ) , Huang et al . ( 2018 ) ) , sophisticated initialization schemes ( He et al . ( 2015 ) , Chen et al . ( 2018 ) ) , and normalization techniques ( Ioffe & Szegedy ( 2015 ) , Miyato et al . ( 2018 ) ) . Therefore , we would argue that there are growing demands for theories which can accommodate such constant-size architectures . In this paper , we analyze the learning ability of ResNet-type ReLU CNNs which have identity mappings and constant-width residual blocks with fixed-size filters . There are mainly two reasons that motivate us to study this type of CNNs . First , although ResNet is the de facto architecture in various practical applications , the approximation theory for ResNet has not been explored extensively , especially from the viewpoint of the relationship between FNNs and CNNs . Second , constant-width CNNs are critical building blocks not only in ResNet but also in various modern CNNs such as Inception ( Szegedy et al . ( 2015 ) ) , DenseNet ( Huang et al . ( 2017 ) ) , and U-Net ( Ronneberger et al . ( 2015 ) ) , to name a few . Our strategy is to replicate the learning ability of FNNs by constructing tailored ResNet-type CNNs . To do so , we pay attention to the block-sparse structure of an FNN , which roughly means that it consists of a linear combination of multiple ( possibly dense ) FNNs ( we define it rigorously in the subsequent sections ) . Block-sparseness decreases the model complexity coming from the combinatorial sparsity patterns and promotes better bounds . Therefore , it is often utilized , both implicitly or explicitly , in the approximation and learning theory of FNNs ( e.g. , Bölcskei et al . ( 2017 ) , Yarotsky ( 2018 ) ) . We first prove that if an FNN is block-sparse with M blocks ( M -way block-sparse FNN ) , we can realize the FNN with a ResNet-type CNN withO ( M ) additional parameters , which are often negligible since the original FNN already has Ω ( M ) parameters . Using this approximation , we give the upper bound of the estimation error of CNNs in terms of the approximation errors of block sparse FNNs and the model complexity of CNNs . Our result is general in the sense that it is not restricted to a specific function class , as long as we can approximate it using block-sparse FNNs . To demonstrate the wide applicability of our methods , we derive the approximation and estimation errors for two types of function classes with the same strategy : the Barron class ( of parameter s = 2 ) and Hölder class . We prove , as corollaries , that our CNNs can achieve the approximation error of order Õ ( M− D+2 2D ) for the Barron class and Õ ( M− β D ) for the β-Hölder class and the estimation error of order Õp ( N − D+2 2 ( D+1 ) ) for the Barron class and Õp ( N− 2β 2β+D ) for the β-Hölder class , where M is the number of parameters ( we used M here , same as the number of blocks because it will turn out that CNNs have O ( M ) blocks for these cases ) , N is the sample size , and D is the input dimension . These rates are same as the ones for FNNs ever known in the existing literature . An important consequence of our theory is that the ResNet-type CNN can achieve the minimax optimal estimation error ( up to logarithmic factors ) for β-Hölder class even if its filter size , channel size and width are constant with respect to the sample size , as opposed to existing works such as Yarotsky ( 2017 ) and Petersen & Voigtlaender ( 2018 ) , where optimal FNNs or CNNs could have a width or a channel size goes to infinity as N →∞ . In summary , the contributions of our work are as follows : • We develop the approximation theory for CNNs via ResNet-type architectures with constant-width residual blocks . We prove any M -way block-sparse FNN is realizable such a CNN with O ( M ) additional parameters . That means if FNNs can approximate a function with O ( M ) parameters , we can approximate the function with CNNs at the same rate ( Theorem 1 ) . • We derive the upper bound of the estimation error in terms of the approximation error of FNNs and the model complexity of CNNs ( Theorem 2 ) . This result gives the sufficient conditions to derive the same estimation error as that of FNNs ( Corollary 1 ) . • We apply our general theory to the Barron class and Hölder class and derive the approximation ( Corollary 2 and 4 ) and estimation ( Corollary 3 and 5 ) error rates , which are identical to those for FNNs , even if the CNNs have constant channel and filter size with respect to the sample size . In particular , this is minimax optimal for the Hölder case . 2 RELATED WORK . We summarize in Table 1 the differences in the CNN architectures between our work and Zhou ( 2018 ) and Petersen & Voigtlaender ( 2018 ) , which established the approximation theory of CNNs via FNNs . First and foremost , Zhou ( 2018 ) only considered a specific function class — the Barron class — as a target function class , although their method is applicable to any function class that can be realized by a 2-layered ReLU FNN . Regarding the architecture , they considered CNNs with a single channel and whose width is “ linearly increasing ” ( Zhou ( 2018 ) ) layer by layer . For regression or classification problems , it is rare to use such an architecture . In addition , since they did not bound the norm of parameters in the approximating CNNs , we can not derive the estimation error from this method . Petersen & Voigtlaender ( 2018 ) fully utilized the group invariance structure of underlying input spaces to construct CNNs . Such a structure makes theoretical analysis easier , especially for investigating the equivariance properties of CNNs since it enables us to incorporate mathematical tools such as group theory , Fourier analysis , and representation theory . Although their results are quite strong in that it is applicable to any function that can be approximated by FNNs , their assumption on the group structure excludes the padding convolution layer , an important and popular type of convolution operations . Another point is that if we simply apply their construction method to derive the estimation error for ( equivariant ) Hölder functions , combined with the approximation result of Yarotsky ( 2017 ) , the resulting CNN that achieves the minimax optimal rate has Õ ( ε− D β ) channels where ε is the approximation error threshold . It is partly because their construction is not aware of the internal sparse structure of approximating FNNs . Finally , the filter size of their CNN is as large as the input dimension . As opposed to these two works , we employ padding- and ResNettype CNNs which have multiple channels , fixed-size filters , and constant widths . Like Petersen & Voigtlaender ( 2018 ) , our result is applicable to any function , as long as the FNNs to be approximated are block sparse , including the Barron and Hölder cases . If we apply our theorem to these classes , we can show that the optimal CNNs can achieve the same approximation and estimation rate as FNNs , while the number of channels is independent of the sample size . Further , this is minimax optimal up to the logarithmic factors for the Hölder class . Due to its practical success , theoretical analysis for ResNet has been explored recently ( e.g. , Lin & Jegelka ( 2018 ) , Lu et al . ( 2018 ) , Nitanda & Suzuki ( 2018 ) , and Huang et al . ( 2018 ) ) . From the viewpoint of statistical learning theory , Nitanda & Suzuki ( 2018 ) and Huang et al . ( 2018 ) investigated the generalization power of ResNet from the perspective of the boosting interpretation . However , they did not discuss the function approximation ability of ResNet . To the best of our knowledge , our theory is the first work to provide the approximation ability of the CNN class that can accommodate the ResNet-type ones . We import the approximation theories for FNNs , especially ones for the Barron class and Hölder class . The approximation theory for the Barron class has been investigated in e.g. , Barron ( 1993 ) , Klusowski & Barron ( 2016 ) , and Lee et al . ( 2017 ) . Originally Barron ( 1993 ) considered the parameter s = 1 ( see Definition 3 ) and the activation function σ satisfying σ ( z ) → 1 as z →∞ and σ ( z ) → 0 as z → −∞ . Later , Klusowski & Barron ( 2016 ) studied the approximation theory with s = 2 and proved that 2-layered ReLU FNNs with M hidden units can approximate functions of this class with the order of Õ ( M− D+2 2D ) . Yarotsky ( 2017 ) proved FNNs with O ( S ) non-zero parameters can approximate β-Hölder continuous functions with the order of Õ ( S− β D ) . Using this bound , Schmidt-Hieber ( 2017 ) proved that the estimation error of the ERM estimator is Õ ( N− 2β 2β+D ) , which is minimax optimal up to logarithmic factors ( see , e.g. , Tsybakov ( 2008 ) ) .

Summary: This manuscript shows the statistical error of the ERM for nonparametric regression using the family of a Resnet-type of CNNs. Specifically, two results are showed. First, the authors show that any block-sparse fully connected neural network can be embedded in CNNs. Second, they show the covering number of the family of CNNs. Combining with the existing results of the approximation error of neural nets (Klusowski&Barron 2016, Yarotsky 2017, Schmidt-Hieber 2017), they show the L2 statistical risk.