Approximation and non-parametric estimation of ResNet-type convolutional neural networks via block-sparse fully-connected neural networks

1 INTRODUCTION . Convolutional Neural Network ( CNN ) is one of the most popular architectures in deep learning research , with various applications such as computer vision ( Krizhevsky et al . ( 2012 ) ) , natural language processing ( Wu et al . ( 2016 ) ) , and sequence analysis in bioinformatics ( Alipanahi et al . ( 2015 ) , Zhou & Troyanskaya ( 2015 ) ) . Despite practical popularity , theoretical justification for the power of CNNs is still scarce from the viewpoint of statistical learning theory . For fully-connected neural networks ( FNNs ) , there is a lot of existing work , dating back to the 80 ’ s , for theoretical explanation regarding their approximation ability ( Cybenko ( 1989 ) , Barron ( 1993 ) , Lu et al . ( 2017 ) , Yarotsky ( 2017 ) , and Petersen & Voigtlaender ( 2017 ) ) and generalization power ( Barron ( 1994 ) , Arora et al . ( 2018 ) , and Suzuki ( 2018 ) ) . See also Pinkus ( 2005 ) and Kainen et al . ( 2013 ) for surveys of earlier works . Although less common compared to FNNs , recently , statistical learning theory for CNNs has been studied , both about approximation ability ( Zhou ( 2018 ) , Yarotsky ( 2018 ) , Petersen & Voigtlaender ( 2018 ) ) and about generalization power ( Zhou & Feng ( 2018 ) ) . One of the standard approaches is to relate the approximation ability of CNNs with that of FNNs , either deep or shallow . For example , Zhou ( 2018 ) proved that CNNs are a universal approximator of the Barron class ( Barron ( 1993 ) , Klusowski & Barron ( 2016 ) ) , which is a historically important function class in the approximation theory . Their approach is to approximate the function using a 2-layered FNN ( i.e. , an FNN with a single hidden layer ) with the ReLU activation function ( Krizhevsky et al . ( 2012 ) ) and transform the FNN into a CNN . Very recently independent of ours , Petersen & Voigtlaender ( 2018 ) showed any function realizable with an FNN can extend to an equivariant function realizable by a CNN that has the same order of parameters . However , to the best of our knowledge , no CNNs that achieves the minimax optimal rate ( Tsybakov ( 2008 ) , Giné & Nickl ( 2015 ) ) in important function classes , including the Hölder class , can keep the number of units in each layer constant with respect to the sample size . Architectures that have extremely large depth , while moderate channel size and width have become feasible , thanks to recent methods such as identity mappings ( He et al . ( 2016 ) , Huang et al . ( 2018 ) ) , sophisticated initialization schemes ( He et al . ( 2015 ) , Chen et al . ( 2018 ) ) , and normalization techniques ( Ioffe & Szegedy ( 2015 ) , Miyato et al . ( 2018 ) ) . Therefore , we would argue that there are growing demands for theories which can accommodate such constant-size architectures . In this paper , we analyze the learning ability of ResNet-type ReLU CNNs which have identity mappings and constant-width residual blocks with fixed-size filters . There are mainly two reasons that motivate us to study this type of CNNs . First , although ResNet is the de facto architecture in various practical applications , the approximation theory for ResNet has not been explored extensively , especially from the viewpoint of the relationship between FNNs and CNNs . Second , constant-width CNNs are critical building blocks not only in ResNet but also in various modern CNNs such as Inception ( Szegedy et al . ( 2015 ) ) , DenseNet ( Huang et al . ( 2017 ) ) , and U-Net ( Ronneberger et al . ( 2015 ) ) , to name a few . Our strategy is to replicate the learning ability of FNNs by constructing tailored ResNet-type CNNs . To do so , we pay attention to the block-sparse structure of an FNN , which roughly means that it consists of a linear combination of multiple ( possibly dense ) FNNs ( we define it rigorously in the subsequent sections ) . Block-sparseness decreases the model complexity coming from the combinatorial sparsity patterns and promotes better bounds . Therefore , it is often utilized , both implicitly or explicitly , in the approximation and learning theory of FNNs ( e.g. , Bölcskei et al . ( 2017 ) , Yarotsky ( 2018 ) ) . We first prove that if an FNN is block-sparse with M blocks ( M -way block-sparse FNN ) , we can realize the FNN with a ResNet-type CNN withO ( M ) additional parameters , which are often negligible since the original FNN already has Ω ( M ) parameters . Using this approximation , we give the upper bound of the estimation error of CNNs in terms of the approximation errors of block sparse FNNs and the model complexity of CNNs . Our result is general in the sense that it is not restricted to a specific function class , as long as we can approximate it using block-sparse FNNs . To demonstrate the wide applicability of our methods , we derive the approximation and estimation errors for two types of function classes with the same strategy : the Barron class ( of parameter s = 2 ) and Hölder class . We prove , as corollaries , that our CNNs can achieve the approximation error of order Õ ( M− D+2 2D ) for the Barron class and Õ ( M− β D ) for the β-Hölder class and the estimation error of order Õp ( N − D+2 2 ( D+1 ) ) for the Barron class and Õp ( N− 2β 2β+D ) for the β-Hölder class , where M is the number of parameters ( we used M here , same as the number of blocks because it will turn out that CNNs have O ( M ) blocks for these cases ) , N is the sample size , and D is the input dimension . These rates are same as the ones for FNNs ever known in the existing literature . An important consequence of our theory is that the ResNet-type CNN can achieve the minimax optimal estimation error ( up to logarithmic factors ) for β-Hölder class even if its filter size , channel size and width are constant with respect to the sample size , as opposed to existing works such as Yarotsky ( 2017 ) and Petersen & Voigtlaender ( 2018 ) , where optimal FNNs or CNNs could have a width or a channel size goes to infinity as N →∞ . In summary , the contributions of our work are as follows : • We develop the approximation theory for CNNs via ResNet-type architectures with constant-width residual blocks . We prove any M -way block-sparse FNN is realizable such a CNN with O ( M ) additional parameters . That means if FNNs can approximate a function with O ( M ) parameters , we can approximate the function with CNNs at the same rate ( Theorem 1 ) . • We derive the upper bound of the estimation error in terms of the approximation error of FNNs and the model complexity of CNNs ( Theorem 2 ) . This result gives the sufficient conditions to derive the same estimation error as that of FNNs ( Corollary 1 ) . • We apply our general theory to the Barron class and Hölder class and derive the approximation ( Corollary 2 and 4 ) and estimation ( Corollary 3 and 5 ) error rates , which are identical to those for FNNs , even if the CNNs have constant channel and filter size with respect to the sample size . In particular , this is minimax optimal for the Hölder case . 2 RELATED WORK . We summarize in Table 1 the differences in the CNN architectures between our work and Zhou ( 2018 ) and Petersen & Voigtlaender ( 2018 ) , which established the approximation theory of CNNs via FNNs . First and foremost , Zhou ( 2018 ) only considered a specific function class — the Barron class — as a target function class , although their method is applicable to any function class that can be realized by a 2-layered ReLU FNN . Regarding the architecture , they considered CNNs with a single channel and whose width is “ linearly increasing ” ( Zhou ( 2018 ) ) layer by layer . For regression or classification problems , it is rare to use such an architecture . In addition , since they did not bound the norm of parameters in the approximating CNNs , we can not derive the estimation error from this method . Petersen & Voigtlaender ( 2018 ) fully utilized the group invariance structure of underlying input spaces to construct CNNs . Such a structure makes theoretical analysis easier , especially for investigating the equivariance properties of CNNs since it enables us to incorporate mathematical tools such as group theory , Fourier analysis , and representation theory . Although their results are quite strong in that it is applicable to any function that can be approximated by FNNs , their assumption on the group structure excludes the padding convolution layer , an important and popular type of convolution operations . Another point is that if we simply apply their construction method to derive the estimation error for ( equivariant ) Hölder functions , combined with the approximation result of Yarotsky ( 2017 ) , the resulting CNN that achieves the minimax optimal rate has Õ ( ε− D β ) channels where ε is the approximation error threshold . It is partly because their construction is not aware of the internal sparse structure of approximating FNNs . Finally , the filter size of their CNN is as large as the input dimension . As opposed to these two works , we employ padding- and ResNettype CNNs which have multiple channels , fixed-size filters , and constant widths . Like Petersen & Voigtlaender ( 2018 ) , our result is applicable to any function , as long as the FNNs to be approximated are block sparse , including the Barron and Hölder cases . If we apply our theorem to these classes , we can show that the optimal CNNs can achieve the same approximation and estimation rate as FNNs , while the number of channels is independent of the sample size . Further , this is minimax optimal up to the logarithmic factors for the Hölder class . Due to its practical success , theoretical analysis for ResNet has been explored recently ( e.g. , Lin & Jegelka ( 2018 ) , Lu et al . ( 2018 ) , Nitanda & Suzuki ( 2018 ) , and Huang et al . ( 2018 ) ) . From the viewpoint of statistical learning theory , Nitanda & Suzuki ( 2018 ) and Huang et al . ( 2018 ) investigated the generalization power of ResNet from the perspective of the boosting interpretation . However , they did not discuss the function approximation ability of ResNet . To the best of our knowledge , our theory is the first work to provide the approximation ability of the CNN class that can accommodate the ResNet-type ones . We import the approximation theories for FNNs , especially ones for the Barron class and Hölder class . The approximation theory for the Barron class has been investigated in e.g. , Barron ( 1993 ) , Klusowski & Barron ( 2016 ) , and Lee et al . ( 2017 ) . Originally Barron ( 1993 ) considered the parameter s = 1 ( see Definition 3 ) and the activation function σ satisfying σ ( z ) → 1 as z →∞ and σ ( z ) → 0 as z → −∞ . Later , Klusowski & Barron ( 2016 ) studied the approximation theory with s = 2 and proved that 2-layered ReLU FNNs with M hidden units can approximate functions of this class with the order of Õ ( M− D+2 2D ) . Yarotsky ( 2017 ) proved FNNs with O ( S ) non-zero parameters can approximate β-Hölder continuous functions with the order of Õ ( S− β D ) . Using this bound , Schmidt-Hieber ( 2017 ) proved that the estimation error of the ERM estimator is Õ ( N− 2β 2β+D ) , which is minimax optimal up to logarithmic factors ( see , e.g. , Tsybakov ( 2008 ) ) .

Summary: The authors demonstrate the function expression properties for the Residual type convolutional neural networks to approximate the block sparse fully connected neural networks. Then it is shown that such Res-CNNs can approximate any function as long as it can be expressed by the block-sparse FNNs, including the Barron class and Holder class functions. The price to pay is that the number of parameters is larger than that of the FNNs by a constant factor.