Improved Generalization Bound for Deep Neural Networks Using Geometric Functional Analysis

1 INTRODUCTION . The problem of generalization of deep neural networks from the perspective of theoretical analysis has recently received a considerable amount of interest ( Neyshabur et al. , 2015 ; Zhang et al. , 2017 ; Dziugaite & Roy , 2017 ; Neyshabur et al. , 2018 ; Golowich et al. , 2018 ; Nagarajan & Kolter , 2018 ; Daniely & Granot , 2019 ) . More specifically , most of the state-of-the-art bound are based on spectral norm based generalization bounds and have shown to give tighter and sharper bounds compared to conventional ones leveraging PAC Bayesian theory ( Bartlett et al. , 2017 ; Neyshabur et al. , 2015 ; 2018 ) . However , the bounds in such works are quite limited as they only apply in cases where the parameters are drawn from a distribution or when they are represented by fewer bits than required ( Nagarajan & Kolter , 2018 ) . In this paper , we provide a novel framework based on the foundational mathematics of geometric functional analysis to obtain a sharp bound . We show that this analysis has many advantages over the conventional conceptions which solely rely on stochastic assumptions ignoring the geometric structure of the neural networks and to the best of our knowledge this is the first work attempts to foray in this direction . We compare our result with that of Neyshabur et al . to show that our bound is better than the bounds derived using spectral norms . Geometric functional analysis deals with infinite dimensional vector spaces from a geometric perspective ( Holmes , 2012 ) , of its many applications it is widely in the theoretical analysis of wavelet theory ( Young , 2012 ) . In fact we believe this is the most suited framework for dealing with deep neural networks , as their parameters are generally huge and any frame work that is finite will not cover all possible models . Specifically , our approach involves using the covering number of a vector space to derive a bound for the generalization error of a given neural network . However , computing the covering number of a high dimensional vector space becomes an intractable problem as the size of the input space becomes sufficiently large . Thus , we make use of the Vitushkin ’ s inequality ( Friedland & Yomdin , 2014 ) to bound the covering numbers using the Lesbesgue measure , which is a tractable quantity . This bound along with functional inequalities forms the groundwork by which we extend it generalization problem of neural networks which also takes into account the geometry of the data input . Another by product of the analysis is that the bound obtained is in terms of the Frobenius norm of the weight matrices rather than its spectral norm , which has shown to scale more rapidly with the size of the matrix ( Vershynin , 2011 ) . We approach the neural network in a rather unorthodox manner by treating it as a recurrent polynomial function which can be approximated to some arbitrary degree ( Telgarsky , 2017 ) , as it is the most amicable framework for exploiting its geometric properties . We explain this in detail in the following sections along with other mathematical preliminaries . This is followed by several intermediate theorems , which are then used to derive our generalization bound . 2 RELATED WORK . The concept of generalization bounds were introduced in ( Bartlett , 1998 ) rather indirectly by analyzing the probability of misclassification . ( Zhang et al. , 2017 ) showed that uniform convergence over data points is what is required to understand the generalization of networks . Dziugaite & Roy ( 2017 ) obtained non vacuous bounds . ( Neyshabur et al. , 2017 ) compared the effect of norm based control , sharpness and robustness on generalization . ( Neyshabur et al. , 2018 ) used spectral norm to give a bound which incorporated the weights of the neural networks . ( Arora et al. , 2018 ) obtained sharp generalization bounds in terms of sample complexity . ( Bietti et al. , 2019 ) used Reproducible kernel Hilbert spaces for studying regularization of deep neural network from analytic viewpoint . 3 MATHEMATICAL PRELIMINARIES . 3.1 POLYNOMIAL FRAMEWORK OF NEURAL NETWORKS . Consider the mathematical formulation of Neural Networks , a neural network N with l layers is a function which takes an input vector x and returns a vector N ( x ) such that : N ( x ) = fl ( Wl . . . f2 ( W2f1 ( W1x ) ) . . . ) ( 1 ) where Wi is the weight matrix of network at layer ith layer , fi is the corresponding non-linear activation function . If xl = [ x1l . . . x nl l ] is the output of the l th layer with nl nodes , then : xl = gl ( Wlxl−1 ) ( 2 ) As mentioned earlier if the activation functions are approximated using polynomials then mathematically there are only polynomials at each node at every layer in the network , so in our analysis we consider the activation functions as polynomials with certain degrees . Let a ( l−1 , l ) ij be the weight of the neural network from the ith node in l − 1th layer to the jth node in the lth layer and Pl be the Polynomial approximation of the activation function . Then based on the above mathematical description of neural networks the output at 2nd layer with n2 nodes is x2 = [ P1 ( n1∑ i=1 a ( 1,2 ) i1 x i 1 ) , P1 ( n1∑ i=1 a ( 1,2 ) i2 x i 1 ) , . . . , P1 ( n1∑ i=1 a ( 1,2 ) in2 xi1 ) ] ( 3 ) Similarly for the 3rd layer , x3 can be obtained as x3 = [ P2 n2∑ j=1 a ( 2,3 ) j1 P1 ( n1∑ i=1 a ( 1,2 ) i1 x i 1 ) , P2 n2∑ j=1 a ( 2,3 ) j2 P1 ( n1∑ i=1 a ( 1,2 ) i2 x i 1 ) , . . . , P2 n2∑ j=1 a ( 2,3 ) jn3 P1 ( n1∑ i=1 a ( 1,2 ) in2 xi1 ) ] ( 4 ) Clearly , by standard properties of polynomials the above tuple is also a polynomial . Now , if we look into the pattern followed by each layer then the general form of the output at any arbitrary lth layer can be obtained as xl = [ Pl−1 nl−1∑ j=1 a ( l−1 , l ) j1 Pl−2 ( nl−2∑ i=1 a ( l−2 , l−1 ) i1 x i l−2 ) , Pl−1 nl−1∑ j=1 a ( l−1 , l ) j2 Pl−2 ( nl−2∑ i=1 a ( l−2 , l−1 ) i2 x i l−2 ) , . . . , Pl−1 nl−1∑ j=1 a ( l−1 , l ) jnl Pl−2 ( nl−2∑ i=1 a ( l−2 , l−1 ) inl−1 xil−2 ) ] ( 5 ) By the same logic applied hitherto the above tuple is also a polynomial . The reason for such an unconventional formulation of a neural network as in the above framework is to analyze the degree of the polynomials using some of its basic properties . Regarding degree of polynomials we have the following standard results : For any c1 , c2 , . . . ck ∈ R and polynomials P1 , P2 , . . . Pk deg ( c1P1 + . . .+ ckPk ) ≤ sup i= { 1 , ... , k } |deg ( Pi ) | ( 6 ) where deg ( . ) is the degree of the polynomial and for any polynomial Pa and Pb deg ( PaPb ) = deg ( Pa ) deg ( Pb ) ( 7 ) By applying equation ( 7 ) on the first element of the vector xl from equation ( 5 ) we get deg Pl−1 nl−1∑ j=1 a ( l−1 , l ) j1 Pl−2 ( nl−2∑ i=1 a ( l−2 , l−1 ) i1 x i l−2 ) = deg ( Pl−1 ) deg nl−1∑ j=1 a ( l−1 , l ) j1 Pl−2 ( nl−2∑ i=1 a ( l−2 , l−1 ) i1 x i l−2 ) ( 8 ) Similarly , by applying equation ( 6 ) on equation ( 8 ) we obtain deg ( Pl−1 ) deg nl−1∑ j=1 a ( l−1 , l ) j1 Pl−2 ( nl−2∑ i=1 a ( l−2 , l−1 ) i1 x i l−2 ) ≤ deg ( Pl−1 ) |deg ( Pl−2 ) | ( 9 ) By iterating over all the layers in the network and applying equation ( 9 ) we get deg ( Pl ) ≤ deg ( Pl−1 ) deg ( Pl−2 ) . . . deg ( P1 ) ( 10 ) We make use of the above inequality in the next sections in obtaining the generalization bound . 3.2 COVERING NUMBER . Covering number of a general topological space counts the number of spherical balls needed to cover the entire space ( Munkres , 2000 ) . This is relevant to our present analysis as it gleans the behaviour of a neural network on an unknown set by understanding its behaviour on a known one . Formally , M ( , V ) is the covering number of the space V , conceptually this gives a measure of generalizability of the neural network on potentially infinite and unknown datapoints when trained on a finite subset ( i.e . the training dataset ) . Thus , any bound on the covering number would imply that we can predict the lower limit of the number of datapoints essential for accurate prediction on unknown sets . 3.3 VITUSHKIN ’ S INEQUALITY . The Vitushkin ’ s inequality ( Friedland & Yomdin , 2014 ; Yomdin , 2011 ) has been widely used in geometric functional analysis for studying the behaviour of level sets of analytic functions ( Kovalenko , 2017 ) . Along with its many applications this inequality can be used to bound the covering number of a metric space V using the lesbesgue measure ( Burk , 2011 ; Bartle , 2014 ) . Formally , let P ( x , n , d ) = P ( x1 , x2 , . . . , xn ) be a polynomial of degree d in n variables , Bn be a ball of unit radius in n-dimensions and Vρ ( P ) be the set of all polynomials that is bounded by ρ , i.e . : Vρ ( P ) = { x ∈ Bn : |P ( x , n , d ) | ≤ ρ } ( 11 ) Then according to Vitushkin ’ s inequality : M ( , V ) ≤ n−1∑ i=0 Ci ( n , d ) ( 1 ) i + µn ( V ) ( 1 ) n ≤Md ( ) + µn ( V ) ( 1 ) n ( 12 ) where µn ( V ) is the n-dimensional Lesbesgue measure ( Bartle , 2014 ) of the set and Md ( ) are variables defined as : Ci ( n , d ) , 2 i ( n i ) ( d− i ) i Md ( ) , n−1∑ i=0 Ci ( n , d ) ( 1 ) i ( 13 ) 3.4 METRIC ( N , D ) -SPAN . Metric ( n , d ) -span measures the accuracy of approximation of the covering number with Lesbesgue measure , that is to say it determines how well the covering number can be approximated by knowing the Lesbesgue measure of that set . This transforms the intractable problem of computing the covering number of the set V into a tractable one , this is extremely important as this quantifies the accuracy of computational experiments to that of theoretical expressions . More formally , for any subset Z ⊂ Bn , we define a constant ωd ( Z ) which is the metric ( n , d ) -span of set Z denoted by : ωd ( Z ) = sup ≥0 n [ M ( , Z ) −Md ( ) ] ( 14 )

Summary: The paper considers the problem of obtaining generalization bounds for deep neural networks. The paper uses tools from geometric function analysis to derive bounds for the covering number of neural networks, and derive generalization bounds using the covering numbers. The main strength of the bound as claimed by the paper is that they depend on Frobenius norm rather than the spectral norm, but I have several concerns about the results.