Is deeper better? It depends on locality of relevant features

1 INTRODUCTION . Deep learning has achieved unparalleled success in various tasks of artificial intelligence such as image classification ( Krizhevsky et al. , 2012 ; LeCun et al. , 2015 ) and speech recognition ( Hinton et al. , 2012 ) . Remarkably , in modern machine learning applications , impressive generalization performance has been observed in an overparameterized regime , in which the number of parameters in the network is much larger than that of training data samples . Contrary to what we learn in classical learning theory , an overparameterized network fits random labels and yet generalizes very well without serious overfitting ( Zhang et al. , 2017 ) . We do not have general theory that explains why deep learning works so well . Recently , the learning dynamics and the generalization power of heavily overparameterized wide neural networks have extensively been studied . It has been reported that training of an overparameterized network easily achieves zero training error without getting stuck in local minima of the loss landscape ( Zhang et al. , 2017 ; Baity-Jesi et al. , 2018 ) . Mathematically rigorous results have also been obtained ( Allen-Zhu et al. , 2019 ; Du et al. , 2019 ) . From a different perspective , theory of the neural tangent kernel ( NTK ) has been developed as a new tool to investigate an overparameterized network with an infinite width ( Jacot et al. , 2018 ; Arora et al. , 2019 ) , which simply explains the reason why a sufficiently wide neural network can achieve a global minimum of the training loss . As for generalization , a “ double-descent ” phenomenon has attracted much attention ( Spigler et al. , 2019 ; Belkin et al. , 2019 ) . The standard bias-variance tradeoff scenario predicts a U-shaped curve of the test error ( Geman et al. , 1992 ) ; however , one finds the double-descent curve , which implies that an increased model capacity beyond the interpolation threshold results in improved performance . This finding triggered detailed studies on the behavior of the bias and variance in an overparameterized regime ( Neal et al. , 2019 ; D ’ Ascoli et al. , 2020 ) . The double-descent phenomenon is not explained by traditional complexity measures such as the Vapnik-Chervonenkis dimension and the Rademacher complexity ( Mohri et al. , 2018 ) , and hence one seeks for new complexity measures of deep neural networks that can prove better generalization bounds ( Dziugaite & Roy , 2017 ; Neyshabur et al. , 2017 ; 2019 ; Arora et al. , 2018 ; Nagarajan & Kolter , 2017 ; Pérez et al. , 2019 ) . These theoretical efforts mainly focus on the effect of increasing the network width , but benefits of the network depth remain unclear . It is known that expressivity of a deep neural network grows exponentially with the depth rather than the width ( Poole et al. , 2016 ) . See also Bianchini & Scarselli ( 2014 ) ; Montúfar et al . ( 2014 ) . However , it is unclear whether exponential expressivity does lead to better generalization ( Ba & Caruana , 2014 ; Becker et al. , 2020 ) . It is also nontrivial whether typical problems encountered in practice require such high expressivity . Although some works ( Eldan & Shamir , 2016 ; Safran & Shamir , 2017 ) have shown that there exist simple and natural functions that are efficiently approximated by a network with two hidden layers but not by a network with one hidden layer , a recent work ( Malach & Shalev-Shwartz , 2019 ) has demonstrated that a deep network can only learn functions that are well approximated by a shallow network by using a gradient-based optimization algorithm , which indicates that benefits of the depth are not due to high expressivity of deep networks . Some other recent works have reported no clear advantage of the depth in an overparameterized regime ( Geiger et al. , 2019a ; b ) . To gain an insight into the advantage of the depth , in the present paper , we report our experimental study on the depth and width dependences of generalization in abstract but simple , well-controlled classification tasks with fully connected neural networks . We introduce local labels and global labels , both of which give simple mappings between inputs and output class labels . By “ local ” , we mean that the label is determined only by a few components of the input vector . On the other hand , a global label is determined by a global feature that is expressed as a sum of local quantities , and hence all components of an input contribute to the global label . Our experiments show strong depth-dependences of the generalization error for those simple input-output mappings . In particular , we find that deeper is better for local labels , while shallower is better for global labels . This result implies that the depth is not always advantageous , but the locality of relevant features would give us a clue for understanding the advantage the depth brings about . We also compare the generalization performance of a trained network of a finite width with that of the kernel method with the NTK . The latter corresponds to the infinite-width limit of a fully connected network with an appropriate initialization and an infinitesimal learning rate ( Jacot et al. , 2018 ) , which is referred to as the NTK limit . In the NTK limit , the network parameters stay close to their initial values during training , which is called the lazy learning ( Chizat et al. , 2019 ) . It is known that a wide but finite network can still be in the lazy learning regime for sufficiently small learning rates ( Ji & Telgarsky , 2019 ; Chen et al. , 2019 ) . We however find that even if the width increases , in some cases the generalization error at an optimal learning rate does not converge to the NTK limit . In such a case , a finite-width network shows much better generalization compared with the kernel learning with the NTK . This finding indicates the importance of the feature learning , in which network parameters change to learn relevant features . 2 SETTING . We consider a classification task with a training dataset D = { ( x ( µ ) , y ( µ ) ) : µ = 1 , 2 , . . . , N } , where x ( µ ) ∈ Rd is an input data and y ( µ ) ∈ { 1 , 2 , . . . , K } is its label . In this work , we consider the binary classification , K = 2 , unless otherwise stated . 2.1 DATASET . Each input x = ( x1 , x2 , . . . , xd ) T is a d-dimensional vector taken from i.i.d . Gaussian random variables of zero mean and unit variance , where aT is the transpose of vector a . For each input x , we assign a label y according to one of the following rules . k-LOCAL LABEL We randomly fix integers { i1 , i2 , . . . , ik } with 1 ≤ i1 < i2 < · · · < ik ≤ d. In the “ k-local ” label , the relevant feature is given by the product of the k components of an input x , that is , the label y is determined by y = { 1 if xi1xi2 . . . xik ≥ 0 ; 2 otherwise . ( 1 ) This label is said to be local in the sense that y is completely determined by just the k components of an input x.1 For fully connected networks considered in this paper , without loss of generality , we can choose i1 = 1 , i2 = 2 , . . . ik = k because of the permutation symmetry with respect to indices of input vectors . k-GLOBAL LABEL We again fix 1 ≤ i1 < i2 < · · · < ik ≤ d. Let us define M = d ∑ j=1 xj+i1xj+i2 . . . xj+ik , ( 2 ) where the convention xd+i = xi is used . The k-global label y for x is defined by y = { 1 if M ≥ 0 ; 2 otherwise . ( 3 ) The relevant feature M for this label is given by a uniform sum of the product of k components of the input vector . Every component of x contributes to this “ k-global ” label , in contrast to the k-local label with k < d . 2.2 NETWORK ARCHITECTURE . In the present work , we consider fully connected feedforward neural networks with L hidden layers of width H . We call L and H the depth and the width of the network , respectively . The output of the network f ( x ) for an input vector x ∈ Rd is determined as follows : f ( x ) = z ( L+1 ) = w ( L+1 ) z ( L ) + b ( L+1 ) ; z ( l ) = ϕ ( w ( l ) z ( l−1 ) + b ( l ) ) for l = 1 , 2 , . . . , L ; z ( 0 ) = x , ( 4 ) where ϕ ( x ) = max { x , 0 } is the component-wise ReLU activation function , z ( l ) is the output of the lth layer , and w ( l ) ∈ R K×H for l = L+ 1 ; R H×H for l = 2 , 3 , . . . , L ; R H×d for l = 1 , b ( l ) ∈ { R K for l = L+ 1 ; R H for l = 1 , 2 , . . . , L ( 5 ) are the weights and the biases , respectively . Let us denote by w the set of all the weights and biases in the network . We focus on an overparameterized regime , where the number of network parameters ( the number of components of w ) exceeds N , the number of training data points . 2.3 SUPERVISED LEARNING . The network parameters w are adjusted to correctly classify the training data . It is done by minimizing the softmax cross-entropy loss L ( w ) given by L ( w ) = 1 N N ∑ µ=1 ℓ ( f ( x ( µ ) ) , y ( µ ) ) , ℓ ( f ( x ) , y ) = − ln e fy ( x ) ∑K i=1 e fi ( x ) = −fy ( x ) + ln K ∑ i=1 efi ( x ) , ( 6 ) where the ith component of f ( x ) is denoted by fi ( x ) . The training of the network is done by the stochastic gradient descent ( SGD ) with learning rate η and the mini-batch size B . That is , for each mini-batch B ⊂ D with |B| = B , the network parameter 1The locality here does not necessarily imply that k points i1 , i2 , . . . , ik are spatially close to each other . Such a usage of the terminology “ k-local ” has been found ( but k-global is also called “ k-local ” ) in the field of quantum information ( Kempe et al. , 2006 ) . wt at time t is updated as wt+1 = wt − η∇wLB ( w ) , LB ( w ) = 1 B ∑ µ∈B ℓ ( f ( x ( µ ) ) , y ( µ ) ) . ( 7 ) Throughout the paper , we fix B = 50 . Meanwhile , we optimize η > 0 before training ( we explain the detail later ) . Biases are initialized to be zero , and weights are initialized using the Glorot initialization ( Glorot & Bengio , 2010 ) .2 The trained network classifies an input xµ to the class ŷ ( µ ) = argmaxi∈ { 1,2 , ... , K } fi ( x ( µ ) ) . Let us then define the training error as Etrain = 1 N N ∑ µ=1 ( 1− δy ( µ ) , ŷ ( µ ) ) , ( 8 ) that is the miss-classification rate for the training data D. We train our network until Etrain = 0 is achieved , i.e. , all the training data samples are correctly classified , which is possible in an overparameterized regime . For a training dataset D , we first perform the 10-fold cross validation to optimize the learning rate η under the Bayesian optimization method ( Snoek et al. , 2012 ) , and then perform the training via the SGD by using the full training dataset . In the optimization of η , we try to minimize the missclassification ratio for the validation data . The generalization performance of a trained network is measured by computing the test error . We prepare the test data Dtest = { ( x′ ( µ ) , y′ ( µ ) ) : µ = 1 , 2 , . . . , Ntest } independently from the training data D. The test error Etest is defined as the miss-classification ratio for Dtest , i , .e. , Etest = 1 Ntest Ntest ∑ µ=1 ( 1− δy′ ( µ ) , ŷ′ ( µ ) ) , ( 9 ) where ŷ′ ( µ ) = argmaxi fi ( x ′ ( µ ) ) is the prediction of our trained network . In our experiment discussed in Sec . 3 , we fix Ntest = 10 5 .

Summary: Recent theoretical study on the training of neural networks has introduced an important kernel function called neural tangent kernel. This paper studies the training of deep ReLU networks and compares it with the training directly using NTK by conducting experiments on synthetic data. Based on the experimental results, the authors conclude that deeper networks perform better on certain datasets whose labels are more “local”, while shallower networks are better at more “global” labels. Moreover, the authors observed that finite-width networks have better generalization than NTK.