Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent-based Optimization

1 INTRODUCTION AND OVERVIEW . Neural networks used in practice often have sufficient effective capacity to learn arbitrary maps from their inputs to their outputs . This is typically demonstrated by training a classification network that achieves good test accuracy on a real dataset S , on a modified version of S ( call it S′ ) where the labels are randomized and observing that the training accuracy on S′ is very high , though , of course , the test accuracy is no better than chance ( Zhang et al. , 2017 ) . This leads to an important open question in the Deep Learning community ( Zhang et al . ( 2017 ) ; Arpit et al . ( 2017 ) ; Bartlett et al . ( 2017 ) ; Kawaguchi et al . ( 2017 ) ; Neyshabur et al . ( 2018 ) ; Arora et al . ( 2018 ) ; Belkin et al . ( 2019 ) ; Rahaman et al . ( 2019 ) ; Nagarajan & Kolter ( 2019 ) , etc . ) : Among all maps that fit a real dataset , how does Gradient Descent ( GD ) find one that generalizes well ? This is the question we address in this paper . We start by observing that this phenomenon is not limited to neural networks trained with GD but also applies to Random Forests and Decision Trees . However , there is no mystery with trees : A typical tree construction algorithm splits the training set recursively into similar subsets based on input features . If no similarity is found , eventually , each example is put into its own leaf to achieve good training accuracy ( but , of course , at the cost of poor generalization ) . Thus , trees that achieve good accuracy on a randomized dataset are much larger than those on a real dataset ( e.g . Chatterjee & Mishchenko ( 2019 , Expt . 5 ) ) . Is it possible that something similar happens with GD ? We believe so . The type of randomized-label experiments described above show that if there are common patterns to be found , then GD finds them . If not , it fits each example on a case-by-case basis . The question then is , what is it about the dynamics of GD that makes it possible to extract common patterns from the data ? And what does it mean for a pattern to be common ? Since the only change to the network parameters in GD comes from the gradients , the mechanism to detect commonality amongst examples must be through the gradients . We propose that this commonality detection can be explained as follows : 1 . Gradients are coherent , i.e , similar examples ( or parts of examples ) have similar gradients ( or similar components of gradients ) and dissimilar examples have dissimilar gradients . 2 . Since the overall gradient is the sum of the per-example gradients , it is stronger in directions where the per-example gradients are similar and reinforce each other and weaker in other directions where they are different and do not add up . 3 . Since network parameters are updated proportionally to gradients , they change faster in the direction of stronger gradients . 4 . Thus the changes to the network during training are biased towards those that simultaneously benefit many examples instead of a few ( or one example ) . For convenience , we refer to this as the Coherent Gradients hypothesis . It is instructive to work through the proposed mechanism in the context of a simple thought experiment . Consider a training set with two examples a and b . At some point in training , suppose the gradient of a , ga , can be decomposed into two orthogonal components ga1 and ga2 of roughly equal magnitude , i.e. , there are two , equally good , independent ways in which the network can better fit a ( by using say two disjoint parts of the network ) . Likewise , for b . Now , further suppose that one of the two ways is common to both a and b , i.e. , say ga2 = gb2 = gab , whereas , the other two are example specific , i.e. , 〈ga1 , gb1〉 = 0 . Now , the overall gradient is g = ga + gb = ga1 + 2 gab + gb1 . Observe that the gradient is stronger in the direction that simultaneously helps both examples and thus the corresponding parameter changes are bigger than those those that only benefit only one example.1 It is important to emphasize that the notion of similarity used above ( i.e. , which examples are considered similar ) is not a constant but changes in the course of training as network parameters change . It starts from a mostly task independent notion due to random initialization and is bootstrapped in the course of training to be task dependent . We say “ mostly ” because even with random initialization , examples that are syntactically close are treated similarly ( e.g. , two images differing in the intensities of some pixels as opposed to two images where one is a translated version of the other ) . The relationship between strong gradients and generalization can also be understood through the lens of algorithmic stability ( Bousquet & Elisseeff , 2002 ) : strong gradient directions are more stable since the presence or absence of a single example does not impact them as much , as opposed to weak gradient directions which may altogether disappear if a specific example is missing from the training set . With this observation , we can reason inductively about the stability of GD : since the initial values of the parameters do not depend on the training data , the initial function mapping examples to their gradients is stable . Now , if all parameter updates are due to strong gradient directions , then stability is preserved . However , if some parameter updates are due to weak gradient directions , then stability is diminished . Since stability ( suitably formalized ) is equivalent to generalization ( Shalev-Shwartz et al. , 2010 ) , this allows us to see how generalization may degrade as training progresses . Based on this insight , we shall see later how a simple modification to GD to suppress the weak gradient directions can dramatically reduce overfitting . In addition to providing insight into why GD generalizes in practice , we believe that the Coherent Gradients hypothesis can help explain several other empirical observations about deep learning in the literature : ( a ) Learning is slower with random labels than with real labels ( Zhang et al. , 2017 ; Arpit et al. , 2017 ) ( b ) Robustness to large amounts of label noise ( Rolnick et al. , 2017 ) ( c ) Early stopping leads to better generalization ( Caruana et al. , 2000 ) ( d ) Increasing capacity improves generalization ( Caruana et al. , 2000 ; Neyshabur et al. , 2018 ) ( e ) The existence of adversarial initialization schemes ( Liu et al. , 2019 ) ( f ) GD detects common patterns even when trained with random labels ( Chatterjee & Mishchenko , 2019 ) 1While the mechanism is easiest to see with full or large minibatches , we believe it holds even for small minibatches ( though there one has to consider the bias in updates over time ) . A direct experimental verification of the Coherent Gradients hypothesis is challenging since the notion of similarity between examples depends on the parameters of the network and thus changes during training . Our approach , therefore , is to design intervention experiments where we establish a baseline and compare it against variants designed to test some aspect or prediction of the theory . As part of these experiments , we replicate the observations ( a ) – ( c ) in the literature noted above , and analyze the corresponding explanations provided by Coherent Gradients ( §2 ) , and outline for future work how ( d ) – ( f ) may be accounted for ( §5 ) . In this paper , we limit our study to simple baselines : vanilla Stochastic Gradient Descent ( SGD ) on MNIST using fully connected networks . We believe that this is a good starting point , since even in this simple setting , with all frills eliminated ( e.g. , inductive bias from architecture or explicit regularization , or a more sophisticated optimization procedure ) , we are challenged to find a satisfactory explanation of why SGD generalizes well . Furthermore , our prior is that the difference between weak and strong directions is small at any one step of training , and therefore having a strong learning signal as in the case of MNIST makes a direct analysis of gradients easier . It also has the benefit of having a smaller carbon footprint and being easier to reproduce . Finally , based on preliminary experiments on other architectures and datasets we are optimistic that the insights we get from studying this simple setup apply more broadly . 2 EFFECT OF REDUCING SIMILARITY BETWEEN EXAMPLES . Our first test of the Coherent Gradients hypothesis is to see what happens when we reduce similarity between examples . Although , at any point during training , we do not know which examples are similar , and which are different , we can ( with high probability ) reduce the similarity among training examples simply by injecting label noise . In other words , under any notion of similarity , adding label noise to a dataset that has clean labels is likely to make similar examples less similar . Note that this perturbation does not reduce coherence since gradients still depend on the examples . ( To break coherence , we would have to make the gradients independent of the training examples which would requiring perturbing SGD itself and not just the dataset ) . 2.1 SETUP . For our baseline , we use the standard MNIST dataset of 60,000 training examples and 10,000 test examples . Each example is a 28x28 pixel grayscale handwritten digit along with a label ( ‘ 0 ’ – ‘ 9 ’ ) . We train a fully connected network on this dataset . The network has one hidden layer with 2048 ReLUs and an output layer with a 10-way softmax . We initialize it with Xavier and train using vanilla SGD ( i.e. , no momentum ) using cross entropy loss with a constant learning rate of 0.1 and a minibatch size of 100 for 105 steps ( i.e. , about 170 epochs ) . We do not use any explicit regularizers . We perturb the baseline by modifying only the dataset and keeping all other aspects of the architecture and learning algorithm fixed . The dataset is modified by adding various amounts of noise ( 25 % , 50 % , 75 % , and 100 % ) to the labels of the training set ( but not the test set ) . This noise is added by taking , say in the case of 25 % label noise , 25 % of the examples at random and randomly permuting their labels . Thus , when we add 25 % label noise , we still expect about 75 % + 0.1 * 25 % , i.e. , 77.5 % of the examples to have unchanged ( i.e . “ correct ” ) labels which we call the proper accuracy of the modified dataset . In what follows , we call examples with unchanged labels , pristine , and the remaining , corrupt . Also , from this perspective , it is convenient to refer to the original MNIST dataset as having 0 % label noise . We use a fully connected architecture instead of a convolutional one to mitigate concerns that some of the difference in generalization between the original MNIST and the noisy variants could stem from architectural inductive bias . We restrict ourselves to only 1 hidden layer to have the gradients be as well-behaved as possible . Finally , the network width , learning rate , and the number of training steps are chosen to ensure that exactly the same procedure is usually able to fit all 5 variants to 100 % training accuracy .

Summary: The surprising generalization properties of neural networks trained with stochastic gradient descent are still poorly understood. The present work suggests that they can be explained at least partly by the fact that patterns shared across many data points will lead to gradients pointing in similar directions, thus reinforcing each other. Artefacts specific to small numbers of data points however will not have this property and thus have a substantially smaller impact on the learning. Numerical experiments on MNIST with label-noise indeed show that even though the neural network is able to perfectly fit even the flipped labels, the "pristine" labels are fittet much earlier during training. The authors also experiment with explicitly clipping "outlier gradients" and show that the resulting algorithm drastically reduces overfitting, thus further supporting the coherent gradient hypothesis.