Optimization Variance:  Exploring Generalization Properties of DNNs

1 INTRODUCTION . Deep Neural Networks ( DNNs ) usually have large model capacity , but also generalize well . This violates the conventional VC dimension ( Vapnik , 1999 ) or Rademacher complexity theory ( ShalevShwartz & Ben-David , 2014 ) , inspiring new designs of network architectures ( Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2015 ; He et al. , 2016 ; Zagoruyko & Komodakis , 2016 ) and reconsideration of their optimization and generalization ( Zhang et al. , 2017 ; Arpit et al. , 2017 ; Wang et al. , 2018 ; Kalimeris et al. , 2019 ; Rahaman et al. , 2019 ; Allen-Zhu et al. , 2019 ) . Model-wise double descent , i.e. , as a DNN ’ s model complexity increases , its test error first shows a classical U-shaped curve and then enters a second descent , has been observed on many machine learning models ( Advani & Saxe , 2017 ; Belkin et al. , 2019a ; Geiger et al. , 2019 ; Maddox et al. , 2020 ; Nakkiran et al. , 2020 ) . Multiple studies provided theoretical evidence of this phenomenon in some tractable settings ( Mitra , 2019 ; Hastie et al. , 2019 ; Belkin et al. , 2019b ; Yang et al. , 2020 ; Bartlett et al. , 2020 ; Muthukumar et al. , 2020 ) . Specifically , Neal et al . ( 2018 ) and Yang et al . ( 2020 ) performed bias-variance decomposition for mean squared error ( MSE ) and the cross-entropy ( CE ) loss , and empirically revealed that the bell-shaped curve of the variance is the major cause of modelwise double descent . Maddox et al . ( 2020 ) proposed to measure the effective dimensionality of the parameter space , which can be further used to explain model-wise double descent . Recently , a new double descent phenomenon , epoch-wise double descent , was observed , when increasing the number of training epochs instead of the model complexity1 ( Nakkiran et al. , 2020 ) . Compared with model-wise double descent , epoch-wise double descent is relatively less explored . Heckel & Yilmaz ( 2020 ) showed that epoch-wise double descent occurs in the situation where different parts of DNNs are learned at different epochs , which can be eliminated by proper scaling of step sizes . Zhang & Wu ( 2020 ) discovered that the energy ratio of the high-frequency components of a DNN ’ s prediction landscape , which can reflect the model capacity , switches from increase to 1In practice , some label noise is often added to the training set to make the epoch-wise double descent more conspicuous . decrease at a certain training epoch , leading to the second descent of the test error . However , this metric fails to provide further information on generalization , such as the early stopping point , or how the size of a DNN influences its performance . This paper utilizes bias-variance decomposition of the zero-one ( ZO ) loss ( CE loss is still used in training ) to further investigate epoch-wise double descent . By monitoring the behaviors of the bias and the variance , we find that the variance plays an important role in epoch-wise double descent , which dominates and highly correlates with the variation of the test error . Though the variance correlates well with the test error , estimating its value requires training models on multiple different training sets drawn from the same data distribution , whereas in practice usually only one training set is available2 . Inspired by the fact that the source of variance comes from the random-sampled training sets , we propose a novel metric , optimization variance ( OV ) , to measure the diversity of model updates caused by the stochastic gradients of random training batches drawn in the same iteration . This metric can be estimated from a single model using samples drawn from the training set only . More importantly , it correlates well with the test error , and thus can be used to determine the early stopping point in DNN training , without using any validation set . Some complexity measures have been proposed to illustrate the generalization ability of DNNs , such as sharpness ( Keskar et al. , 2017 ) and norm-based measures ( Neyshabur et al. , 2015 ) . However , their values rely heavily on the model parameters , making comparisons across different models very difficult . Dinh et al . ( 2017 ) shows that by re-parameterizing a DNN , one can alter the sharpness of its searched local minima without affecting the function it represents ; Neyshabur et al . ( 2018 ) shows that these measures can not explain the generalization behaviors when the size of a DNN increases . Our proposed metric , which only requires the logit outputs of a DNN , is less dependent on model parameters , and hence can explain many generalization behaviors , e.g. , the test error decreases as the network size increases . Chatterji et al . ( 2020 ) proposed a metric called Model Criticality that can explain the superior generalization performance of some architectures over others , yet it is unexplored that whether this metric can be used to indicate generalization in the entire training procedure , especially for some relatively complex generalization behaviors , such as epoch-wise double descent . To summarize , our contributions are : • We perform bias-variance decomposition on the test error to explore epoch-wise double descent . We show that the variance dominates the variation of the test classification error . • We propose a novel metric , OV , which is calculated from the training set only and correlates well with the test classification error . • Based on the OV , we propose an approach to search for the early stopping point without using a validation set , when the zero-one loss is used in test . Experiments verified its effectiveness . The remainder of this paper is organized as follows : Section 2 introduces the details of tracing bias and variance over training epochs . Section 3 proposes the OV and demonstrates its ability to indicate the test behaviors . Section 4 draws conclusions and points out some future research directions . 2 BIAS AND VARIANCE IN EPOCH-WISE DOUBLE DESCENT . This section presents the details of tracing the bias and the variance during training . We show that the variance dominates the epoch-wise double descent of the test error . 2.1 A UNIFIED BIAS-VARIANCE DECOMPOSITION . Bias-variance decomposition is widely used to analyze the generalization properties of machine learning algorithms ( Geman et al. , 1992 ; Friedman et al. , 2001 ) . It was originally proposed for 2Assume the training set has n samples . We can partition it into multiple smaller training sets , each with m samples ( m < n ) , and then train multiple models . However , the variance estimated from this case would be different from the one estimated from training sets with n samples . We can also bootstrap the original training set into multiple ones , each with n samples . However , the data distribution of each bootstrap replica is different from the original training set , and hence the estimated variance would also be different . the MSE loss and later extended to other loss functions , e.g. , CE and ZO losses ( Kong & Dietterich , 1995 ; Tibshirani , 1996 ; Kohavi et al. , 1996 ; Heskes , 1998 ) . Our study utilizes a unified bias-variance decomposition that was proposed by Domingos ( 2000 ) and applicable to arbitrary loss functions . Let ( x , t ) be a sample drawn from the data distribution D , where x ∈ Rd denotes the ddimensional input , and t ∈ Rc the one-hot encoding of the label in c classes . The training set T = { ( xi , ti ) } ni=1 ∼ Dn is utilized to train the model f : Rd → Rc . Let y = f ( x ; T ) ∈ Rc be the probability output of the model f trained on T , and L ( t , y ) the loss function . The expected loss ET [ L ( t , y ) ] should be small to ensure that the model both accurately captures the regularities in its training data , and also generalizes well to unseen data . According to Domingos ( 2000 ) , a unified bias-variance decomposition3 of ET [ L ( y , t ) ] is : ET [ L ( t , y ) ] = L ( t , ȳ ) ︸ ︷︷ ︸ Bias +β ET [ L ( ȳ , y ) ] ︸ ︷︷ ︸ Variance , ( 1 ) where β takes different values for different loss functions , and ȳ is the expected output : ȳ = arg min y∗∈Rc ∣∣∑c k=1 y ∗ k=1 , y ∗ k≥0 ET [ L ( y∗ , y ) ] . ( 2 ) ȳ minimizes the variance term in ( 1 ) , which can be regarded as the “ center '' or “ ensemble '' of y w.r.t . different T . Table 1 shows specific forms of L , ȳ , and β for different loss functions ( the detailed derivations can be found in Appendix A ) . This paper focuses on the bias-variance decomposition of the ZO loss , because epoch-wise double descent of the test error is more obvious when the ZO loss is used ( see Appendix C ) . To capture the overall bias and variance , we analyzed Ex , tET [ L ( t , y ) ] , i.e. , the expectation of ET [ L ( t , y ) ] over the distribution D . 2.2 TRACE THE BIAS AND VARIANCE TERMS OVER TRAINING EPOCHS . To trace the bias term Ex , t [ L ( t , ȳ ) ] and the variance term Ex , tET [ L ( ȳ , y ) ] w.r.t . the training epoch , we need to sample several training sets and train models on them respectively , so that the bias and variance terms can be estimated from them . Concretely , let T ∗ denote the test set , f ( x ; Tj , q ) the model f trained on Tj ∼ Dn ( j = 1 , 2 , ... , K ) for q epochs . Then , the estimated bias and variance terms at the q-th epoch , denoted as B ( q ) and V ( q ) , respectively , can be written as : B ( q ) = E ( x , t ) ∈T ∗ [ L ( t , f̄ ( x ; q ) ) ] , ( 3 ) V ( q ) = E ( x , t ) ∈T ∗ 1 K K∑ j=1 L ( f̄ ( x ; q ) , f ( x ; Tj , q ) ) , ( 4 ) 3In real-world situations , the expected loss consists of three terms : bias , variance , and noise . Similar to Yang et al . ( 2020 ) , we view t as the groundtruth and ignore the noise term . where f̄ ( x ; q ) = H K∑ j=1 H ( f ( x ; Tj , q ) ) , ( 5 ) is the voting result of { f ( x ; Tj , q ) } Kj=1 . We should emphasize that , in real-world situations , D can not be obtained , hence Tj in our experiments was randomly sampled from the training set ( we sampled 50 % training data for each Tj ) . As a result , despite of showing the cause of epoch-wise double descent , the behaviors of bias and variance may be different when the whole training set is used . We considered ResNet ( He et al. , 2016 ) and VGG ( Simonyan & Zisserman , 2015 ) models4 trained on SVHN ( Netzer et al. , 2011 ) , CIFAR10 ( Krizhevsky , 2009 ) , and CIFAR100 ( Krizhevsky , 2009 ) . SGD and Adam ( Kingma & Ba , 2014 ) optimizers with different learning rates were used . The batchsize was set to 128 , and all models were trained for 250 epochs with data augmentation . Prior to sampling { Tj } Kj=1 ( K = 5 ) from the training set , 20 % labels of the training data were randomly shuffled to introduce epoch-wise double descent . Figure 1 shows the expected ZO loss and its bias and variance . The bias descends rapidly at first and then generally converges to a low value , whereas the variance behaves almost exactly the same as the test error , mimicking even small fluctuations of the test error . To stabilize that , we performed additional experiments with different optimizers , learning rates , and levels of label noise ( see Appendices E and G ) . All experimental results demonstrated that it is mainly the variance that contributes to epoch-wise double descent .

Summary: The paper under review studies the epoch wise double descent phenomena empirically. The epoch wise double descent phenomena is the observation that the risk of a large neural network trained with SGD first decreases, then increases, and finally decreases again as a function of the epochs or SGD steps. In addition, it proposes a quantity called ``optimization variance (OV)'', and it demonstrate that OV correlates with the test error. Based on this observation, it proposes to early stop when the OV reaches a minimum.