On Nondeterminism and Instability in Neural Network Optimization

1 INTRODUCTION . Consider this common scenario : you have a baseline “ current best ” model , and are trying to improve it . Now , one of your experiments has produced a model whose metrics are slightly better than the baseline . Yet you have your reservations — how do you know the improvement is “ real ” , and not due to random fluctuations that create run-to-run variability ? Similarly , consider performing hyperparameter optimization , in which there are many possible values for a set of hyperparameters , and you find minor differences in performance between them . How do you pick the best hyperparameters , and how can you be sure that you ’ ve actually picked wisely ? In both scenarios , the standard practice is to perform multiple independent training runs of your model to understand its variability . While this does indeed help address the problem , it can be extremely wasteful , increasing the time required for effective research , using more computing power , and making reproducibility more difficult , while still leaving some uncertainty . Ultimately , the source of this problem is the nondeterminism in optimizing models — randomized components of model training that cause each training run to produce different models with their own performance characteristics . Nondeterminism itself occurs due to many factors : while the most salient source is the random initialization of parameters , other sources exist , including random shuffling of training data , per-example stochasticity of data augmentation , any explicit random operations ( e.g . dropout ( Srivastava et al. , 2014 ) ) , asynchronous model training ( Recht et al. , 2011 ) , and even nondeterminism in low-level libraries such as cuDNN ( Chetlur et al. , 2014 ) , which are present to improve throughput on hardware accelerators . Despite the clear impact nondeterminism has on the efficacy of modeling , relatively little attention has been paid towards understanding its mechanisms , even in the classical supervised setting . In this work , we establish an experimental protocol for analyzing the impact of nondeterminism in model training , allowing us to quantify the independent effect of each source of nondeterminism . In doing so , we make a surprising discovery : each source has nearly the same effect on the variability of final model performance . Further , we find each source produces models of similar diversity , as measured by correlations between model predictions , functional changes in model performance while ensembling , and state-of-the-art methods of model similarity ( Kornblith et al. , 2019 ) . To emphasize one particularly interesting result : nondeterminism in low-level libraries like cuDNN can matter just as much with respect to model diversity and variability as varying the entire network initialization . We explain this mystery by demonstrating that it can be attributed to an inherent numerical instability in optimizing neural networks — when training with SGD-like approaches , we show that small changes to initial parameters result in large changes to final parameter values . In fact , the instabilities in the optimization process are extreme : changing a single weight by the smallest possible amount within machine precision ( ∼6 ∗ 10−11 ) produces nearly as much variability as all other sources combined . Therefore , any source of nondeterminism that has any effect at all on model weights is doomed to inherit at least this level of variability . Last , we present promising results in reducing the effects of instability . While we find that many approaches result in no apparent change , we demonstrate that methods for accelerated model ensembling actually do reduce the variability of trained models without an increase in model training time , providing the first encouraging signs for tractability of the problem . 2 RELATED WORK . NONDETERMINISM . Relatively little prior work has studied the effects of nondeterminism on model optimization . Within reinforcement learning , nondeterminism is recognized as a significant barrier to reproducibility and evaluating progress in the field ( Nagarajan et al. , 2018 ; Henderson et al. , 2018 ; Islam et al. , 2017 ; Machado et al. , 2018 ) . In the setting of supervised learning , though , the focus of this work , the problem is much less studied . Madhyastha & Jain ( 2019 ) aggregate all sources of nondeterminism together into a single random seed and analyze the variability of model attention and accuracy as a function of it across various NLP datasets . They also propose a method for reducing this variability ( see Sec . A for details of our reproduction attempt ) . More common in the field , results across multiple random seeds are reported ( see Erhan et al . ( 2010 ) for a particularly extensive example ) , but the precise nature of nondeterminism ’ s influence on variability goes unstudied . INSTABILITY . We use the term “ stability ” to refer to numerical stability , in which a stable algorithm is one for which the final output ( converged model ) does not vary much as the input ( initial parameters ) are changed . Historically , the term “ stability ” has been used both in learning theory ( Bousquet & Elisseeff , 2002 ) , in reference to vanishing and exploding gradients ( Haber & Ruthotto , 2017 ) , and in the adversarial robustness community for a particular form of training ( Zheng et al. , 2016 ) . 3 NONDETERMINISM . Many sources of nondeterminism exist when optimizing neural networks , each of which can affect the variability and performance of trained models . We begin with a very brief overview : PARAMETER INITIALIZATION . When training a model , parameters without preset values are initialized randomly according to a given distribution , e.g . a Gaussian with mean 0 and variance determined by the number of input connections to the layer ( Glorot & Bengio , 2010 ; He et al. , 2015 ) . DATA SHUFFLING . In stochastic gradient descent ( SGD ) , the overall gradient is approximated by the gradient on a random subset of examples . Most commonly , this is implemented by shuffling the training data , after which the data is iterated through in order . Shuffling may happen either once , before training , or in between each epoch of training , the variant we use in this work . DATA AUGMENTATION . A very common practice , data augmentation refers to randomly altering each training example to artificially expand the training dataset . For example , in the case of images , it is common to randomly flip an image , which encourages invariance to left/right orientation . STOCHASTIC REGULARIZATION . Some forms of regularization , such as Dropout ( Srivastava et al. , 2014 ) , take the form of stochastic operations in a model during training . Dropout is the most common instance of this type of regularization , with a variety of others also in relatively common use , such as DropConnect ( Wan et al. , 2013 ) , variational dropout ( Gal & Ghahramani , 2016 ) , and variable length backpropagation through time ( Merity et al. , 2017 ) , among many others . LOW-LEVEL OPERATIONS . An underlooked source of nondeterminism , the very libraries that many deep learning frameworks are built on , such as cuDNN ( Chetlur et al. , 2014 ) often are run nondeterministically for performance reasons . This nondeterminism is small in magnitude — in one test we performed this caused a difference of roughly 0.003 % . In the case of cuDNN , the library we test with , it is possible to disable nondeterministic behavior , incurring a speed penalty typically on the order of∼15 % . However , unlike the other sources of nondeterminism , it is not possible to “ seed ” this nondeterminism ; it is only possible to turn it on or off , but not control its nondeterministic behavior . 3.1 PROTOCOL FOR TESTING EFFECTS OF NONDETERMINISM . DIVERSITY IN PERFORMANCE . Our protocol for testing the effects of sources of nondeterminism is based on properly controlling for each source . In general , suppose there are N sources of nondeterminism , with source i controlled by a seed Si . To test the effect of source i , we keep all values Sj , j 6= i set to a constant , and vary Si with R different values , where R is the number of independent training runs performed . For sources of nondeterminism which can not be effectively seeded , such as cuDNN , we indicate one of these values as the deterministic value , which it must be set to when varying the other sources of nondeterminism . For example , suppose that we wish to study three sources of nondeterminism , denoting S1 the seed for random parameter initialization , S2 for training data shuffling , and S3 for cuDNN , where S3 = 1 is the deterministic value for cuDNN . To test the effect of random parameter initialization , with a budget of R = 30 training runs , then we set S3 to the deterministic value of 1 , S2 to an arbitrary constant ( also 1 for simplicity ) , and test 30 different values of S1 . All together , this corresponds to training models for ( S1 , S2 , S3 ) ∈ { ( i , 1 , 1 ) } 30i=1 , producing a set of 30 models . To look at variability according to a particular evaluation metric ( e.g . cross-entropy or accuracy for classification ) , we calculate the standard deviation ( across all R = 30 models ) of the metric . Note that it is also possible to test the effects of several sources of nondeterminism in tandem this way . For example , to test all sources of nondeterminism together , the set of models can be changed to ( S1 , S2 , S3 ) ∈ { ( i , i , 0 ) } Ri=1 . DIVERSITY IN REPRESENTATION . Beyond looking at diversity of test set generalization , though , it is worth examining how different the representations of trained models actually are — even though the diversity in performance might be similar between models trained with different types of nondeterminism , it might be the case that one type of nondeterminism produces models that have learned largely similar concepts , with the variance in generalization due to other factors . In order to rigorously examine these , we consider four distinct analyses on the functional behavior of models : The first and most straightforward metric we consider is the average disagreement between pairs of models , where higher disagreement corresponds to higher levels of diversity . In contrast to our other metrics , this considers only the argmax of a model ’ s predictions , which makes it both the most limited and the most interpretable of the group . This metric has also been used recently to compare network similarity in the context of network ensembles ( Fort et al. , 2019 ) . Second , we consider the average correlation between the predictions of two models , i.e . the expectation ( across pairs of models from the same nondeterminism source ) , of the correlation of predictions , calculated across examples and classes . For example , for a classification problem , the predicted logits from each of R models are flattened into vectors of length N ∗ C ( with N the number of test examples and C the number of classes ) , and we calculate the mean correlation coefficient of the predictions across all ( R 2 ) pairs of models . We use Spearman ’ s ρ for the correlation coefficient , but note that others such as Pearson ’ s r are possible and yield similar conclusions . For this metric , a lower score indicates a more diverse set of models . The third analysis we perform examines the change in performance from ensembling two models from the same source of nondeterminism . The intuition is as follows : If a pair of models are completely redundant , then ensembling them would result in no change in performance . However , if models actually learn different representations , then we expect an improvement from ensembling , with a greater improvement the greater the diversity in a set of models . Denoting by f ( Si ) some particular evaluation metric f calculated on the predictions of model Si , this change is equivalent to : 1 ( R 2 ) R∑ i=1 R∑ j=i+1 ( f ( Si + Sj 2 ) − f ( Si ) + f ( Sj ) 2 ) ( 1 ) Last , for a more detailed view of learned representations within a network , we consider a stateof-the-art method for measuring the similarity of learned neural network representations , centered kernel alignment ( CKA ) ( Kornblith et al. , 2019 ) , which has been used to analyze models trained with different random initializations , widths , and even entirely different architectures . We use the linear version of CKA , which Kornblith et al . ( 2019 ) found performed similarly to an RBF kernel .

Summary: This paper sheds light on the impact of nondeterminism to the run-to-run variability of neural network performance---a situation many people using neural networks have experienced. The authors establish an experimental strategy to analyze the different sources of nondeterminism. Some sources of nondeterminism are parameter initialization, data shuffling, data augmentation, regularization and cuDNN.