On the Reproducibility of Neural Network Predictions

1 INTRODUCTION . Deep neural networks ( DNNs ) have seen remarkable success in a range of complex tasks , and significant effort has been spent on further improving their predictive accuracy . However , an equally important desideratum of any machine learning system is stability or reproducibility in its predictions . In practice , machine learning models are continuously ( re ) -trained as new data arrives , or to incorporate architectural and algorithmic changes . A model that changes its predictions on a significant fraction of examples after each update is undesirable , even if each model instantiation attains high accuracy . Reproducibility of predictions is a challenge even if the architecture and training data are fixed across different training runs , which is the focus of this paper . Unfortunately , two key ingredients that help deep networks attain high accuracy — over-parameterization , and the randomization of their training algorithms — pose significant challenges to their reproducibility . The former refers to the fact that NNs typically have many solutions that minimize the training objective ( Neyshabur et al. , 2015 ; Zhang et al. , 2017 ) . The latter refers to the fact that standard training of NNs involves several sources of randomness , e.g. , initialization , mini-batch ordering , non-determinism in training platforms and in some cases data augmentation . Put together , these imply that NN training can find vastly different solutions in each run even when training data is the same , leading to a reproducibility challenge . The prediction disagreement between two models is referred to as churn ( Cormier et al. , 2016 ) 1 . Concretely , given two models , churn is the fraction of test examples where the predictions of the two models disagree . Clearly , churn is zero if both models have perfect accuracy – an unattainable goal for most of the practical settings of interest . Similarly , one can mitigate churn by eliminating all sources of randomness in the underlying training setup . However , even if one controls the seed used for random initialization and the order of data , inherent non-determinism in the current computation platforms is hard to avoid ( see §2.3 ) . Moreover it is desirable to have stable models with predictions 1 Madani et al . ( 2004 ) referred to this as disagreement and used it as an estimate for generalization error and model selection unaffected by such factors in training . Thus , it is critical to quantify churn , and develop methods that reduce it . In this paper , we study the problem of churn in NNs for the classification setting . We demonstrate the presence of churn , and investigate the role of different training factors causing it . Interestingly our experiments show that churn is not avoidable on the computing platforms commonly used in machine learning , further highlighting the necessity of developing techniques to mitigate churn . We then analyze the relation between churn and predicted class probabilities . Based on this , we develop a novel regularized co-distillation approach for reducing churn . Our key contributions are summarized below : ( i ) Besides the disagreement in the final predictions of models , we propose alternative soft metrics to measure churn . We demonstrate the existence of churn on standard image classification tasks ( CIFAR-10 , CIFAR-100 , ImageNet , SVHN and iNaturalist ) , and identify the components of learning algorithms that contribute to the observed churn . Furthermore , we analyze the relationship between churn and model prediction confidences ( cf . § 2 ) . ( ii ) Motivated from our analysis , we propose a regularized co-distillation approach to reduce churn that both improves prediction confidences and reduces prediction variance ( cf . §3 ) . Our approach consists of two components : a ) minimum entropy regularizers that improve prediction confidences ( cf . §3.1 ) , and b ) a new variant of co-distillation ( Anil et al. , 2018 ) to reduce prediction variance across runs . Specifically , we use a symmetric KL divergence based loss to reduce model disagreement , with a linear warmup and joint updates across multiple models ( cf . §3.2 ) . ( iii ) We empirically demonstrate the effectiveness of the proposed approach in reducing churn and ( sometimes ) increasing accuracy . We present ablation studies over its two components to show their complementary nature in reducing churn ( cf . §4 ) . 1.1 RELATED WORK . Reproducibility in machine learning . There is a broad field studying the problem of reproducible research ( Buckheit & Donoho , 1995 ; Gentleman & Lang , 2007 ; Sonnenburg et al. , 2007 ; Kovacevic , 2007 ; Mesirov , 2010 ; Peng , 2011 ; McNutt , 2014 ; Braun & Ong , 2014 ; Rule et al. , 2018 ) , which identifies best practices to facilitate the reproducibility of scientific results . Henderson et al . ( 2018 ) analysed reproducibility of methods in reinforcement learning , showing that performance of certain methods is sensitive to the random seed used in the training . While the performance of NNs on image classification tasks is fairly stable ( Table 2 ) , we focus on analyzing and improving the reproducibility of individual predictions . Thus , churn can be seen as a specific technical component of this reproducibility challenge . Cormier et al . ( 2016 ) defined the disagreement between predictions of two models as churn . They proposed an MCMC approach to train an initial stable model A so that it has a small churn with its future version , say model B . Here , future versions are based on slightly modified training data with possibly additional features . In Goh et al . ( 2016 ) ; Cotter et al . ( 2019 ) , constrained optimization is utilized to reduce churn across different model versions . In contrast , we are interested in capturing the contribution of factors other than training data modification that cause churn . More recently . Madhyastha & Jain ( 2019 ) study instability in the interpretation mechanisms and average performance for deep NNs due to change in random seed , and propose a stochastic weight averaging ( Izmailov et al. , 2018 ) approach to promote robust interpretations . In contrast , we are interested in robustness of individual predictions . Ensembling and online distillation . Ensemble methods ( Dietterich , 2000 ; Lakshminarayanan et al. , 2017 ) that combine the predictions from multiple ( diverse ) models naturally reduce the churn by averaging out the randomness in the training procedure of the individual models . However , such methods incur large memory footprint and high computational cost during the inference time . Distillation ( Hinton et al. , 2015 ; Bucilua et al. , 2006 ) aims to train a single model from the ensemble to alleviate these costs . Even though distilled model aims to recover the accuracy of the underlying ensemble , it is unclear if the distilled model also leads to churn reduction . Furthermore , distillation is a two-stage process , involving first training an ensemble and then distilling it into a single model . To avoid this two-stage training process , multiple recent works Anil et al . ( 2018 ) ; Zhang et al . ( 2018 ) ; Lan et al . ( 2018 ) ; Song & Chai ( 2018 ) ; Guo et al . ( 2020 ) have focused on online distillation , where multiple identical or similar models ( with different initialization ) are trained while regularizing the distance between their prediction probabilities . At the end of the training , any of the participating models can be used for inference . Notably , Anil et al . ( 2018 ) , while referring to this approach as co-distillation , also empirically pointed out its utility for churn reduction on the Criteo Ad dataset2 . In contrast , we develop a deeper understanding of co-distillation framework as a churn reduction mechanism by providing a theoretical justification behind its ability to reduce churn . We experimentally show that using a symmetric KL divergence objective instead of the cross entropy loss for co-distillation ( Anil et al. , 2018 ) leads to lower churn and better accuracy , even improving over the expensive ensembling-distillation approach . Entropy regularizer . Minimum entropy regularization was earlier explored in the context of semi-supervised learning ( Grandvalet & Bengio , 2005 ) . Such techniques have also been used to combat label noise ( Reed et al. , 2015 ) . In contrast , we utilize minimum entropy regularization in fully supervised settings for a distinct purpose of reducing churn and experimentally show its effectiveness . 1.2 NOTATION . Multi-class classification . We consider a multi-class classification setting , where given an instance x P X , the goal is to classify it as a member of one of K classes , indexed by the set Y , rKs . Let W be the set of parameters that define the underlying classification models . In particular , for w PW , the associated classification model fp¨ ; wq : X Ñ ∆K maps the instance x P X in the K-dimensional simplex ∆K Ă RK . Given fpx ; wq , x is classified as element of class ŷx ; w such that ŷx ; w “ arg maxjPY fpx ; wqj . ( 1 ) This gives the misclassification error ` 01 ` y , ŷx ; w ˘ “ 1tŷx ; w‰yu , where y is the true label for x . Let PX , Y be the joint distribution over the instance and label pairs . We learn a classification model by minimizing the risk for some valid surrogate loss ` of the misclassification error ` 01 : L ` w ˘ , EX , Y “ ` pY , fpX ; wqq ‰ . In practice , since we have only finite samples S P pXˆ Yqn , we minimize the corresponding empirical risk . L ` w ; S ˘ , 1 |S| ÿ px , yqPS ` ` y , fpx ; wq ˘ . ( 2 ) 2 CHURN : MEASUREMENT AND ANALYSIS . In this section we define churn , and demonstrate its existence on CIFAR and ImageNet datasets . We also propose and measure alternative soft metrics to quantify churn , that mitigate the discontinuity of churn . Subsequently , we examine the influence of different factors in the learning algorithm on churn . Finally , we present a relation between churn and prediction confidences of the model . We begin by defining churn as the expected disagreement between the predictions of two models ( Cormier et al. , 2016 ) . Definition 1 ( Churn between two models ) . Let w1 , w2 P W define classification models fp¨ ; w1q , fp¨ ; w2q : XÑ ∆K , respectively . Then , the churn between the two models is Churnpw1 , w2q “ EX “ 1tŶx ; w1‰Ŷx ; w2u ‰ “ PX “ ŶX ; w1 ‰ ŶX ; w2 ‰ , ( 3 ) where Ŷx ; w1 , arg maxjPY fpx ; w1qj and Ŷx ; w2 , arg maxjPY fpx ; w2qj . Note that if the models have perfect test accuracy , then their predictions always agree with the true label , which corresponds to zero churn . In practice , however , this is rarely the case . The following rather straightforward result shows that churn is upper bounded by the sum of the test error of the models . See Appendix B for the proof . We note that a similar result was shown in Theorem 1 of Madani et al . ( 2004 ) . Lemma 1 . Let PErr , w1 , PX , YrY ‰ ŶX ; w1s and PErr , w2 , PX , YrY ‰ ŶX ; w2s be the misclassification error for the models w1 and w2 , respectively . Then , Churnpw1 , w2q ď PErr , w1 ` PErr , w2 . 2https : //www.kaggle.com/c/criteo-display-ad-challenge Despite the worst-case bound in Lemma 1 , imperfect accuracy does not preclude the absence of churn . As the best-case scenario , two imperfect models can agree on the predictions for each example ( whether correct or incorrect ) , causing the churn to be zero . For example , multiple runs of a deterministic learning algorithm produce models with zero churn , independent of their accuracy . This shows that , in general , one can not infer churn from test accuracy , and understanding churn of an algorithm requires independent exploration .

Summary: The paper proposes methods to address churn in deep neural networks for classification, defined as the extent of disagreements in predictions of two models trained on the same data with the same algorithm. In addition to an existing measure of churn that is based on exact match of predicted classes, the paper introduces a soft measure of churn that measures disagreement by comparing the two models' class probability distributions. The paper proposes three regularization terms that can be added to the primary loss function used during training to reduce churn: two single-model regularization terms (based on cross entropy and KL divergence respectively) that encourage the model to output a more uneven probability distribution for an example, and a divergence-based term that is used by training two models simultaneously and that imposes a KL-divergence-based penalty that encourages the two models to output probability distributions that are as similar as possible. Experiments with ResNet architectures on CIFAR-10/100 and ImageNet indicate that the proposed approaches and their combination indeed reduce churn and do so to a larger extent than the two-model divergence-based approach applied in conjunction with cross-entropy that was proposed in work by Anil et al. in 2018 (although the improvement seems quite minor on ImageNet).