Year: 2020

Abstract:
Standard training techniques for neural networks involve multiple sources of randomness, e.g., initialization, mini-batch ordering and in some cases data augmentation. Given that neural networks are heavily over-parameterized in practice, such randomness can cause churn – disagreements between predictions of the two models independently trained by the same algorithm, contributing to the ‘reproducibility challenges’ in modern machine learning. In this paper, we study this problem of churn, identify factors that cause it, and propose two simple means of mitigating it. We first demonstrate that churn is indeed an issue, even for standard image classification tasks (CIFAR and ImageNet), and study the role of the different sources of training randomness that cause churn. By analyzing the relationship between churn and prediction confidences, we pursue an approach with two components for churn reduction. First, we propose using minimum entropy regularizers to increase prediction confidences. Second, we present a novel variant of co-distillation approach (Anil et al., 2018) to increase model agreement and reduce churn. We present empirical results showing the effectiveness of both techniques in reducing churn while improving the accuracy of the underlying model.

heading: 1 INTRODUCTION
text: Deep neural networks (DNNs) have seen remarkable success in a range of complex tasks, and significant effort has been spent on further improving their predictive accuracy. However, an equally important desideratum of any machine learning system is stability or reproducibility in its predictions. In practice, machine learning models are continuously (re)-trained as new data arrives, or to incorporate architectural and algorithmic changes. A model that changes its predictions on a significant fraction of examples after each update is undesirable, even if each model instantiation attains high accuracy.
Reproducibility of predictions is a challenge even if the architecture and training data are fixed across different training runs, which is the focus of this paper. Unfortunately, two key ingredients that help deep networks attain high accuracy — over-parameterization, and the randomization of their training algorithms — pose significant challenges to their reproducibility. The former refers to the fact that NNs typically have many solutions that minimize the training objective (Neyshabur et al., 2015; Zhang et al., 2017). The latter refers to the fact that standard training of NNs involves several sources of randomness, e.g., initialization, mini-batch ordering, non-determinism in training platforms and in some cases data augmentation. Put together, these imply that NN training can find vastly different solutions in each run even when training data is the same, leading to a reproducibility challenge.
The prediction disagreement between two models is referred to as churn (Cormier et al., 2016)1. Concretely, given two models, churn is the fraction of test examples where the predictions of the two models disagree. Clearly, churn is zero if both models have perfect accuracy – an unattainable goal for most of the practical settings of interest. Similarly, one can mitigate churn by eliminating all sources of randomness in the underlying training setup. However, even if one controls the seed used for random initialization and the order of data, inherent non-determinism in the current computation platforms is hard to avoid (see §2.3). Moreover it is desirable to have stable models with predictions
1 Madani et al. (2004) referred to this as disagreement and used it as an estimate for generalization error and model selection
unaffected by such factors in training. Thus, it is critical to quantify churn, and develop methods that reduce it.
In this paper, we study the problem of churn in NNs for the classification setting. We demonstrate the presence of churn, and investigate the role of different training factors causing it. Interestingly our experiments show that churn is not avoidable on the computing platforms commonly used in machine learning, further highlighting the necessity of developing techniques to mitigate churn. We then analyze the relation between churn and predicted class probabilities. Based on this, we develop a novel regularized co-distillation approach for reducing churn.
Our key contributions are summarized below:
(i) Besides the disagreement in the final predictions of models, we propose alternative soft metrics to measure churn. We demonstrate the existence of churn on standard image classification tasks (CIFAR-10, CIFAR-100, ImageNet, SVHN and iNaturalist), and identify the components of learning algorithms that contribute to the observed churn. Furthermore, we analyze the relationship between churn and model prediction confidences (cf. § 2).
(ii) Motivated from our analysis, we propose a regularized co-distillation approach to reduce churn that both improves prediction confidences and reduces prediction variance (cf. §3). Our approach consists of two components: a) minimum entropy regularizers that improve prediction confidences (cf. §3.1), and b) a new variant of co-distillation (Anil et al., 2018) to reduce prediction variance across runs. Specifically, we use a symmetric KL divergence based loss to reduce model disagreement, with a linear warmup and joint updates across multiple models (cf. §3.2).
(iii) We empirically demonstrate the effectiveness of the proposed approach in reducing churn and (sometimes) increasing accuracy. We present ablation studies over its two components to show their complementary nature in reducing churn (cf. §4).

heading: 1.1 RELATED WORK
text: Reproducibility in machine learning. There is a broad field studying the problem of reproducible research (Buckheit & Donoho, 1995; Gentleman & Lang, 2007; Sonnenburg et al., 2007; Kovacevic, 2007; Mesirov, 2010; Peng, 2011; McNutt, 2014; Braun & Ong, 2014; Rule et al., 2018), which identifies best practices to facilitate the reproducibility of scientific results. Henderson et al. (2018) analysed reproducibility of methods in reinforcement learning, showing that performance of certain methods is sensitive to the random seed used in the training. While the performance of NNs on image classification tasks is fairly stable (Table 2), we focus on analyzing and improving the reproducibility of individual predictions. Thus, churn can be seen as a specific technical component of this reproducibility challenge.
Cormier et al. (2016) defined the disagreement between predictions of two models as churn. They proposed an MCMC approach to train an initial stable model A so that it has a small churn with its future version, say model B. Here, future versions are based on slightly modified training data with possibly additional features. In Goh et al. (2016); Cotter et al. (2019), constrained optimization is utilized to reduce churn across different model versions. In contrast, we are interested in capturing the contribution of factors other than training data modification that cause churn.
More recently. Madhyastha & Jain (2019) study instability in the interpretation mechanisms and average performance for deep NNs due to change in random seed, and propose a stochastic weight averaging (Izmailov et al., 2018) approach to promote robust interpretations. In contrast, we are interested in robustness of individual predictions.
Ensembling and online distillation. Ensemble methods (Dietterich, 2000; Lakshminarayanan et al., 2017) that combine the predictions from multiple (diverse) models naturally reduce the churn by averaging out the randomness in the training procedure of the individual models. However, such methods incur large memory footprint and high computational cost during the inference time. Distillation (Hinton et al., 2015; Bucilua et al., 2006) aims to train a single model from the ensemble to alleviate these costs. Even though distilled model aims to recover the accuracy of the underlying ensemble, it is unclear if the distilled model also leads to churn reduction. Furthermore, distillation is a two-stage process, involving first training an ensemble and then distilling it into a single model.
To avoid this two-stage training process, multiple recent works Anil et al. (2018); Zhang et al. (2018); Lan et al. (2018); Song & Chai (2018); Guo et al. (2020) have focused on online distillation, where
multiple identical or similar models (with different initialization) are trained while regularizing the distance between their prediction probabilities. At the end of the training, any of the participating models can be used for inference. Notably, Anil et al. (2018), while referring to this approach as co-distillation, also empirically pointed out its utility for churn reduction on the Criteo Ad dataset2. In contrast, we develop a deeper understanding of co-distillation framework as a churn reduction mechanism by providing a theoretical justification behind its ability to reduce churn. We experimentally show that using a symmetric KL divergence objective instead of the cross entropy loss for co-distillation (Anil et al., 2018) leads to lower churn and better accuracy, even improving over the expensive ensembling-distillation approach.
Entropy regularizer. Minimum entropy regularization was earlier explored in the context of semi-supervised learning (Grandvalet & Bengio, 2005). Such techniques have also been used to combat label noise (Reed et al., 2015). In contrast, we utilize minimum entropy regularization in fully supervised settings for a distinct purpose of reducing churn and experimentally show its effectiveness.

heading: 1.2 NOTATION
text: Multi-class classification. We consider a multi-class classification setting, where given an instance x P X, the goal is to classify it as a member of one of K classes, indexed by the set Y , rKs. Let W be the set of parameters that define the underlying classification models. In particular, for w PW, the associated classification model fp¨;wq : X Ñ ∆K maps the instance x P X in the K-dimensional simplex ∆K Ă RK . Given fpx;wq, x is classified as element of class ŷx;w such that
ŷx;w “ arg maxjPY fpx;wqj . (1)
This gives the misclassification error `01 ` y, ŷx;w ˘ “ 1tŷx;w‰yu, where y is the true label for x. Let PX,Y be the joint distribution over the instance and label pairs. We learn a classification model by minimizing the risk for some valid surrogate loss ` of the misclassification error `01: L ` w ˘
, EX,Y “ `pY, fpX;wqq ‰
. In practice, since we have only finite samples S P pXˆ Yqn, we minimize the corresponding empirical risk.
L ` w;S ˘ , 1 |S| ÿ px,yqPS ` ` y, fpx;wq ˘ . (2)

heading: 2 CHURN: MEASUREMENT AND ANALYSIS
text: In this section we define churn, and demonstrate its existence on CIFAR and ImageNet datasets. We also propose and measure alternative soft metrics to quantify churn, that mitigate the discontinuity of churn. Subsequently, we examine the influence of different factors in the learning algorithm on churn. Finally, we present a relation between churn and prediction confidences of the model.
We begin by defining churn as the expected disagreement between the predictions of two models (Cormier et al., 2016).
Definition 1 (Churn between two models). Let w1, w2 P W define classification models fp¨;w1q, fp¨;w2q : XÑ ∆K , respectively. Then, the churn between the two models is
Churnpw1, w2q “ EX “ 1tŶx;w1‰Ŷx;w2u ‰ “ PX “ ŶX;w1 ‰ ŶX;w2 ‰ , (3)
where Ŷx;w1 , arg maxjPY fpx;w1qj and Ŷx;w2 , arg maxjPY fpx;w2qj .
Note that if the models have perfect test accuracy, then their predictions always agree with the true label, which corresponds to zero churn. In practice, however, this is rarely the case. The following rather straightforward result shows that churn is upper bounded by the sum of the test error of the models. See Appendix B for the proof. We note that a similar result was shown in Theorem 1 of Madani et al. (2004).
Lemma 1. Let PErr,w1 , PX,YrY ‰ ŶX;w1s and PErr,w2 , PX,YrY ‰ ŶX;w2s be the misclassification error for the models w1 and w2, respectively. Then, Churnpw1, w2q ď PErr,w1 ` PErr,w2 .
2https://www.kaggle.com/c/criteo-display-ad-challenge
Despite the worst-case bound in Lemma 1, imperfect accuracy does not preclude the absence of churn. As the best-case scenario, two imperfect models can agree on the predictions for each example (whether correct or incorrect), causing the churn to be zero. For example, multiple runs of a deterministic learning algorithm produce models with zero churn, independent of their accuracy.
This shows that, in general, one cannot infer churn from test accuracy, and understanding churn of an algorithm requires independent exploration.

heading: 2.1 DEMONSTRATION OF CHURN
text: In principle, there are multiple sources of randomness in standard training procedures that make NNs susceptible to churn. We now verify this hypothesis by showing that, in practice, these sources indeed result in a non-trivial churn for NNs on standard image classification datasets.
Table 2 reports churn over CIFAR-10, CIFAR-100 and ImageNet, with ResNets (He et al., 2016a;b) as the underlying model architecture. As per the standard practice in the literature, we employ SGD with momentum and step-wise learning rate decay to train ResNets. Note that the models obtained in different runs indeed disagree in their predictions substantially. To measure if the disagreement comes mainly from misclassified examples, we measure churn on two slices of examples, correctly classified (Churn correct) and misclassified (Churn incorrect). We notice churn even among the examples that are correctly classified, though we see a relatively higher churn among the misclassified examples.
This raises two natural questions: (i) do the prediction probabilities differ significantly, or is the churn observed in Table 2 mainly an artifact of the argmax operation in (1), when faced with small variation in prediction probabilities across models?, and (ii) what causes such a high churn across runs? We address these questions in the following sections.

heading: 2.2 SURROGATE CHURN
text: Churn in Table 2 could potentially be a manifestation of applying the arg max operation in (1), despite the prediction probabilities being close. To study this, we consider the following soft metric to measure churn, which takes the models’ entire prediction probability mass function into account. Definition 2 (Surrogate churn between two models). Let fp¨;w1q, fp¨;w2q : XÑ ∆K be two models defined by w1, w2 PW , respectively. Then, for α P R`, the surrogate churn between the models is
SChurnαpw1, w2q “ 1
2 ¨ EX
„ ›
›
›
›
ˆ
fpX;w1q max fpX;w1q
˙α
´ ˆ fpX;w2q max fpX;w2q ˙α› › › ›
1

. (4)
As αÑ8, this reduces to the standard Churn definition in (3). In Table 2 we measure SChurnα for α “ 1, which shows that even the distance between prediction probabilities is significant across runs. Thus, the churn observed in Table 2 is not merely caused by the discontinuity of the arg max, but it indeed highlights the instability of model predictions caused by randomness in training.

heading: 2.3 WHAT CAUSES CHURN?
text: We now investigate the role played by randomness in initialization, mini-batch ordering, data augmentation, and non-determinism in the computation platform in causing churn. Even though these aspects are sources of randomness in training, they are not necessarily sources of churn. We experiment by holding some of these factors constant and measure the churn across 5 different runs. We report results in Table 1, where the first column gives the baseline churn with no factor held constant across runs.
Model initialization is a source of churn. Simply initializing weights from identical seeds (odd columns in Table 1) can decrease churn under most settings. Other sources of randomness include mini-batch ordering and dataset augmentation. To hold the former constant, we ensure that every model iterates over the dataset in the same order; to hold the latter constant, we remove all augmentation during training. These two aspects contribute to randomness between training runs, but fixing them does not decrease churn; rather, they appear to have a regularizing affect on our hardware platform.
Finally, there is churn resulting from an unavoidable source during training, the non-determinism in computing platforms used for training machine learning models, e.g., GPU/TPU (Morin & Willetts, 2020; PyTorch, 2019; Nvidia, 2020). The experiments in Table 1 were run on TPU. Even when all other aspects of training are held constant (rightmost column), model weights diverge within 100 steps (across runs) and the final churn is significant. We verified that this is the sole source of non-determinism: models trained to 10,000 steps on CPUs under these settings had identical weights.
These experiments underscore the importance of developing and incorporating churn reduction strategies in training. Even with extreme measures to eliminate all sources of randomness, we continue to observe churn due to unavoidable hardware non-determinism.

heading: 2.4 REDUCING CHURN: RELATION TO PREDICTION PROBABILITIES
text: We now focus on the main objective of this paper – churn reduction in NNs. Many factors that have been shown to cause churn in § 2.3 are crucial for the good performance of NNs and thus cannot be controlled or eliminated without causing performance degradation. Moreover, controlling certain factors such as hardware non-determinism is extremely challenging, especially in the large scale distributed computing setup.
Towards reducing churn, we first develop an understanding of the relationship between the distribution of prediction probabilities of a model and churn. Intuitively, encouraging either larger prediction confidence or smaller variance across multiple training runs should result in reduced churn.
To formalize this intuition, let us consider the prediction confidence realized by the classification model fp¨;wq on the instance x P X:
γx;w “ fpx;wqŷx;w ´ arg maxj‰ŷx;w fpx;wqj , (5)
where ŷx;w is the model prediction in (1). Note that γx;w denotes the difference between the probabilities of the most likely and the second most likely classes under the distribution fpx;wq, and is not the same as the standard multi-class margin (Koltchinskii et al., 2001): it captures how confident the model fp¨;wq is about its prediction ŷx;w, without taking the true label into account. The following result relates the prediction confidence to churn. See Appendix B for the proof. Lemma 2. Let γx;w1 and γx;w2 be the prediction confidences realized on x P X by the classification models fp¨;w1q and fp¨;w2q, respectively. Then,
Churnpw1, w2q “ PXtŶX;w1 ‰ ŶX;w2u ď PX “ DL1pfpX;w1q, fpX;w2qq ą mintγX;w1 , γX;w2u ‰ .
Here,DL1pfpx;w1q, fpx;w2qq “ ř jPY |fpx;w1qj´fpx;w2qj |measures theL1 distance. Lemma 2 establishes that, for a given instance x, the churn between two models fp¨;w1q and fp¨;w2q becomes less likely as their confidences γx;w1 and γx;w2 increase. Similarly, the churn becomes smaller when the difference between their prediction probabilities, DL1pfpx;w1q, fpx;w2qq, decreases.

heading: 3 REGULARIZED CO-DISTILLATION FOR CHURN REDUCTION
text: In this section we present our approach for churn reduction. Motivated by Lemma 2, we consider an approach with two complementary techniques for churn reduction: 1) We first propose entropy based regularizers that encourage solutions w PW with large prediction confidence γx;w for each instance x. 2) We next employ co-distillation approach with novel design choices that simultaneously trains two models while minimizing the distance between their prediction probabilities.
Note that the entropy regularizers themselves cannot actively enforce alignment between the prediction probabilities across multiple runs as the resulting objective does not couple multiple models together.
On the other hand, the co-distillation in itself does not affect the prediction confidence of the underlying models (as verified in Figure 2). Thus, our combined approach promotes both large model prediction confidence and small variance in prediction probabilities across multiple runs.

heading: 3.1 MINIMUM ENTROPY REGULARIZERS
text: Aiming to increase the prediction confidences of the trained model tγx;wuxPX , we propose novel training objectives that employ one of two possible regularizers based on: (1) entropy of the model prediction probabilities; and (2) negative symmetric KL divergence between the model prediction probabilities and the uniform distribution. Both regularizers encourage the prediction probabilities to be concentrated on a small number of classes, and increase the associated prediction confidence.
Entropy regularizer. Recall that the standard training minimizes the risk Lpwq. Instead, for α P r0, 1s and S “ tpxi, yiquiPrns, we propose to minimize the following regularized objective to increase the prediction confidence of the model3.
Lentropypw;Sq “ p1´ αq ¨ Lpw;Sq ` α ¨ 1
n
ÿ
iPrns H ` fpxi, wq ˘ , (6)
where H ` fpx;wq ˘ “ ´ ř jPrKs fpx;wqj log fpx;wqj denotes the entropy of predictions fpx;wq.
Symmetric KL divergence regularizer. Instead of encouraging the low entropy for the prediction probabilities, we can alternatively maximize the distance of the prediction probability mass function from the uniform distribution as a regularizer to enhance the prediction confidence. In particular, we utilize the symmetric KL divergence as the distance measure. Let Unif P ∆K be the uniform distribution. Thus, given n samples S “ tpxi, yiquiPrns, we propose to minimize
LSKLpw;Sq , p1´ αq ¨ Lpw;Sq ´ α ¨ 1
n
ÿ
iPrns SKL
` fpxi;wq,Unif ˘ , (7)
where SKL ` fpx;wq,Unif ˘ “ KL ` fpx;wq||Unif ˘ `KL ` Unif||fpx;wq ˘ .
As discussed in the beginning of the section, the intuition behind utilizing these regularizers is to encourage spiky prediction probability mass functions, which lead to higher prediction confidence. The following result supports this intuition in the binary classification setting. See Appendix B for the proof. As for multi-class classification, we empirically verify that the proposed regularizers indeed lead to increased prediction confidences (cf. Figure 2).
Theorem 3. Let fp¨;wq and fp¨;w1q be two binary classification models. For a given x P X , if we have H pfpx;wqq ď H pfpx;w1qq or SKLpfpx;wqq ě SKLpfpx;w1qq, then γx;w ě γx;w1 .
Note that the effect of these regularizers is different from increasing the temperature in softmax while computing fpx;wq. Similar to max entropy regularizers (Pereyra et al., 2017), they are independent of the label, a crucial difference that allows them to reduce churn even among misclassified examples.

heading: 3.2 CO-DISTILLATION
text: As the second measure for churn reduction, we now focus on designing training objectives that minimize the variance among the prediction probabilities for the same instance across multiple runs of the learning algorithm. Towards this, we consider novel variants of co-distillation approach (Anil et al., 2018). In particular, we employ co-distillation by using symmetric KL divergence (Co-distillSKL), with a linear warm-up, to penalize the prediction distance between the models instead of the popular step-wise cross entropy loss (Co-distillCE) used in Anil et al. (2018) (see § C). We first motivate the proposed objective for reducing churn using Lemma 2.
Recall from Lemma 2 that, for a given instance x, churn across two models fp¨;w1q and fp¨;w2q decreases as the distance between their prediction probabilities DL1pfpx;w1q, fpx;w2qq becomes smaller. Motivated by this, given training samples S “ tpxi, yiquiPrns, one can simultaneously train two models corresponding to w1, w2 PW by minimizing the following objective and keeping either
3We restrict ourselves to the convex combination of the original risk and the regularizer terms as this ensures that the scale of the proposed objective is the same as that of the original risk. This allows us to experiment with the same learning rate and other hyperparameters for both.
of the two models as the final solution.
LCo-distillL1pw1, w2;Sq , Lpw1;Sq ` Lpw2;Sq ` β
n
ÿ
iPrns DL1pfpxi;w1q, fpxi;w2qq (8)
From Pinsker’s inequality, DL1pfpxi;w1q, fpxi;w2qq ď a 2 ¨KLpfpxi;w1q||fpxi;w2qq. Thus, one can alternatively utilize the following objective.
LCo-distillSKLpw1, w2;Sq , Lpw1;Sq ` Lpw2;Sq ` β
n
ÿ
iPrns SKL
` fpxi;w1q, fpxi;w2q ˘ , (9)
where SKL ` fpx;w1q, fpx;w2q ˘ “ KL ` fpx;w1q||fpx;w2q ˘ ` KL ` fpx;w2q||fpx;w1q ˘
denotes the symmetric KL-divergence between the prediction probabilities of the two models. In what follows, we work with the objective in (9) as we observed this to be more effective in our experiments, leading to both smaller churn and higher model accuracy.
In addition to the co-distillation objective, we introduce two other changes to the training procedure: joint updates and linear rampup of the co-distillation loss. We discuss these differences in § C.
Regularized co-distillation. In this paper, we also explore regularized co-distillation, where we utilize the minimum entropy regularizes (cf. § 3.1) in our co-distillation framework. We note that the best results for the combined approach are achieved when we use a linear warmup for the regularizer coefficient as well. Note that combining the Co-distillSKL objective with an entropy regularizer is not the same as using the cross entropy loss, due to the use the different weights for each method, and rampup of the regularizer coefficients. This distinction is important in reducing churn ( Table 2).

heading: 4 EXPERIMENTS
text: We conduct experiments on 5 different datasets, CIFAR-10, CIFAR-100, ImageNet, SVHN and iNaturalist 2018. We use LeNet5 for experiments on SVHN, ResNet-56 for experiments on CIFAR10 and CIFAR-100, and ResNet-v2-50 for experiments on ImageNet and iNaturalist. We use the
same hyperparameters for all the experiments on a dataset. We use the Cross entropy loss and the Momentum optimizer. For complete details we refer to the Appendix A.
Topk regularizer. For problems with a large number of outputs, e.g. ImageNet, it is not required to penalize all the predictions to reduce churn. Recall that prediction confidence is only a function of the top two prediction probabilities. Hence, on ImageNet, we consider the topk variants of the proposed regularizers, and penalize the entropy/SKL only on the top k predictions, with k “ 10. Accuracy and churn. In Table 2, we present the accuracy and churn of models trained with the minimum entropy regularizers, co-distillation and their combination. For each dataset we report the values for the best α and β. We notice that our proposed methods indeed reduce the churn and Schurn. While both minimum entropy regularization and co-distillation are consistently effective in reducing Schurn, their effect on churn varies, potentially due to the discontinuous nature of the churn. Also note that the proposed methods reduce churn both among the correctly and incorrectly classified examples. Our co-distillation proposal, Co-distillSKL consistently showed a significant reduction in churn, and has better performance than Co-distillCE (Anil et al., 2018) that is based on the cross-entropy loss, showing the importance of choosing the right objective for reducing churn. We present additional comparison in Figure 2, showing churn and Top-1 error on ImageNet, computed for
different values of β (cf. (9)). Finally, we achieve an even further reduction in churn by the combined approach of entropy regularized co-distillation. Interestingly, the considered methods also improve the accuracy, showing their utility beyond churn reduction.
Ensembling. We also compare with the ensembling-distillation approach (Hinton et al., 2015) in Table 2, where we use a 2 teacher ensemble for distillation. We show that the proposed methods consistently outperform ensembling-distillation approach, despite having a lower training cost.
Ablation. We next report results of our ablation study on the entropy (top10) regularizer coefficient α, and the Co-distillSKL coefficient β in Figure 1. While both methods improve accuracy and churn, the prediction entropy shows different behavior between these methods. While entropy regularizers improve churn by reducing the entropy, co-distillation reduces churn by reducing the prediction distance between two models, resulting in an increase in entropy. This complementary nature explains the advantage in combining these two methods (cf. Table 2).
We next present ablation studies on weight decay regularization in Table 3, showing that the proposed approaches improve churn on models trained both with and without weight decay.
Fixed initialization. Earlier in Table 1, we showed that fixing the initialization of five runs of the baseline model lowers churn from 5.81˘ 0.08 to 5.56˘ 0.12 for CIFAR-10 on ResNet-56. However, our proposed Co-distillSKL model is more effective, reducing churn to 4.29 ˘ 0.14 (see Table 2). Using fixed initialization on this model does not significantly affect churn (4.24˘ 0.07).

heading: 5 DISCUSSION
text: Connection to label smoothing and calibration. Max entropy regularizers, such as label smoothing, are often used to reduce prediction confidence (Pereyra et al., 2017; Müller et al., 2019) and provide better prediction calibration. The minimum entropy regularizers studied in this paper have the opposite effect and increase the prediction confidence, whereas co-distillation reduces the prediction confidences. To study this more concretely we measure the effect of churn reduction methods on the calibration.
We compute Expected Calibration Error (ECE) (Guo et al., 2017) for methods considered in this paper and report the results in Table 4. We notice that the minimum entropy regularizers, predictably, increase the calibration error, whereas the propose co-distillation approach (Co-distillSKL) significantly reduces the calibration error. We notice that our joint approach is competitive with the ensemble distillation approach on CIFAR datasets but incurs higher error on ImageNet. Developing approaches that jointly optimize for churn and calibration is an interesting direction of future work.

heading: A EXPERIMENTAL SETUP
text: Architecture: For our experiments we use the standard image classification datasets CIFAR and ImageNet. For our experiments with CIFAR, we use ResNet-32 and ResNet-56 architectures, and for ImageNet we use ResNet-v2-50. These architectures have the following configuration in terms of pnlayer, nfilter, strideq, for each ResNet block:
• ResNet-32- [(5, 16, 1), (5, 32, 2), (5, 64, 2)] • ResNet-56- [(9, 16, 1), (9, 32, 2), (9, 64, 2)] • ResNet-v2-50- [(3, 64, 1), (4, 128, 2), (6, 256, 2), (3, 512, 2)],
where ‘stride’ refers to the stride of the first convolution filter within each block. For ResNet-v2-50, the final layer of each block has 4 ˚ nfilter filters. We use L2 weight decay of strength 1e-4 for all experiments.
Learning rate and Batch size: For our experiments with CIFAR, we use SGD optimizer with a 0.9 Nesterov momentum. We use a linear learning rate warmup for first 15 epochs, with a peak learning rate of 1.0. We use a stepwise decay schedule, that decays learning rate by a factor of 10, at epoch numbers 200, 300 and 400. We train the models for a total of 450 epochs. We use a batch size of 1024.
For the ImageNet experiments, we use SGD optimizer with a 0.9 momentum. We use a linear learning rate warmup for first 5 epochs, with a peak learning rate of 0.8. We again use a stepwise decay schedule, that decays learning rate by a factor of 10, at epoch numbers 30, 60 and 80. We train the models for a total of 90 epochs. We use a batch size of 1024.
Data augmentation: We use data augmentation with random cropping and flipping.
Parameter ranges: For our experiments with the min entropy regularizers, we use α P r0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.5s, and report the α corresponding to best churn in Table 2. For our experiments with the co-distillation approach, we use β P r0.01, 0.02, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08s, and report the β corresponding to best churn in Table 2.
For our experiments on entropy regularized co-distillation, we use a linear warmup of the regularizer coefficient. We use the same range for α as described above, and use only the best β value from the earlier experiments.
Finally, we run our experiments using TPUv3. Experiments on CIFAR datasets finish in an hour, experiments on ImageNet take around 6-8 hours.
Hyper parameters for Table 2: Finally we list in Table 5 the best hyperparameters used to obtain the results in Table 2. For reference we also include the accuracy and the churn again for all methods.

heading: B PROOFS
text: Proof of Lemma 1. Recall from Definition 1 that
Churnpw1, w2q “ PX “ ŶX,w1 ‰ ŶX,w2 ‰ “ PX,Y “ ŶX,w1 ‰ ŶX,w2 ‰
“ PX,Y “ tŶX,w1 “ Y, ŶX,w2 ‰ Yu Y tŶX,w1 ‰ Y, ŶX,w1 ‰ ŶX,w2u ‰
piq ď PX,Y “ ŶX,w1 “ Y, ŶX,w2 ‰ Y ‰ ` P “ ŶX,w1 ‰ Y, ŶX,w1 ‰ ŶX,w2 ‰
piiq ď P “
ŶX,w2 ‰ Y ‰ ` P “ ŶX,w1 ‰ Y ‰
“ PErr,w2 ` PErr,w1 , (10) where piq and piiq follow from the union bound and the fact that PrAs ď PrBs whenever A Ď B, respectively.
Proof of Lemma 2. Note that, for any j ‰ ŷx,w1 , fpx;w2qŷx,w1 ´ fpx;w2qj
“ fpx;w2qŷx,w1 ´ fpx;w1qŷx,w1 ` fpx;w1qŷx,w1 ´ fpx;w1qj looooooooooooooomooooooooooooooon
ěγx,w1
`fpx;w1qj ´ fpx;w2qj
ě γx,w1 ´ ÿ
jPY |fpx;w1qj ´ fpx;w2qj |
“ γx,w1 ´DL1 ` fpx;w1q, fpx;w2q ˘
. (11) Similarly, for any j ‰ ŷx,w2 , we can establish that
fpx;w1qŷx,w2 ´ fpx;w1qj ě γx,w2 ´DL1 ` fpx;w1q, fpx;w2q ˘ . (12) Note that experiencing churn between the two models on x is equivalent to tŷx,w1 ‰ ŷx,w2u Ď
Dj ‰ ŷx,w1 : fpx;w2qŷx,w1 ă fpx;w2qj (
ď
Dj ‰ ŷx,w2 : fpx;w1qŷx,w2 ă fpx;w1qj (
piq Ď DL1 ` fpx;w1q, fpx;w2q ˘ ą γx,w1 ( ď DL1 ` fpx;w1q, fpx;w2q ˘ ą γx,w2 ( , (13)
where piq follows from (11) and (12). Now, (13) implies that PXtŶX,w1 ‰ ŶX,w2u ď PX “ DL1pfpX;w1q, fpX;w2qq ą mintγX,w1 , γX,w2u ‰ . (14)
Proof of Theorem 3. Let the prediction for a given x be p “ fpx;wq. W.L.O.G. let p ě 1´ p. The prediction confidence is then
γx,w “ p´ p1´ pq “ 2p´ 1. (15)
Entropy case. Now we can write entropy in terms of the prediction confidence (cf. (15))
H pfpx;wqq “ ´p logppq ´ p1´ pq logp1´ pq
“ ´p1` γx,wq 2 ˆ log ˆ 1` γx,w 2 ˙ ´ p1´ γx,wq 2 ˆ log ˆ 1´ γx,w 2 ˙
, gpγx,wq. (16)
Now the gradient of gp¨q is ∇γx,wgpγx,wq “ 12 log ´ 1´γx,w 1`γx,w ¯
, which is less than 0 for γx,w P r0, 1s. Hence, the function gpγx,wq is a decreasing function for inputs in range r0, 1s. Hence, if gpγx,w1q ě gpγx,wq implies γx,w1 ď γx,w.
Symmetric-KL divergence case. Recall that
SKLpfpx;wq,Unifq “ KL ` fpx;wq||Unif ˘ `KL ` Unif||fpx;wq ˘
“ ÿ
jPrKs
` fpx;wqj ´ 1{K ˘ ¨ log ` K ¨ fpx;wqj ˘
“ ÿ
j
fpx;wqj ¨ log pfpx;wqjq ´ 1
K log pfpx;wqjq .
For binary classification this reduces to,
SKLpfpx;wq,Unifq “ ´1 2 logppp1´ pqq ´Hppq.
By using (15) and (16), we can rewrite this function in terms of γx,w as:
SKLpfpx;wq,Unifq “ ´1 2 logppp1´ pqq ´Hppq
“ ´1 2 ¨ log
ˆ
1` γx,w 2 ˆ 1´ γx,w 2
˙
´ gpγx,wq “ ´ 1
2 ¨ log
˜
1´ γ2x,w 4
¸
´ gpγx,wq.
Now notice that both the above terms are an increasing function of γx,w, as log is an increasing function, and gpγx,wq is a decreasing function. Hence SKLpfpx;wq,Unifq ě SKLpfpx;w1q,Unifq implies γx;w ě γx;w1 .

heading: C DESIGN CHOICES IN THE PROPOSED VARIANT OF CO-DISTILLATION
text: Here, we discuss two important design choices that are crucial for successful utilization of codistillation framework for churn reduction while also improving model accuracy.
Joint vs. independent updates. Anil et al. (2018) consider co-distillation with independent updates for the participating models. In particular, with two participating models, this corresponds to independently solving the following two sub-problems during the t-th step.
wt1 “ arg minw1PW Lpw1;Sq ` β ¨ 1
n
ÿ
iPrns KL
` fpxi;wt´T2 q||fpxi;w1q ˘
(17a)
wt2 “ arg minw2PW Lpw2;Sq ` β ¨ 1
n
ÿ
iPrns KL
` fpxi;wt´T1 q||fpxi;w2q ˘ , (17b)
where wt´T1 and w t´T 2 corresponds to earlier checkpoints of two models being used to compute
the model disagreement loss component at the other model. Note that since wt´T2 and w t´T 1 are not being optimized in (17a) and (17b), respectively. Thus, for each model, these objectives are equivalent to regularizing its empirical loss via a cross-entropy terms that aims to align its prediction probabilities with that of the other model. In particular, the optimization steps in (17) are equivalent to
wt1 “ arg minw1PW Lpw1;Sq ` β ¨ 1
n
ÿ
iPrns H ` fpxi;wt´T2 q, fpxi;w1q ˘
(18a)
wt2 “ arg minw2PW Lpw2;Sq ` β ¨ 1
n
ÿ
iPrns H ` fpxi;wt´T1 q, fpxi;w2q ˘ , (18b)
where H ` fpx;w1q, fpx;w2q ˘ “ ´ ř j fpx;w1qj log fpx;w2qj denotes the cross-entropy between the probability mass functions fpx;w1q and fpx;w2q. In our experiments, we verify that this independent updates approach leads to worse results as compared to jointly training the two models using the objective in (9).
Deferred model disagreement loss component in co-distillation. In general, implementing co-distillation approach by employing the model disagreement based loss component (e.g.,SKL ` fpx;w1q, fpx;w2q ˘
in our proposal) from the beginning leads to sub-optimal test accuracy as the models are very poor classifiers at the initialization. As a remedy, Anil et al. (2018) proposed a step-wise ramp up of such loss component, i.e., a time-dependent βptq such that βptq ą 0 iff t ą t0, with t0 burn-in steps. Besides step-wise ramp up, we experimented with linear ramp-up, i.e., βt “ mintc ¨ t, βu and observed that linear ramp up leads to better churn reduction. We believe that, for churn reduction, it’s important to ensure that the prediction probabilities of the two models start aligning before the model diverges too far during the training process.

heading: D COMBINED APPROACH: CO-DISTILLATION WITH ENTROPY REGULARIZER
text: In this paper, we have considered two different approaches to reduce churn, involving minimum entropy regularizers and co-distillation framework, respectively. This raises an important question if these two approaches are redundant. As shown in Figures 2, 1, and 3, this is not the case. In fact, these two approaches are complementary in nature and combining them together leads to better results in term of both churn and accuracy.
Note that the minimum entropy regularizers themselves cannot actively enforce alignment between the prediction probabilities across multiple runs as the resulting objective does not couple multiple models together. On the other hand, as verified in Figure 2, the co-distillation framework in itself does not affect the prediction confidence of the underlying models. Thus, both these approaches can be combined together to obtain the following training objective which promotes both large model prediction confidence and small variance in prediction probabilities across multiple runs.
LcombinedCo-distillSKLpw1, w2;Sq “ LCo-distillSKLpw1, w2;Sqpw1, w2;Sq ` α ¨ ERegpw1, w2;Sq, (19) where
ERegpw1, w2;Sq “ 1
n
ÿ
iPrns H ` fpxi, w1q ˘ `H ` fpxi, w2q ˘
or
ERegpw1, w2;Sq “ ´ 1
n
ÿ
iPrns
´
SKL ` fpxi;w1q,Unif ˘ ` SKL ` fpxi;w2q,Unif ˘
¯
.

heading: E ADDITIONAL EXPERIMENTAL RESULTS
text: In this section we present additional experimental results.
Ablation on CIFAR-100. Next we present our ablation studies, similar to results in Figure 1, for CIFAR-100. In Figure 3, we plot the mean prediction entropy, accuracy and churn, for the proposed SKL regularizer (7) and the co-distillation approach (9), for varying α and β, respectively. Similar to Figure 1, the plots show the effectiveness of the proposed approaches in reducing churn, while improving accuracy.
E.1 ON HARDWARE NONDETERMINISM: GPU BASELINE ABLATION RESULTS
We now repeat the experiments in Sec. 2.3 on GPUs to investigate the unavoidable nondeterminism introduced by different hardware platforms.
We report results in Table 6. Compared to Table 1, there are two main differences. First, these experiments are run on GPU with smaller ResNet-32 models. Second, these results always use data
augmentation, but control for the randomness differently by ensuring that all models within each run perform the same augmentations during training; we call these “identical input data” ablations because they control for both minibatch ordering and data augmentation. Again, the the first column gives the baseline churn with no factor held constant across runs, and the last column has every possible source of churn except hardware nondeterminism held constant across runs.
Model initialization is a significant source of churn. Simply initializing weights from identical seeds (columns 1-2 in Table 1) can significantly decrease churn, no matter what other aspects of training are held constant. Holding input data constant across runs does reduce randomness and further reduces churn, but is less significant compared to constant initialization.
The rightmost column eliminates all possible sources of churn except for unavoidable nondeterminism in the computing platform used for training machine learning models, e.g., GPU/TPU (Morin & Willetts, 2020; PyTorch, 2019; Nvidia, 2020). The experiments in Table 6 were run on GPU, and even when all other aspects of training are held constant (rightmost column), model weights diverge within 100 steps (across runs) and the final churn is significant. We verified that this is the sole source of non-determinism: models trained to 10,000 steps on CPUs under these settings had identical weights. Comparing these results to the rightmost column of Table 1, it seems TPU platforms currently introduce a higher baseline amount of churn than GPUs. However, as shown in this work, our Co-distillSKL method can alleviate this somewhat.
E.2 2 ENSEMBLE
In Table 7 we provide the results for 2-ensemble models and compare with distillation and codistillation approaches. We notice that the proposed approach with entropy regularizer and codistillation achieve similar or better churn, with lower inference costs. However the the ensemble models achieve better accuracy on certain datasets, and could be an alternative in settings where higher inference costs are acceptable.

heading: F LOSS LANDSCAPE: ENTROPY REGULARIZATION VS. TEMPERATURE SCALING
text: Figure 4 visualises the entropy regularised loss (6) in binary case. We do so both in terms of the underlying loss operating on probabilities (i.e., log-loss), and the loss operating on logits with an implicit transformation via the sigmoid (i.e., the logistic loss). Here, the regularised log-loss is ` : r0, 1s Ñ R`, where `ppq “ p1´ αq ¨ ´ log p` α ¨Hbinppq, for binary entropy Hbin. Similarly, the logistic loss is ` : RÑ R`, where `pfq “ p1´ αq ¨ ´ log σpfq ` α ¨Hbinpσpfqq, for sigmoid σp¨q. The effect of increasing α is to dampen the loss for high-confidence predictions – e.g., for the logistic loss, either very strongly positive or negative predictions incur a low loss. This encourages the model to make confident predictions.
Figure 5, by contrast, illustrates the effect of changing the temperature τ in the softmax. Temperature scaling is a common trick used to decrease classifier confidence. This also dampens the loss for high-confidence predictions. However, with strongly negative predictions, one still incurs a high loss compared to strongly positive predictions. This is in contrast to the more aggressive dampening of the loss as achieved by the entropy regularised loss.

Review summaries:
The paper proposes methods to address churn in deep neural networks for classification, defined as the extent of disagreements in predictions of two models trained on the same data with the same algorithm. In addition to an existing measure of churn that is based on exact match of predicted classes, the paper introduces a soft measure of churn that measures disagreement by comparing the two models' class probability distributions. The paper proposes three regularization terms that can be added to the primary loss function used during training to reduce churn: two single-model regularization terms (based on cross entropy and KL divergence respectively) that encourage the model to output a more uneven probability distribution for an example, and a divergence-based term that is used by training two models simultaneously and that imposes a KL-divergence-based penalty that encourages the two models to output probability distributions that are as similar as possible. Experiments with ResNet architectures on CIFAR-10/100 and ImageNet indicate that the proposed approaches and their combination indeed reduce churn and do so to a larger extent than the two-model divergence-based approach applied in conjunction with cross-entropy that was proposed in work by Anil et al. in 2018 (although the improvement seems quite minor on ImageNet).

The paper investigates two methods to reduce churn in neural network classification prediction. Churn is when two networks trained on the same data produce outputs that disagree, due to randomness in the training process. The authors identify several sources of randomness, from underlying hardware differences to parameter initialization and more. The authors propose two ways to mitigate churn. One is to use entropy minimization to favor more confident predictions. The second is to use co-distillation, a form of online ensemble learning. The authors show that both together do a good job of reducing churn on three data sets.