Anti-Distillation: Improving Reproducibility of Deep Networks
Authors: Gil I Shamir
Lorenzo Coviello
Venue:  (2021)
Keywords: Deep networks
ensembles
reproducibility

Abstract: Deep networks have been revolutionary in improving performance of machine learning and artificial intelligence systems.
Their high prediction accuracy, however, comes at a price of model irreproducibility in very high levels that do not occur with classical linear models. Two models, even if they are supposedly identical, with identical architecture and identical trained parameter sets, and that are trained on the same set of training examples, while possibly providing identical average prediction accuracies, may predict very differently on individual, previously unseen, examples. Prediction differences may be as large as the order of magnitude of the predictions themselves. Ensembles have been shown to somewhat mitigate this behavior, but without an extra push, may not be utilizing their full potential. In this work, a novel approach, Anti-Distillation, is proposed to address irreproducibility in deep networks, where ensemble models are used to generate predictions. Anti-Distillation forces ensemble components away from one another by techniques like de-correlating their outputs over mini-batches of examples, forcing them to become even more different and more diverse.  Doing so enhances the benefit of ensembles, making the final predictions more reproducible. Empirical results demonstrate substantial prediction difference reductions achieved by Anti-Distillation on benchmark and real datasets.

Under review as a conference paper at ICLR 2021
ANTI-DISTILLATION IMPROVING REPRODUCIBILITY OF DEEP NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Deep networks have been revolutionary in improving performance of machine learning and artificial intelligence systems. Their high prediction accuracy, how- ever, comes at price of model irreproducibility in very high levels that do not occur with classical linear models. Two models, even if they are supposedly iden- tical, with identical architecture and identical trained parameter sets, and that are trained on the same set of training examples, while possibly providing identical average prediction accuracies, may predict very differently on individual, previ- ously unseen, examples. Prediction differences may be large as the order of magnitude of the predictions themselves. Ensembles have been shown to some- what mitigate this behavior, but without extra push, may not be utilizing their full potential. In this work, a novel approach, Anti- Anti-Distillation, is proposed to address irreproducibility in deep networks, where ensemble models are used generate predictions. Anti- aati-Distillation forces ensemble components away from one another by techniques like de-correlating their outputs over mini-batcher of examples, forcing them to become even more different and more diverse. Doing so enhances the benefit of ensembles, making the final predictions more repro- ducible. Empirical results demonstrate substantial prediction difference reduc- tions achieved by anti-Ditiillation on benchmark and real datasets.
INTRODUCTION
In the last decade. deep networks provided revolutionary breakthroughs in machine learning. achiev- ing capabilities that were not even imagined ten years ago, and penetrating every domain of our lives. They have been shown be substantially superior classical techniques that optimized linear mod- els on convex objectives. With this success, however, comes price; the price of irreproducibility at high levels that were unseen before with classical models (Dusenberry et al., 2020). Training even the same exact model with identical parameters and architecture the same set of training exam- ples can produce very different models trained more than once. Two such models can have equal average prediction accuracy validation data, but predict very differently individual examples. The problem can be as extreme as having a Prediction Difference (PD) that is the same order of magnitude the predictions themselves. Perhaps some applications but imagine applications as medical ones, where for the same symptoms, one model would predict one disease, and the other model would predict another. While in some way, this mimics real life, where individuals make conclusions based on what they learned and what they know, or depen- dent the order in which they learn different topics (see, e.g., Achille et al. (2017); Bengio et al. (2009)), this is definitely not a desired behavior. Overall, perhaps, predictions are better, but for the individual cases in which predictions are different, the consequences can be irreversible.
tolerate such differences,
Deep models are usually trained on highly parallelized distributed systems. They normally are initialized randomly, and are expected find a nonlinear solution that fits the data best, minimizing non-convex loss objective. Applying determinism (see, e.g., Nagarajan et al. (201 8)) to the order in which data is seen and/or updated may not be an option, especially extremely large scale systems. Due to such systems, even if the models are initialized identically (to some identical pseudorandom set of initialization values), the trained model sees the training examples in some random order and the updates applied also with some randomness. Due to the non-convex nature, different training instances of the same model on the same dataset, may still find and converge to different optima, which may be all equal in average accuracy, but very different for individual examples.

---

Under review as a conference paper at ICLR 2021
This problem becomes even more critical in re-enforcement and online (or mini-batch) learning, where the model needs to make predictions and decisions while it continues to train on new addi- tional examples. The decisions the model makes also dictate what new data is seen. If two models diverge from one another on the same data, they can then make different future decisions, affecting what additional data is seen by the model. This can enhance the divergence of two supposedly iden- tical models even more. Online Tlick-Through-Ras Rate (CTR) prediction (see, e.g., McMahan et al. (2013)), where ads are shown based on the current state of the model and then the model updates based on user reactions to the shown ads, is an example where irreproducibility can lead to two models which diverge over time very largely from one another.
Using ensembles of models (see, e.g., Dietterich (2000); Koren (2009)) has become very popular technique in machine learning, and also in deep networks. Ensembles can reduce prediction un- certainty as shown in Lakshminarayanan et al. (2017). Averaging the predictions of multiple deep networks, referred to as ensemble components, in an ensemble turns out to also help reduce predic- tion differences between two (or more) such ensembles. Because of irreproducibility, and especially each component is initialized differently, each of the components diverges to focus on a differ- ent slice of the solution space. Thus the average prediction, which has reduced variance, is more reproducible. One point to note, however, is that this approach can trade performance accuracy to improve reprodueibility. If the number of operations (or the number of learned parameters) is con- straint, then, the components of the ensemble must have narrower layers than a single deep network applied to the same task if comparison done where complexity and resources are kept equal to both systems. While ensembling on the narrower components has better accuracy than that of a sin- gle narrow component, the ensemble as whole may have inferior accuracy than comparable single network with the same complexity (or number of parameters). This will happen in very large scale systems where the single components of the model are not significantly over weer-pramettrized to the training data. (In smaller scales, it is possible that narrower models are still expressive enough for the specific problem, and adding more parameters is no longer helpful in improving model accuracy.)
Distillation (see Hinton et al. (2015), and also Lan et al. (2018)) is technique that is gaining popularity in deep networks to transfer knowledge of a complex model or complex ensemble of models (Mosca & Magoulas, 2018) to simple model in way that would allow the simple model to exhibit similar or close performance to that of the complex model. This is particularly important to models deployed student model uses the prediction of the complex teacher model as a label, to try and become more similar to the teacher, which is stronger model with better predictions. Various methods in Crowley et al. (2018); Gou et al. (2020); Kim et al. (2018); Muller et al. (2020); Mun et al. (2018); Tang et al. (2020) have been developed to extract the most from distillation.
small devices that do not have the capacity of larger systems. The simple
Contribution: Unlike distillation methods, in this paper, we apply a technique of opposite nature, iati-Distillation (AD). Leveraging the benefit of ensembles, we try to make the components as different as possible. so together they capture a larger subset of the solution space as an ensemble. This done by adding a (regularization) loss that forces different components to diverge from one another. This can increase the diversification of the ensemble. Various regularization losses can achieve the effect of diversifying the components. We focus on correlation and covariance losses. A de-correlation loss can be obtained by minimizing the square of the Frobenius norm of the off-diagonal terms of the correlation matrix of the predictions of the ensemble components. De-correlation loss can be applied over correlations estimated from the predictions of a mini-batch of training examples. De-correlation has been applied internally on neurons of hidden layers in neural networks for decades (see, e.g., Shamir et al. (1993)). It was used in attempts to simplify the representations of the networks, by pruning neurons whose contributions are correlated with others, or for reducing overfitting (Cogswell al., 2015). Unlike these works, here we apply de-correlation on full model predictions of the components of the ensemble for the mere purpose of diversifying the predictions of these components, so that the overall averaged prediction of the ensemble captures a wider portion of the optimization parameter space, and by doing so, is more reproducible.
Related Work: Over- preerparameterizationn of deep networks has been studied in many references (see, e.g., Denil et al. (2013); Han et al. (2015) and many others). Deep networks can thus find multiple explanations to the same datasets (even when examples are randomized (Zhang et al., 2016)). As described above, such different explanations may yield non-identical individual predictions, despite even consistent identical average performance over validation sets.
2

---

Under review as a conference paper at ICLR 2021
Techniques, such ensembles and distillation are becoming very popular in deep networks, mainly in an attempt to deploy smaller models to which better knowledge is distilled. Specifically, for improving deployed models' reproducibility, co-distillation was proposed in Anil et al. (2018) (see also Zhang et al. (2018)). Co-distillation distills information in all directions between two (or more) models. Each model learns from the others, where they all try agree some solution. Then, only one model needs to be deployed, representing the solution that was agreed on. The deployed model is more reproducible relative to a model of equal capacity, but because models attempt to agree on solution which is between the optima they converged to, there is some degradation in prediction accuracy. This degradation can be compensated by making the individual models more complex. While only one of the individual models is deployed, all of them must train, leading, despite deployment resource savings, to increased use of training resources.
A different technique that can improve model reproducibilityy is transfer learning (see, e.g., Raffel et al. (2019) and references within). While the motivation for transfer learning is to leverage knowl- edge that was learned in one domain to another, one can also initialize parameters of new model to the values given by a converged model which has already been trained for the same task. On one hand, this would guarantee that the model is more likely to converge to the same solution, but on the other, it may not let the model explore other possible solutions. Unlike ensembles, that smoothen the objective, this approach anchors the model to solution the original highly non-convex objective. Transferring only a partial set of parametens may not he sufficient, as the model must still adjust its other parameters to the transferred values. Hence, this method may not be sufficient when we want to train models to express new information and parameters.
A different line of work was focused more on measuring various effects in training deep networks. For example, in Shallue et al. (2018), the effect of data parallelism in training of deep networks was studied. Such parallelism does affect irreproducibility. least indirectly. Subsequently to our work presented here, correlation between activation strengths and PD was studied in Chen et al. (2020).
The remainder of the paper is organized as follows. In Section 2, we describe the ensemble system on which we apply anti-distillation. Section 3 outlines different metries that can be used to measure prediction differences between deep models. Section 4 deseribes Anti-Distillation. Finally, in Section 5, experimental results are described.
Paper Outline:
2 SYSTEM DESCRIPTION
We consider a standard ensemble of deep networks as depicted in Fig. 1. Several (3 in the figure) deep networks (which can be shaped like towers or differently), each with its own inputs, that can be identical for all components or different, produces logit outputs. For binary labels, these are scalars, that are converted to probabilities via the Sigmoid function a = 1/(1 + exp(-2)). For multi-label problems, these can be vectors, converted to probabilities by Softmax. (Other scores and target values can also be used.) For inference, the outputs of the components are averaged either in probability (Output 1) or in logit (Output 2). Averaging can be uniform, or can be done as mixture of experts (Shazeer et al., 2017). with some gating mechanism. In training, gradients are normally not allowed to propagate from the ensemble output to each component, which trains on its own on the training objective loss, although some systems may allow such propagation. Supervised training is done with standard training objectives with labeled examples as cross-entropy loss. The training label loss can be the same or different for each of the ensemble components, but inference generates predictions for each of the components, which are averaged as described. The components may be identical in structure and training hyper-parameters (as in Fig. 1). or may have different structures and/or different training rates, and even training algorithm). The components may be trained on the same data examples, or on different subsets of the training datasets, as long as there a non-empty intersection training set, on which AD training loss can be applied. Inputs enter the bottom layers in Fig. 1 of each of the ensemble components. These can be any types of inputs, including embeddings of non-measurable features that are learned together with the rest of the link weights and biases that are learned by the network. The parameters of link weights, biases, and embeddings can be initialized randomly with some variance. Specifically, sparse embedding inputs can be randomly initialized with some variance. In order to generate diversity between the different components, the initialization of each component is different from the others. For Anti-distillation, we can control the variance level of
some label loss, such
parameerssss (such as learning
3

---

Under review as a conference paper at ICLR 2021
Output
Output
Sinnaic
Meant
Covamtance
Loss
Sgmodl
Sigools
Eouartanca
Loo5
Figure 1: Ensemble of components with Anti-Distillation Each component is trained with its own loss. The ensemble prediction is an average of the predictions of the components (Output 1: in probability, Output 2: in logits). An additional AD loss is applied either on the logit values (A) or directly on the predictions (B).
the initialization of the input parameters (as well as other parameters). In addition to the training loss, an additional Anti-Distillation loss is added the output (either in logits A in Fig. or in probability - B). This loss attempts to push the components away from one another via de- correlation another mechanism as will be described Section 4. The AD loss can applied on mini-batches of training examples, or continuously over training. While applied in training. AD loss does not change the inference, and has no effect on how trained model is deployed for inference (in, e.g., data center or a device).
PREDICTION DIFFERENCE
There are multiple approaches to measure Prediction Difference (PD) between trained models which perform inference over some validation dataset. PD can function of the model, its architecture, its parameters, the training algorithm used, the initialization, the random effects in the training system, and the dataset itself (both training and validation). We will measure PD on models that supposed to be identical. Such models are defined with the same architecture, the same learned parameters, learning algorithm, and hyper-parameters and are trained on same training dataset. assume that all the models in the set which measure PD are initialized initialization, the same sequence of pseudo-randos values used initialize all parameters in any model, including embedding inputs and link weights and biases. Thus for any pair in the set, any learned parameter is initialized to the same value for the first and second models in the pair. Randomness leading to irreproducibility thus the result of random effects on training and updates.
In some settings, identically. With pseudo-randed
PD be measured the prediction score for regression and for classification (logit in the binary classification case) or the actual prediction values on the predicted labels in classification. We focus on the classification setup, and choose measure PD on label predictions. although, this choice does not affect any of the conclusions. One can measure basic PD as some distance norm between a model's prediction and some baseline. Since in real systems, we do not have the "true" value of prediction even if label is observed, this value must be substituted by function of the predictions on the given example in the set of models which we compare. Also, by definition, the metric we want to measure is the difference in prediction values among a set of models, so for such a difference the "true" probability does not play a direct role. In systems which many multiple models can be trained, one can measure the PD for given example relative to its expected prediction over all models. Let M denote the number of different models, and N the number validation examples. Let P. m be the distribution over labels predicted by model for example n, where Pa,m (f) the
4

---

Under review as a conference paper at ICLR 2021
probability predicted for label l. Let Pn Σn. Pn.m/M be the averaged distribution over labels over all M models. Then, the PD, Ap. can be defined as the average 4p norm of the difference between the distribution predicted by any of the models in the set and the expected distribution
M Σ n=1 m=1
N
N
M
1/p
A
M P Pullp
Σ Pu Pa(())))
(1)
N
N
M n=1 m=1
Since we consider probabilities, p 1 is reasonable. One can also weight the differences by (E) so that unlikely labels do not influence the measure. However, this is already achieved to some extent by measuring prediction differences on prediction probabilities instead of on the raw (logit for the binary case) scores. An alternative approach would be to use the KL--diergence between the predicted label distributions, or, since KL divergence is asymmetric, use the Jensen-Shannon divergence (see, e.g. Gutman (1989)). However, for small deviations, using Pinsker's inequality (Cover& & Thomas, 2006), one can show that the KL divergence will be quadratic in the difference, and for small label predictions it will be inflated by the reciprocal of the lower label probabilities.
In practical very large scale problems, where training costs are high and resource availability is low, it may suffice to train small set M of models to have notion of the PD behavior of a given model on specific dataset. The definition in (1) generalizes to small values of M, un- like estimating, e.g., the prediction standard deviation among M models. Specifically, for bi- nary classification or regression problems, when one can effort training only M 2 models, = 1/N En, (1) - (1)), where 1 is the positive label of interest.
In some applications, a more relevant bit of information what fraction of the prediction differs between a model and baseline. For example, in CTR prediction, where true CTRs are usually probabilities much smaller than 0.5, the absolute differences may not be as informative. For L1, we can thus define relative PD, Ai, by normalizing the summand in (1) by (E). In binary problems, since there is only a single probability value to be estimated, that of a positive label, the summand can be normalized by (1) instead, leading to a slightly different metric denoted by Á Instead, however, we can compute relative PD only on the label that is actually observed, replacing the sum on in (1) by the absolute difference for the true label, normalizing by Pr. We denote this metric by Af. If one has no access to fiue. Åÿ can be used instead of Af. Finally, in classification, one can use Hamming PD; A specifying the average of the labels predicted differently between model pairs. In the sequel we will show results focusing on A1, Ag, Ái (binary case), A4, and A
4 ANTI-DISTILLATION
Consider an ensemble of internal components. Let i, be the signal on which AD loss is applied for the jth ensemble component on the 7th training example. As noted in Fig. 1, 2j,r can be either the logit/score value at the top of the jth deep component, or a prediction output of that component. We describe the approach for a scalar prediction value, but the methodology generalizes to vector of values, e.g., representing scores or probabilities for different label values in the multi-label case. In that case, loss applied in similar way for each component of zj.t, which is an output for a different label. While Fig. restricts the signal to the prediction of the component, it is not necessarily the only point in which can apply AD. The loss can be applied only on a subset network of the jth component. If the component consists of more complicated models combining linear components with deep ones, 2j,t can also be only the output of a deep network part or of some combination of the elements of the component model. We now restrict the discussion to single mini batch of training examples. Instead of using 2j,t, with some abuse of notation, let T; r = 1,2, ..... k, be the example index in a mini-batch of & examples, and let žj.r denote the prediction of the jth ensemble component on the rth example in the mini-batch.
Cogswell et al. (2015) used de-covariance loss on neurons of hidden layer to push them apart. Here, we use either de-correlation or de-covariance on the predictions Zj.r- Now, let C. be the estimated correlation matrix between the J/ prediction vectors in the mini-batch. The i, component of C. can be estimated by
k 25,r Zj.r-
1
Czsis
(2)
T
7=1
5

---

Under review as a conference paper at ICLR 2021
Similarly, we can estimate the expected prediction over the mini-batch of the ith ensemble component by 14i - 1/k Σx Ri, and estimate the elements of the batch covariance matrix Σg by
1 k 1: Fi. ma)(zj, -14)
Instead, let 14r _ 1/J Ei=1 Ri,r be the average ensemble prediction for example T in the mini- batch. Then, we can compute the residue correlation between the components as
Czn.- E (2 (+) (2,, -.-)-
(Normalizing by k and not k - 1 in (2)-(4) has no significant difference here, as batches can be large, and this term is absorbed in the learning rate parameters and the temperature of this loss relative to the main training loss, described below.) The AD loss is defined as
- [[ca1]: ||dial ||diag (C:)
LAD (C=) =
(5)
where || denotes the Frobenius norm of the matrix A, and (iag(A) represents the diagonal matrix consisting of the diagonal elements of A. Hence, the correlation loss objective attempts to minimize the sum of squares of off-diagonal elements of the correlation matrix, thereby attempting to minimize the correlations between prediction vectors of the components of the ensemble. Simi- larly, we can replace C₂ in (5) by the covariance matrix Σ,, or by the residue correlation matrix C.. Correlation loss may be advantageous in most common situations where we want to de-correlate the full prediction signal, including the bias, and specifically if ensemble components are completely disjoint. If examples in a mini-batch share a subset of identical common features, and we want emphasize de-correlations on the contributions of other features, which they do not share, ing covariance over the batch is reasonable. This may be the case sponsored advertising, where a mini-batch may include examples of impressions for some common query, where all examples share query features, but not ad features. In very large scale systems, it may be too costly to replicate all model parts in an ensemble, and ensemble components may share parts, such as embeddings, or par- tial embeddings. Models may also have components responsible to different parts of the signal, that may be shared. If shared components are linear and more reproducible, may be more beneficial to use residue correlation loss, after subtracting the common contribution shared among components.
The AD loss should weighted with some temperature À relative to the training (label) loss of the ensemble component. Thus the overall loss can be expressed as
Liabel + ALAD
Increasing À will, up to some point, provide smaller PD values between two or more AD ensemble models, but it may degrade the average prediction accuracy of the model. Such degradation can be compensated by larger network. Tuning À can be used to generate favorable tradeoff's between accuracy and PD with minimal expansion of the model. Values too large, however, will diminish prediction effectiveness of the model. If AD loss is present with À small enough, random initiailiza- tion of model parameters (especially of parameter rich embeddings and layers closer the input) with higher variance may improve PD without affecting accuracy.
The de-correlation loss can be further enhanced by applying techniques like Gram-Schmidt orthog- onalization on the mini batch vectors 2; before applying the AD de-correlation loss. This could ensure that the loss does not count correlated components more than once if the number of ensemble components is large. Similar or close effect, however, may be achievable by tuning the learning parameters and the temperature À of the AD loss.
In continuous online training regimes, it may be possible to extend the correlation estimates to be updated continuously over the batches of data observed, and provide a more global estimate of the correlation. This may be beneficial if the mini-batches are small, but may not give much benefit if mini-batches are large enough. Generally, a global estimate of the prediction correlation between the ensemble components may not give current training view of the mini-batch correlations, as AD is attempting to lower it. It may thus dilute the effects of the AD loss attained by previous mini-batches, possibly overestimating the already lowered correlation at given mini-batch.

---

Under review as a conference paper at ICLR 2021
Table
Anti Distillation on logit scores on MNIST with orrelation loss LAD (C2).
A
Log Loss
Acc
A
A₂
A
A H
No Ad
0.615
99.2%
0.0777
0.0305
0.0183
0.0064
0.02
0.615
99.2%
0.0040
0.0025
0.0052
0.0038
0.05
0.615
99.2%
0.0037
0.0024
0.0051
0.0036
0.1
0.616
99.2%
0.0036
0.0023
0.0051
0.0037
0.2
0.615
99.2%
0.0036
0.0024
0.0052
0.0035
0.5
0.615
99.2%
0.0036
0.0024
0.0052
0.0037
1
0.619
99.0%
0.0063
0.0040
0.0081
0.0057
Table 2: Anti-Distillation on MNIST with correlation loss and varying ensemble size J.
No Anti-Distillation A₂
Anti-Distillation (L=)(C2), À 0.5)
Acc
A
A
Acc A
A2
A
2
99.2%
0.0777
0.0305
0.0183
99.2%
0.0036
0.0024
0.0052
4
99.2%
0.0702
0.0273
0.0237
99.2%
0.0018
0.0012
0.0037
6
99.2%
0.0612
0.0236
0.0265
99.2%
0.0012
0.0008
0.0029
8
99.1%
0.0459
0.0177
0.0236
99.2%
0.0009
0.0006
0.0024
Finally, other forms of loss can also be used, as long as they generate similar effect of pushing predictions of components of the ensemble away from one another. A simple example is square loss that pushes the predictions away from one another. On a mini-batch, the loss can be expressed as
J
- =
LAD
-
J
(2i.7 23,7)
(7)
Liight
r=1
The negative sign forces the predictions away from one another instead of towards each other. This loss tries to maximize the differences between predictions of the ensemble components on individual examples, whereas de-correlation loss can tolerate individual predictions that are similar as long as they are offset by other examples in the mini-batch. Alternatively, one can use 1/(LL'nll + s) with some small 0 to guarantee large gradients when the predictions are close other than L2, can be used instead, as well as cross entropy losses. However, most experimental results suggest that de-correlation has been superior to such other losses, possibly because such losses are too restrictive on the individual examples. As such they may be pushing too much against the top level label training loss, whereas de-correlation may push in different directions on the batch vector. If the loss is applied on the batch, the network can still find different optima for each component that still generate predictions that are as good in average among the different ensemble components, but provide different individual predictions to some examples but not to all of them.
another. Norms,
5 EXPERIMENTS
We applied AD on the MNIST (LeCun, 1998). An ensemble component produces 10 logit outputs, on which softmax applied produce a label probability. Each component trains separately with cross entropy loss. AD loss (5) is applied to each of the 10 logit outputs of ensemble components (one loss for each label value). Inference on the test set only forward propagates, with no effect of the AD loss. Each model was trained over the 60,000 images in the MNIST training set, and ensemble prediction probabilities were averaged for each of the N = 10, 000 images in the MNIST test set. Each configuration
MNIST:
trained M 20 independent times.
Table 1 shows loss, accuracy and PD metrics on the MNIST test set of ensembles that consist of J 2, 2 layer components each, with both layers of width 1 1200, trained over 150 epochs. using AdaGrad (Duchi et al., 2011) (learning rate 0.02, with accumulators initialized to 0.1), zero initialization of biases, and uniform initialization of the weights. Results shown for different strengths À of AD

---

Under review as a conference paper at ICLR 2021
Table 3: Anti Distillation real data with various losses.
Log Loss
Rank Loss
AD type
change %
change %
A
No AD
0.000
0.000
0.0650
(C2
0.5
0.379
0.073
0.0473
0.903
0.126
0.0422
4.660
0.342
0.0288
0.5
0.379
0.071
0.0452
0.903
0.128
0.0403
4.656
0.324
0.0288
0.5
10
0.381
0.078
0.0411
0.907
0.143
0.0365
10
4.660
0.349
0.0266
0.018
0.011
0.0612
LA
0.069
0.120
0.0521
with correlation loss on logits. The results demonstrate that as soon apply AD, while accuracy and loss are not affected, all measures PD improve substantially. The effect of the AD strength on PD here is not substantial, long AD is not applied too aggressively. Applying AD with too much strength degrades accuracy and PD. Similar behavior to Table was obtained with covariance loss. Applying AD loss on probabilities did not improve PD.
In Table the effect the ensemble size is studied. Each component is a 2 layer network as above. Correlation loss uses À 0.5. The results show improvements all PD metrics with effect on accuracy as we increase the number of ensemble components. However, even with only two components, with AD, PD better than components without AD, and for every configuration, substantial improvements to PD are obtained with AD.
Real Data: AD tested on large real-world private dataset for ad Click-Through-Ratt (CTR) prediction for sponsored advertisement. Training was applied on hundreds of billions of examples a single pass over tini-batches of the data. Progressive validation methodology (Blum al., 1999), standard in this domain (McMahan et al., 2013), measured validation performance. Results are reported for relative PD; Åj, and logarithmic and ranking losses. We trained network, consist- ing of 3 ensemble components with several sparse embedding input vectors representing informative input features, feeding into towers of narrowing layers, activated with ReLU. A single logit output positive label. The models were trained using AdaGrad with ran- dom truncated normal initialization with standard variance normalized by the layer width. Input embeddings are trained with the network and were initialized similarly to hidden weights. We also studied the effect of scaling the input initialization variance by strength s. Each of the M models compared used the same pseudorandom generation for initialization. Results show improved PD with increased X, but with degradation in accuracy. PD also improves without effect accuracy with increasing but improvements diminish with larger X. Both covariance and difference losses improve PD less than correlation. Improvements obtained applying PD on logits, but not on probabilities.
converted to a prediction
CONCLUSIONS
Irreproducibility is a problem when predicting on individual examples with deep networks. We presented various metrics to measure Prediction Differences. We proposed Anti-Distillation, tech- nique that leverages ensembles by encouraging their components to capture different optima in the objective space. AD improves reproducibility beyond ensembling, while maintaining the superior accuracy benefits of deep models. We studied various trade-offs with AD, and reported empirical results on benchmark and real datasets, demonstrating how AD can lower PD validation data.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Alessandro Achille, Matteo Rovere, and Stefano Soatto. Critical learning periods in deep neural networks. arXiv preprint arXiv:111.008856 2017.
Rohan Amil, Gabriel Pereyra, Alexandre Passos, Robert Ormandi, George Dahl, and Geoffrey Hinton. Large scale distributed neural network training through online distillation. arXiv preprint 10X.v:8804.03355. 2018.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference machine learning, pp- 41-48, 2009.
Avrim Blum, Adam Kalai, and John Langford. Beating the hold-out: Bounds for k-fold and progres- sive cross-validation. In Proceedings of the twelfth annual conference on Computational learning theory, pp. 203-208, 1999.
Zhe Chen, Yuyan Wang, Dong Lin, Derek Cheng, Lichan Hong, Ed Chi, and Claire Cui. Beyond point estimate: Inferring ensemble prediction variation from neuron activation strength in recom- mender systems. arXiv preprint Stfi:22008.000322 2020.
Michael Cogswell, Faruk Ahmed, Ross Girshick, Larry Zitnick, and Dhruv Batra. Reducing overfit- ting in deep networks by decorrelating representations. arXiv preprint arXiv: 15 1.06668, 2015.
Thomas M. Cover and Joy A. Thomas. Elements of Information Theory (Wiley Series in Telecom- munications and Signal Processing) Wiley-Interscience, USA, 2006. ISBN 0471241954.
Elliot J Crowley, Gavin Gray, and Amos J Storkey. Moonshine: Distilling with cheap convolutions. In Advances in Neural Information Processing Systems, pp.
2018
Misha Denil, Babak Shakibi, Laurent Dinh, Marc' Aurelio Ranzato, and Nando De Freitas. Pre- dicting parameters in deep learning. Advances neural information processing systems, pp- 1156,116, 2013.
T. G. Dietterich. Ensemble methods in machine learning. Lecture Notes in Computer Science, pp- 1-15, 2000.
Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12:2121-2159, Feb. 2011.
Michael W Dusenberry, Dustin Tran, Edward Choi, Jonas Kemp, Jeremy Nixon, Ghassen Jerfel, Katherine Heller, and Andrew M Dai. Analyzing the role of model uncertainty for electronic health records. In Proceedings of the ACM Conference on Health, Inference, and Learning, pp- 204-213, 2020.
Jianping Gou, Baosheng Yu, Stephen John Maybank, and Dacheng Tao. Knowledge distillation: A survey, arXiv preprint S00iv::2006055222) 2020.
Michael Gutman. Asymptotically optimal classification for multiple tests with empirically observed statistics. IEEE Transactions on Information Theory, 35(2):401-408, 1989.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint 151.00/510.00/:: 2015.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv 2015.
preprint urXiv:1503.022
Jangho Kim, SeongUk Park, and Nojun Kwak. Paraphrasing complex network: Network compres- sion via factor transfer. In Advances in neural information processing systems, pp. 2760-2769, 2018.
Ychuda Koren. The bellkor solution to the netflix grand prize. 2009.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predic- tive uncertainty estimation using deep ensembles. In Advances in neural information processing systems, pp. 6402-6413, 2017.

---

Under review as a conference paper at ICLR 2021
Xu Lan, Xiatian Zhu, and Shaogang Gong. Knowledge distillation by on-the-fty native ensemble. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. desa-Bianchii, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 7517-7527. Curran Associates, Inc., 2018.
Yann LeCun. The mnist database of handwritten digits. hllp://yann.. lecun. com/exdb/mmist// 1998.
H Brendan McMahan, Gary Holt, David Sculley, Michael Young, Dietmar Ebner, Julian Grady, Lan Nie, Todd Phillips, Eugene Davydov, Daniel Golovin, et al. Ad click prediction: a view from the trenches. In Proceedings of the 19th ACM SIGKDD international conference Knowledge discovery and data mining, pp. 1222-1230, 2013.
Alan Mosca and George D Magoulas. Distillation of deep learning ensembles as a regularisation method. In Advances in Hybridization of Intelligent Methods, pp. 97-118. Springer, 2018.
Rafael Muller, Simon Korblith, and Geoffrey Hinton. Subclass distillation. arXiv:2002.03936. 2020.
arXiv preprint
Jonghwan Mun, Kimin Lee, Jinwoo Shin, and Bohyung Han. Learning to specialize with knowledge distillation for visual question answering. Advances in neural information processing systems, pp. 8081-8091, 2018.
Prabhat Nagarajan, Garrett Warnell, and Peter Stone. Deterministic implementations for repro- ducibility in deep reinforcement learning. arXiv preprint arXiv: 2018.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint (tt.::/91...88.. 2019.
Christopher Shallue, Jachoon Lee, Joseph Antognini, Jascha Sohl- Sohll-Dccstein, Roy Frostig, and George E Dahl. Measuring the effects of data parallelism on neural network training. arXiv preprint ar arXiv:1811.03600 2018.
Nachum Shamir, David Saad, and Emanuel Marom. Using the functional behavior of neurons for genetic recombination in neural nets training. Complex Systems, 7:445-467, 1993.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely- ppatsely-gated mixture-of-expprtt layer. arXiv preprint arXiv: / 2017.
Jiaxi Tang, Rakesh Shivanna. Zhe Zhao, Dong Lin, Anima Singh, Ed H Chi, and Sagar Jain. Under- standing and improving knowledge distillation. arXiv preprint arXiv:2002.03532, 2020.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv: 1.03530, 2016.
Ying Zhang, Tao Xiang. Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4320- 4328, 2018
10


Reviews:
[{"review_id": "lF6qxjgftSs", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nThe author proposes a method to train ensembles to have different prediction by using a correlation loss between the model's predictions. The authors show that their loss decreases the relative prediction difference between the models in the ensemble.\n\nStrong points:\nThe correlation loss is a great idea to make model predictions as different as possible.\n\nWeak points:\n- The paper is difficult to read, and the line of reasoning is difficult to follow. It might be that I am just unfamiliar with the details of literature, but for me, it seems the authors are jumping back and forth between different interpretations and motivations.\n- It is not entirely clear why the authors work on an advertisement dataset and why consistent predictions between ensembles are relevant.\n- The experimental compare only against a baseline model, but the results are not grounded in the literature by comparing it to other methods.\n- It is not entirely clear why de-correlation within an ensemble can be seen as \"anti-distillation.\"\n\nRecommendation (short):\nThis work's motivation seems to be misaligned with the content of the paper, and the experimental setup offers no comparison with similar work. I do not think this work reaches the standard of ICLR and would like to see it rejected.\n\nRecommendation (long):\nPlease see the weak points.\n\nComments for authors:\nDo not be discouraged by my recommendation. This work has some interesting results, but it currently just a little bit below scientific standards. The major flaws are the experiments are difficult to compare on their own without any comparison with other methods. The private(?) data does not help make it easy to understand the results, and the MNIST results are a bit toy-ish. I see related work does use the Criteo Display Ad Challenge dataset, which is also an advertisement dataset which you can compare with other methods. Beyond this, your motivation is initially about false positive/false negatives in the medical realm and their impact on people as a motivation for more robust methods. For me, it is not entirely clear why de-correlation helps and how the work can be seen as \"anti-distillation.\" I can see the analogy to co-distillation, so for me, it looks like co-distillation with de-correlation loss, but why is this interesting? Furthermore, the writing is difficult to follow, and you do not seem to use a latex library for citations/figures, which makes it difficult to navigate the citations. I also see some obvious spelling and grammatical errors — please run your paper through a spellchecker like Grammarly. Some of the related work is strange: Why is your work related to the model pruning literature?\n\nOverall, I would recommend working more on this paper, cleaning up all the major issues, and resubmitting it to another conference for more feedback and learning more from the process. If possible, it might also be useful for you to look for more experienced collaborators to improve writing and the overall experimental design process and the laying out of an argument based on those experiments.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "hmy359yEghf", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\n\nThe paper describes the problem of \"model irreproducibility\" - the fact that two models (with same architecture and initial weight values) trained on the same data (in different order) do not give the same prediction on the same data point at test time, while they do have the same accuracy on the test set.\n\nThe authors propose a method to quantify this effect using by looking at the absolute distance between the average prediction of an ensemble and predictions of elements of the ensemble.\n\nThe authors also propose a loss penalty to combat this effect called \"anti-distillation\". The penalty computes the correlation matrix between the average prediction of an ensemble and the prediction of an element of the ensemble and is trained to reduce the sum of squares of the off diagonal elements.\n\n## Review\n\nOne of the motivations of the authors to quantify and attempt to reduce the effect of \"model irreproducibility\" is medical imaging, where one model would predict the current image contains a pathology but another one wouldn't. This is indeed a worrisome prospect of the use of ML in diagnostics.\n\nThis problem has however been widely studied in the Bayesian literature, and is an instance of model uncertainty. If we take for example a Gaussian Process and sample a function from its predictive posterior, then that function has some predictions for certain inputs, however if we sample another function it might be different. In fact, whenever we find a lot of variation in the predictions for the same input then we consider to have large uncertainty on that input.\n\nEnsembles of neural networks allow for an effective way of quantifying uncertainty by looking at the entropy of the average prediction. If this entropy is high (meaning different classes where predicted by different ensemble elements, or the ensemble elements predict with low confidence) then the predictive uncertainty is high.\n\nUsing the concept of uncertainty, it's possible to signal to a medical practitioner that the model should not be trusted on this particular input. This approach is much preferable than if we attempt to hide this diversity, and instead focus all models on a single prediction. In fact: the diversity is important information that we can use! Avoiding it using anti-distillation, might give a false sense of security for the user of the system, but the underlying uncertainty is still there.\n\nIf the authors are interested in distilling an ensemble of models into a single model (for improved accuracy/uncertainty), which would be the outcome if the AD regularizer goes to zero, then additional comparisons need to be made for example with for example Malinin et al, 2019.\n\nIn conclusion, I think the authors look at the important problem applying ML in automated decision making, but the proposed approach should be reworked to consider the concept of uncertainty and is currently not a viable solution to the problem set out.\n\n### References:\n\nGal, Yarin. \"Uncertainty in deep learning.\" University of Cambridge 1.3 (2016).\n\nGal, Yarin, and Zoubin Ghahramani. \"Dropout as a bayesian approximation: Representing model uncertainty in deep learning.\" international conference on machine learning. 2016.\n\nKendall, Alex, and Yarin Gal. \"What uncertainties do we need in bayesian deep learning for computer vision?.\" Advances in neural information processing systems. 2017.\n\nLakshminarayanan, Balaji, Alexander Pritzel, and Charles Blundell. \"Simple and scalable predictive uncertainty estimation using deep ensembles.\" Advances in neural information processing systems. 2017.\n\nMalinin, Andrey, Bruno Mlodozeniec, and Mark Gales. \"Ensemble distribution distillation.\" arXiv preprint arXiv:1905.00076 (2019).\n\nExample medical applications:\nFilos, Angelos, et al. \"A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Tasks.\" arXiv preprint arXiv:1912.10481 (2019).\n\n## Notes\n\nTable 1: \"orrelation\" -> \"correlation\"\n\nMNIST results: difficult to interpret the deltas. You improved on reducing prediction diversity, it's not unreasonable that this should work given your objective, but why is it useful? \n\nWhy doesn't an ensemble of models improve the accuracy? On other datasets (such as cifar-10) that does happen and you might be able to measure more PD difference there.\n\nDifficult to interpret \"Real Data\", not enough details on the dataset. \"Training was applied on billions of examples in a single pass over mini-batches of data\" - does this mean you trained using one epoch? Why does it matter for the CTR if models disagree on their prediction? Isn't the original data very sparse/noisy anyway? It's unclear to me why this dataset was chosen given the motivation in the introduction.\n\nThe paper is written with many tangents, where alternatives are suggested but never analysed and limited motivation is given for the current approach. For example, why don't you use an entropy penalty on the average prediction of the ensemble? The paper can benefit from increasing the clarity, motivation and additional discussion of the results.", "extra_scores": {}, "extra_text": {}}, {"review_id": "h-_YbptgpZK", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes the Anti-Distillation method to encourage prediction diversity in an ensemble model, in order to improve reproducibility. As I understand, the reproducibility defined in this paper refers to the prediction variance w.r.t. the random factors during training, e.g., SGD.\n\nHowever, a trivial but complete reproducibility can be achieved by simply fixing the random seeds during training (without affecting model performance). Then why we would prefer the (incomplete) reproducibility induced by Anti-Distillation? If the reproducibility is the metric, then a single model or an ensemble model with fixed random seeds would trivially be the best model.\n\nBesides, the experiments are done on MNIST and a private dataset. I suggest the authors evaluating their method on public datasets like CIFAR or ImageNet, where there are many existing baselines to compare the model performance.", "extra_scores": {}, "extra_text": {}}, {"review_id": "eUD5VOHWyT", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This `work applies the batch decorrelation loss proposed in  Reducing Overfitting in Deep Networks By Decorrelating Representations [Cogswell '15] to the final predictions an ensemble of neural networks in order to produce more diverse ensembles.\n\nThe Anti-Distillation approach proposed is a method for learning diverse models which may reduce overfitting and improve performance in real-world task. Outside of applying this loss to the logits or predictions from various networks it is unclear what is the main contribution of this work relative to related works. There are mentions of various potential types of AD (Anti-distillation loss) but no ablative studies and the loss used seems identical to prior work. The tasks which this approach is tested on, MNIST and private CTR data, show negligible performance change and the results aren't clearly explained. Please provide clear captions on the charts in future and perhaps test on a tasks where ensemble methods show significant performance improvements.\n\nAn approach of this style may very much be beneficial and useful for the community in setting where overfitting significantly impacts model performance. I'd encourage the authors to investigate more challenging domains, ColoredMNIST may be an appropriate alternative. As is the work should be arranged to more clearly present the contributions of the authors and the effects of the proposed method should be more clearly illustrated.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper tries to argue the value of making ensembles more
reproducible through the use of a correlation loss to try to make
components as different as possible. The paper is tough to follow and
the high level motivation is unclear. As one of the reviewers points
out, don't ensembles provide an estimate of uncertainty and 
calibration?  Further, the experiments were quite limited. Studying
the proposed approach in a small, controlled setting might also be
revealing.
Decision: Reject