Ensemble Distribution Distillation
Authors: Andrey Malinin
Bruno Mlodozeniec
Mark Gales
Venue:  (2020)
Keywords: Ensemble Distillation
Knowledge Distillation
Uncertainty Estimation
Density Estimation

Abstract: Ensembles of models often yield improvements in system performance. These ensemble approaches have also been empirically shown to yield robust measures of uncertainty, and are capable of distinguishing between different forms of uncertainty. However, ensembles come at a computational and memory cost which may be prohibitive for many applications. There has been significant work done on the distillation of an ensemble into a single model. Such approaches decrease computational cost and allow a single model to achieve an accuracy comparable to that of an ensemble. However, information about the diversity of the ensemble, which can yield estimates of different forms of uncertainty, is lost. This work considers the novel task of Ensemble Distribution Distillation (EnD^2) - distilling the distribution of the predictions from an ensemble, rather than just the average prediction, into a single model. EnD^2 enables a single model to retain both the improved classification performance of ensemble distillation as well as information about the diversity of the ensemble, which is useful for uncertainty estimation. A solution for EnD^2 based on Prior Networks, a class of models which allow a single neural network to explicitly model a distribution over output distributions, is proposed in this work. The properties of EnD^2 are investigated on both an artificial dataset, and on the CIFAR-10, CIFAR-100 and TinyImageNet datasets, where it is shown that EnD^2 can approach the classification performance of an ensemble, and outperforms both standard DNNs and Ensemble Distillation on the tasks of misclassification and out-of-distribution input detection.

Published as a conference paper at ICLR 2020
ENSEMBLE DISTRIBUTION DISTILLATION
Andrey Malinin
Bruno Mlodozeniec Department of Engineering University of Cambridge bkm28 @ cam. ac. uk
Mark Gales
Yandex am am969eyandex-team rv
Department of Engineering
University of Cambridge
mjfg@eng cam. ac uk
ABSTRACT
Ensembles of models often yield improvements in system performance. These ensemble approaches have also been empirically shown to yield robust measures of uncertainty, and are capable of distinguishing between different forms of un- certainty. However, ensembles come at computational and memory cost which may be prohibitive for many applications. There has been significant work done on the distillation of an ensemble into a single model. Such approaches decrease computational cost and allow a single model to achieve an accuracy comparable to that of an ensemble. However, information about the diversity of the ensemble, which can yield estimates of different forms of uncertainty. is lost. This work considers the novel task of Ensemble Distribution Distillation (EnD²) - distilling the distribution of the predictions from an ensemble, rather than just the average prediction, into a single model. EnD? enables single model to retain both the improved classification performance of ensemble distillation as well as information about the diversity of the ensemble, which is useful for uncertainty estimation. A solution for EnD² based on Prior Networks, a class of models which allow a single neural network to explicitly model distribution over output distributions, is proposed in this work. The properties of EnD² are investigated on both an artificial dataset, and on the CIFAR-10, CIFAR- 100 and Iiny cltasstts, where it is shown that EnD² can approach the classification performance of an ensemble, and outperforms both standard DNNs and Ensemble Distillation on the tasks of misclassification and out-of-distribution input detection.
INTRODUCTION
Neural Networks (NNs) have emerged as the state-of-the-an approach to a variety of machine
learning tasks (eeCun et 2015) in domains such as computer vision (Girshick] [2015] Simonyan
& Zisserman 2015 Villegas et al.] 2017), natural language processing (Mikolov et al. 2013b a
2010), speech recognition (Hinton et al. 2012; Hannun et al. (2014) and bio-informatics (Caruana ct al. 2015] Alipanahi et al. 2015). Despite impressive supervised learning performance, NNs tend to make over-confident predictions (Lakshminarayanann et al.) (017)) and, until recently, have been unable to provide measures of uncertainty in their predictions. As NNs are increasingly being applied to safety-critical tasks such medical diagnosis (De Fauw et al. 2018), biometric identification (Schroff et al. 2013) and self driving cars, estimating uncertainty in model's 's predictions is crucial, as enables the safety of an AI system (Amodei al. 2016) to be improved by acting on the predictions in an informed manner.
Ensembles of NNs are known to yield increased accuracy over single model (Murphy 2012). allow useful measures of uncertainty to be derived (Lalshminarayanan et al. 2017), ), and also provide defense against adversarial attacks (Smith & Gal| 2018). There is both range Bayesian Monte- Carlo approaches (Gal & Ghahramani (216)] Welling & Tch 2011 Garipov et al. 2018 Maddox et al. 2019, as well as non- BayesBa approaches, such as 2017) and bagging (Murphy) 2012 OObaand et al. 2016), to generating ensembles. Crucially, ensemble approaches allow total uncertainty in predictions to be decomposed into knowledge uncertainty and data uncertainty. Data uncertainty is the irreducible uncertainty in predictions which arises due to the complexity, multi-modality and noise in the data. Knowledge uncertainty, also known as epistemic
initializzzionnonn aakssmiiaaayannnn et al.
"Equal Contribution

---

Published as a conference paper at ICLR 2020
uncertainty (Gal 2016) distributional uncertainry (Malinin & Gales 2018), is uncertainty due to lack of understanding or knowledge on the part of the model regarding the current imput for which the model is making prediction. This form of uncertainty arises when the test input x" comes either from a different distribution than the one that generated the training data or from an in-domain region which is sparsely covered by the training data. Mismatch between the test and training distributions is also known as a dataset shift "Quiñonero-Cander
2009), and is a situation which often arises for real world problems. Distinguishing between sources of uncertainty is important, as in certain machine learning applications it may be necessary to know not only whether the model is uncertain, but also why. For instance, in active learning. additional training data should be collected from regions with high knowledge uncertainly, but not data uncertainty.
fundamental limitation of ensembles is that the computational cost of training and, more importantly, inference can be many times greater than that of a single model. One solution to speed up inference to distill an ensemble of models into a single network to yield the mean predictions of the ensemble (Hinton et al. 2015; Korattikara Balan et al. 2015). However, this collapses an ensemble of conditional distributions over classes into a single point-estimate conditional distribution over classes. As a result, information about the diversity of the ensemble is lost. This prevents measures of knowledge uncertainty, such as mutual information (Malinin & Gales 2018 Depeweg et al. 2017a). from being estimated.
In this work, we investigate the explicit modelling of the distribution over the ensemble predictions, rather than just the mean, with single model. This problem referred to Ensemble Distribution Distillation (EnD²)- yields a method that preserves both the distributional information and improved classification performance of an ensemble within a single neural network model. It is important to highlight that Ensemble Distribution Distillation is a novel task which, to our knowledge, has not been previously investigated. Here, the goal is to extract as much information as possible from an ensemble of models and retain it within a single, possibly simpler, model. As an initial solution to this problem, this paper makes use of a recently introduced class of models, known as Prior Networks (Malinin & Gales 2018 2019), which explicitly model a conditional distribution over categorical distributions by parameterizing a Dirichlet distribution. Within the context of EnD? this effectively allows single model to emulate the complete ensemble.
The contributions of this work are as follows. Firstly, we define the task of Ensemble Distribution Distillation (EnD²) as a new challenge for machine learning research. Secondly, we propose and evaluate a solution to this problem using Prior Networks. EnD² is initially investigated on artificial data, which allows the behaviour of the models be visualized. It is shown that distribution- distilled models are able to distinguish between data uncertainty and knowledge uncertainty. Finally, EnD² is evaluated on CIFAR- CIFAR-100 and Imngenteeeet datasets, where it is shown that EnD² yields models which approach the classification performance of the original ensemble and outperform standard DNNs and regular Ensemble Distillation (EnD) models on the tasks of identifying misclassifications and out-of-distribute (OOD) samples.
2 ENSEMBLES
In this work, a Bayesian viewpoint on ensembles is adopted, provides particularly elegant prob- abilistic framework, which allows knowledge uncertainty to be linked to Bayesian model uncertainty. However, is also possible to construct ensembles using a range of non-Bayesian approaches. For example, it is possible to explicitly construct an ensemble of M models by training on the same data with different random seeds (Lakshminarayanan et al. (017) and/or different model architectures. Alternatively, it is possible to generate ensembles via Bootstrap methods (Murphy) 2012) Osband ctt al. 2016) in which each model is trained on a re-sampled version of the training data.
The essence of Bayesian methods is to treat the model parameters O as random variables and place prior distribution p(0) over them to compute a posterior distribution p(0) D) via Bayes' rule:
p(D p(D(0)p(0) 0)p(0) p(D)0)p(()
p P(
(1)
(DD
Here, model uncertainty is captured in the posterior distribution p() Consider an ensemble of models {P(y)æ", ) sampled from the posterior:
{P(ylæ*, )(m)) M m=1 + +P(y) (mm))) m π (m)) f(a=; ((m)), 0 ((m) ~ p(((D)
(2)
2

---

Published as a conference paper at ICLR 2020
(a) Confident Prediction
(b) Data Uncertainty
(c) Knowledge Uncertainty
Figure 1: Desired behaviors of ensemble on simplex of categorical probabilities.
where π are the parameters of categorical distribution [P(y = w1), P(y = wx) )F" and x" is test input. The expected predictive distribution, or predictive posterior, for test input x" obtained by taking the expectation with respect to the model posterior:
P(ylæ", D) (pporp) [(yy (a))a=""))) 0)]
Each of the models x ((m)) yields different estimate of data uncertainty. Uncertainty in predictions due to model uncertainty is expressed as the level of spread, or 'disagreement' of an ensemble sampled from the posterior. The aim is to craft posterior p(0 via an appropriate choice prior p(0), which yields an ensemble that exhibits the set of behaviours described in figure [ Specifically, for in-domain test input x",, the ensemble should produce consistent set of predictions with little spread, as described in figure Ta and figure db In other words, the models should agree in their estimates of data uncertainty. On the other hand, for inputs which are different from the training data, the models in the ensemble should 'disagree" and produce diverse set of predictions, as shown in figure [c] Ideally, the models should yield increasingly diverse predictions as input x" moves further away from the training data. If an input is completely unlike the training data, then the level of disagreement should be significant. Hence, the measures of model uncertainty will capture knowledge uncertainty given an appropriate choice of prior.
Given an ensemble (P(y) x=" " (((m))) m= which exhibits the desired set of behaviours, the entropy of the expected distribution P(y|a>" DD can be used as measure of total uncertainty in the prediction. Uncertainty in predictions due to knowledge uncertainty can be assessed via measures of the spread, or 'disagreement', of the ensemble such as Mutual Information:
MI 0 "c"" , D)
H [Epco (Ply|=" 0)]]
Eo(orD) (H(((vlx' 0)]
(4)
K Knowledge Uncertainty
Total Uncertainty
Expected Data Uncertainty
This formulation of mutual information allows the total uncertainty to decomposed into knowledge uncertainty and expected data uncertuinty (Depeweg et al. 2017a b). The entropy of the predictive posterior, or total uncertainty, will be high whenever the model is uncertain both in regions of severe class overlap and out-of-domain. However, the difference of the entropy of the predictive posterior and the expected entropy of the individual models will be non-zero only if the models disagree. For example, in regions of class overlap, each member of the ensemble will yield a high entropy distribution (figure [ip) the entropy of the predictive posterior and the expected entropy will be similar and mutual information will be low. In this situation total uncertainty is dominated by data uncertainty. On the other hand, for out-of-domain inputs the ensemble yields diverse distributions over classes such that the predictive posterior is near uniform (figure Ic while the expected entropy of each model may be much lower. In this region of input space the models' understanding of data is low and, therefore, knowledge uncertainty high.
ENSEMBLE DISTRIBUTION DISTILLATION
Previous work Hinton et al. 2015 Korattikara Balan al. 2015 Wong & Gales 2017 Wang et al.)
2018 Papamakarios 2015 Bucilua et al.) 2006) has investigated distilling a single large network into smaller and an ensemble of networks into single neural network. In general, distillation is done by minimizing the KL-divergence between the model and the expected predictive distribution of ensemble:
L(Ø, Dena) = -E(c))) KL Epcerp) PPyy le;; 0)] || P(yla; ]]]]
(5)
3

---

Published as a conference paper at ICLR 2020
This approach essentially aims to train a single model that captures the mean of an ensemble, allowing the model to achieve a higher classification performance at a far lower computational cost. The use of such models for uncertainty estimation Azizpour 2019). However, the limitation of this approach with regards to uncertainty estimation is that the information about the diversity of the ensemble is lost. As result, it is longer possible to decompose total uncertainty into knowledge uncertainty and data uncertainty via mutual information as in equation | In this work we propose the task of Ensemble Distribution Distillation, where the goal is to capture not only the mean of the ensemble, but also its diversity. In this section, we outline an initial solution to this task.
investigated in (Li & Hoiem 2019: Englesson &
An ensemble can be viewed as set of samples from an implicit distribution of output distributions:
(P(ylæ*, , A + (P(y) (m))) m= ) 7(
p
Recently, new class of models was proposed, called Prior Networks (Malinin & Gales 2018 2019), which explicitly parameterize a conditional distribution over output distributions p( π x.; using a single neural network parameterized by point estimate of the model parameters d. Thus, Prior Network is able to effectively emulate an ensemble, and therefore yield the same measures of uncertainty, A Prior Network p(T æ=; Φ) models a distribution over categorical output distributions by parameterizing the Dirichlet distribution.
p[ Φ) = -ir( à= f(;(z;)) âe > 0, ā0
The distribution is parameterized by its concentration parameters a, which can be obtained by placing an exponential function the output of Prior Network: å, = E where & are the logits predicted by the model. While a Prior Network could, in general, parameterize arbitrary distributions over categorical distributions, the Dirichlet is chosen due its tractable analytic properties, which allow closed form expressions for all measures of uncertainty to be obtained. However, it is important to note that the Dirichlet distribution may be too limited to fully capture the behaviour of an ensemble and other distributions may need to be considered.
In this work we consider how an ensemble, which is set of samples from an implicit distribution over distributions, can be distribution distilled into explicit distribution over distributions modelled using a single Prior Network model,
{P(u) lx; ((m))) m= -
p(T x Φ)
This is accomplished in several steps. Firstly, a transfer dataset Dena = (x) ( (1.1:M) J.N. ~
p(xx TT) is composed of the inputs from the original training set 'D = (x((), ( 1 and the categorical distributions { (i.1-M)) Ji=1 derived from the ensemble for each input. Secondly, given
this transfer set, the model p( (x) Φ)) is trained by minimizing the negative log- likelihood of each categorical distribution π (imm:
L(Ø, Dane) Ep(ar) (pprrt) [npp(T/); Φ)]]
1
K Σ In
- -
InI((
M
K
(a.
(im)
(8)
M
Thus, Ensemble Distribution Distillation with Prior Networks is a straightforward application of estimation. Given a distribution-distillee Prior Network, the predictive distribu- tion is given by the expected categorical distribution Å under the Dirichlet prior:
āe
(y " ?) E pp (P(y = wel [T)] = *
Σk k= OR Ek=1 elc
Separable measures of uncertainty can be obtained by considering the mutual information between the prediction y and the parameters of π of the categorical:
MI S)
H P( P(y|T)]]
E P(T : H(P(()|T))
(10)
K Knowledge Uncertainty
Total U Uncertainty
Expected Data Uncertainty
Similar to equation! 4 this expression allows total uncertainty, given by the entropy of the expected distribution, to be decomposed into data uncertainty and knowledge uncertainty (Malinin & Gales 2018). If Ensemble Distribution Distillation is successful, then the measures uncertainty derivable from distribbtioon-distillee model should be identical to those derived from the original ensemble.
4

---

Published as a conference paper at ICLR 2020
3.1 TEMPERATURE ANNEALING
Minimization of the negative log-likelihood of the model on the transfer dataset Dana {x(), π (6.1::M), Ji=1 is equivalent to minimization the KL-divergence between the model and the empirical distribution p(x, TT). On training data, this distribution is often 'sharp' at of the corners of the simplex. At the same time, the Dirichlet distribution predicted by the model has its mode near the center of the simplex with little support at the corners at initialization. Thus, the com- mon support between the model and the target empirical distribution is limited. Optimization of the KL-divergence between distributions with limited non-zero common support is particularly difficult. To alleviate this issue, and improve convergence, the proposed solution is to use temperature to 'heat up' both distributions and increase common support by moving the modes of both distributions closer together. The empirical distribution is 'heated up' by raising the temperature T of the softmax of each model in the ensemble the same way as (Hinton al. 2013). This moves the predictions of the ensemble closer to the center of the simplex and decreases their diversity, making it better modelled by a sharp Dirichlet distribution. The output distribution of the EnD² model ( [æ:; Φ) is heated up by raising the temperature of the concentration parameters: å, = E making support more uniform across the simplex. An annealing schedule is used to re-emphasize the diversity of the empirical distribution and return it to its 'natural' state by lowering the temperature down to 1 as training progresses.
4 EXPERIMENTS ON ARTIFICIAL DATA
40
-400
400
500
1000
(a) Spiral Dataset
(b) Spiral Dataset with AUX data (+)
Figure 2: 3-spiral dataset with 1000 examples per class
The current section investigates Ensemble Distribution Distillation (EnD?) on an artificial dataset shown in figure Ph. This dataset consists of three spiral arms extending from the center with both increasing noise and distance between the arms. Each arm corresponds to a single class. This dataset chosen such that is not linearly separable and requires a powerful model correctly model the decision boundaries, and also such that there are definite regions of class overlap.
In the following set of experiments, an ensemble of 100 neural networks constructed by training neural networks from 100 different random initializations. A smaller (sub) ensemble of only 10 neural networks also considered. The models are trained on 3000 data-points sampled from the spiral dataset, with 1000 examples per class. The classification performance of EnD is compared to the performance of individual neural networks, the overall ensemble and regular Ensemble Distillation (EnD). The results are presented in table [
Table I: Classification Performance (% Error) on Drest of size 1000, trained on Dern of size 1000 with spiral classes. Dataset sizes given as number of examples per class.
Num. models
Individual
Ensemble
EnD
EnD
10
12.63
12.57
12.52
13.21
100
12.37
12.39
12.47
The results show that an ensemble of 10 models has a clear performance gain compared to the mean performance of the individual models. An ensemble of 100 models has smaller performance gain
5

---

Published as a conference paper at ICLR 2020
ensemble of only 10 models. Ensemble Distillation (EnD) is able to recover the classification performance of both an ensemble of and 100 models with only very minor degradation in perfor- mance. Finally, Ensemble Distribution Distillation is also able to recover most of the performance gain of an ensemble, but with slightly larger degradation. This is likely due to foreing a single model to learn not only the mean, but also the distribution around it, which likely requires more capacity from the network. The measures of uncertainty derived form an ensemble of 100 models and
(a) Ensm. Total Uncertainty
(b) Ensm. Data Uncertainty
(c) Ensm. Knowledge Uncertainty
(d) D Total Uncertainty
EnD? Data Uncertainty
(f) EnD Knowledge Uncertainty
(g) EnD; aux Total Uncertainty
(h) EnD" aux Data Uncertainty
(i) EnD Am Knowledge Uncertainty
Figure 3: Comparison of measures of uncertainty derived from an Ensemble, EnD? and EnD?
+AUx
from Ensemble Distribution Distillation are presented figures B and figures 3-f,, respectively. The results show that EnD² successfully captures data uncertainty and also correctly decomposes total uncertainty into knowledge uncertainty and data uncertainty. However, it fails to appropriately capture knowledge uncertainty further away from the training region, there are obvious dark holes in figure B where the model yields low knowledge uncertainty far from the region of training data.
In order to overcome these issues, thick ring of inputs far from the training data was sampled as depicted in figure The predictions of the ensemble were obtained for these input points and used as additional auxiliary training data DAUX- Table shows how using the auxiliary training data affects the performance of the Ensemble Distillation and Ensemble Distribution Distillation. There is minor drop performance both distillation approaches. However, the overall level performance not compromised and is still higher than the average performance of each individual DNN model. The behaviour of measures of uncertainty derived from Ensemble Distribution Distillation with auxiliary training data (EnD? aux) is shown in figures B These results show that successful Ensemble Distribution Distillation of the out-of tott-f--iitribution behaviour of an ensemble based purely observations of the in-domain behaviour challenging and may require the of additional

---

Published as a conference paper at ICLR 2020
training data. This is compounded by the fact that the diversity of an ensemble training data that the model has seen is typically smaller than on a heldout test-set.
Table Classification Performance (% Error) on Dtests trained on either Dern or Dera - DAUX- All datasets of size 1000. Data for ensemble of a 100 models.
Distillation Data
Individual
Ensemble
EnD
EnD
Dern Dern + DAUX
12.39
12.47
13.21
12.37
12.41
12.50
5 EXPERIMENTS ON IMAGE DATA
Having confirmed the properties of EnD² on an artificial dataset, we now investigate Ensemble Distribution Distillation on the CIFAR-10 (C10), CIFAR-100 (C100) and TinyImageNet (TIM) Krizhevsky 2009 CS231N 2017) datasets. Similarly to section an ensemble of 100 models constructed by training NNs C10/100/TIM, data from different random initializations. The transfer datasel is constructed from C10/100/TIN inputs and ensemble logits to allow for lemperature annealing during training, which found to be essential to getting EnD² to train well. In addition, we also consider Ensemble Distillation and Ensemble Distribution Distillation on a transfer set that contains both the original C10/C100/TIM training data and auxiliary (AUX) data taken from the other datase(" termed EnD++ux +Axx and EnD; + AUX respectively. It is important to note that the auxiliary data has been treated in the same way as main data during construction of the transfer set and distillation. This offers an advantage over traditional Prior Network training (Malinin & Gales) 2018 (2019), where the knowledge which examples are in-domain and out-of-distribution is required a-priori. this work we also make a comparison with Prior Networks trained via reverse KL-divergence (Malinin & Gales) 2019), where the Prior Networks (PN) are trained on the same datasets, both main and auxiliary, as the EnD X and EnD? x models? Note, that for Ensemble Distribution Distillation, models can be distribution-ddistlleedd using any (potentially unlabeled) auxiliary data on which ensemble predictions be obtained. Further note that in these experiments
explicitly chose use the simpler VGG-16 (Simonyan & Zisserman 2013) architecture rather than more modern architectures like ResNet (He) al. (2166) the goal of this work is to analyse the properties of Ensemble Distribution Distillation in a clean and simple configuration. The datasets and training configurations for all models are detailed in appendix A
Table 3: Mean Classification Error, % PRR, test-set negative log- likelhoooo (NLL) and expected calibration error (ECE) on C10/C100/TIM across three models +20.
DSET
CRIT.
IND
ENSM
EnD
EnD>
EnD AUX
EnD? EnD +AUX
PN+
ERR
8.0
6.2 NA
6.7 #0.3
7.3 20.2
6.7 10.2
6.9 40.2
7.5 #0.3
PRR
84.6 #1.2
86.8 NA
84.8 #0.8
85.3 #1.1
85.1 #0.1
85.7 +0.3
82.0 #1.4
C10
BCE
2.2 20.4
1.3 NA
2.6 10.2
1.0 10.2
2.6 10.6
2.2 10.4
12.0 10.7
NLL
0.25 10.01
0.19 NA
0.22 =0.01
0.25 #0.01
0.22 20.01
0.24 #0.00
0.38 20.01
ERR
30.4 #0.3
26.3 NA
28.0 #0.4
27.9 +0.3
28.2 +0.3
28.0 +o.a
28.0 +0.7
PRR
72.5 11.0
75.0 = NA
73.1 10.5
73.7 20.7
74.0 10.3
74.0 20.2
63.7 10.8
C100
RCE
9.3 20.8
1.2 NA
8.2 20.3
4.9 20.5
1.9 20.3
5.6 10.5
37.9 10.4
NLL
1.16 +00.03
0.88 NA
1.06 00.01
1.14 40.01
0.98 #0.00
1.14 +0.01
1.87 #.0.3
ERR
41.8 20.6
36.6 ANA
38.3 #0.2
37.6 #0.2
38.5 20.3
37.3 40.5
40.0 #0.6
PRR
70.8 11.1
73.8 NA
72.2 +0.2
73.1 #0.1
72.6 +1.3
72.7 #1.1
62.3 #0.6
TIM
BCE
18.3 20.8
3.8 NA
14.8 10.4
7.2 10.4
14,9 10.3
7.2 40.2
39,1 11.0
NLL
2.15 10.05
1.51 NA
1.77 00.01
1.83 10.02
1.78 20.01
1.84 10.02
2.61 20.01
The auxiliary training data for CIFAR-10 is CIFAR-100, and CIFAR-10 for CIFAR-100/iny 10TT-1000/miyymaeeeett This is consistent configuration to the AUX and EnD? x models, but constitutes degraded OOD detection baseline for CIFAR-100 and TinyImageNet datasets due the choice of auxiliary data.
7

---

Published as a conference paper at ICLR 2020
Firstly, we investigate the ability of single model to retain the ensemble's classification and prediction-reeection (misclassification detection) performance after either Ensemble Distillation (EnD) or Ensemble Distribution Distillation (EnD²), with results presented in table 3 in terms of error rate and prediction rejection ratio (PRR), respectively. A higher PRR indicates that the model is able to better detect and reject incorrect predictions based measures of uncertainty. This metric is detailed in appendix [ Note that for prediction rejection we used the confidence of the max class, which also measure of total uncertainty, like entropy, but more sensitive to the prediction Malinin & Gales 2018).
Table shows that both EnD and EnD² are able retain both the improved classification and prediction-rejection performance of the ensemble relative to individual models trained with maximum likelihood on all datasets, both with and without auxiliary training data. Note, that on C100 and TIM, EnD² yields marginally better classification performance than EnD. Furthermore, EnD? also either consistently outperforms or matches EnD in terms of PRR on all datasets, both with and without auxiliary training data. This suggests that EnD is able to yield benefits top of standard Ensemble Distillation due to retaining information about the diversity of the ensemble. In comparison, Prior Network, while yielding classification performance between that EnD X- performs consistently worse than all models in terms of PRR and for all datasets. This is likely because of the inappropriate choice of auxiliary training data and auxiliary loss weight (Malinin & Gales 2019) for this task.
individual model and
Secondly, it is known that ensembles yield improvements in the calibration of model's predictions (Lakshminarayanse yanan et al. 2017); Ovadia et al.) 2019). Thus, it is interesting to see whether EnD and EnD models retain these improvements terms of test-set negative log-likelihood (NLL) and expected calibration error (ECE). Note that calibration assesses uncertainty quality on a per-dataset, rather than per-predeee per-prediction, level. From table B we can see that both Ensemble Distillation and Ensemble Distribution Distillation seem to give similarly minor gains in NLL over a single model. However, EnD seems to have marginally better NI.I. performance, while EnD² tends to yield better calibration performance. There are seemingly limited gains in ECE and NLL when using auxiliary data during distillation for EnD?, and sometimes even a degradation in NLL and ECE. This may be due to the Dirichlet output distribution attempting to capture non- con-Drrichlet-eee -hrrcheee-distribueed ensemble predictions (especially on auxiliary data) and over-estimating the support, and thereby failing to fully reproduce the calibration of the original ensemble. Furthermore, metrics like ECE and NLL are evaluated on in-domain data, which would explain the lack of improvement from distilling ensemble behaviour on auxiliary data. However, all distillation models are (almost) always calibrated as better than individual models. At the same time, the Prior Network models yield significantly worse calibration performance in terms of NLL and ECE than all other models. For the CIFAR- 100 and TIM datasets this may be due to the target being flat Dirichlet distribution for the auxiliary data (CIFAR-10), which drives the model to be very under-confident.
Table 4: OOD detection performance (mean % AUC-ROC +o2 for C10/C100/TIM models using measures of total (T.Unc) and knowledge (K.Unc) uncertainty.
Train. Data
OOD
Unc.
Individual
Ensemble
EnD
EnD
EnD +AUX
EnD? +Aux
PN
Data
T.Unc Kne
91.3 11.3
94.5 1NA
89.0 11.3
91.5 20.8
88.6 #1.1
94.4 20.7
95.7 20.9
LSUN
94.4
92.2 40.7
93.8 #0.7
95.8 40.8
C10
T.Unc K.Une
88.9 +1.6
91.8
86.9
88.6 41.5
86.5
91.3
95.7 40.7
TIM
91.4 ANIA
88.8 11.6
90.6 =0.7
95.8 20.7
Unc K. Une
75.6 111
82.4 ANIA
73.8 10.6
80.6 111
80.6 10.6
83.6 20.5
74.8 11177
LSUN
88.4 =NIA
83.8 #0.9
86.5 20.6
73.8 42.2
C100
T.Une Une
70.5 #1.6
76.6 =NIA
68.5 #1.2
74.4 11.6
74.2 20.8
77.7 20.9
73.4 42.9
TIM
81.7 1NA
77.2 1.1
80.5 1111
72.7 13.33
T.Une
67.5 +1.3
69.7 =NA
68.7 +0.2
69.6
68.8 #0.2
69.2 #0.7
63.7 40.1
LSUN
KnCne
69.3 AN/A
70.4 1.3
70.3 20.4
59.5 #1.9
TIM
T.Unc Unc
71.7 #2.5
75.2 AN/A
73.1 10.7
74.8 #0.3
73.1 20.3
74. 1..
100.0 10.0
C100
78.8 +NIA
76.7 40.5
75.3 #0.7
100.0 +0.00
8

---

Published as a conference paper at ICLR 2020
Thirdly, Ensemble Distribution Distillation is investigated the task of out-of-domain (OOD) input detection (Hendrycks & Gimpel 2016). where measures of uncertainty are used to classify inputs as either in-domain (ID) or OOD. The ID examples are the test set of C10/100/TIM, and the test OOD examples are chosen to be the test sets of LSUN (Yu et al. 2015). C100 or TIM, such that the test OOD data is never seen by the model during training. Table 4 shows that the measures of uncertainty derived from the ensemble outperform those from a single neural network. Curiously, standard ensemble distillation (EnD) clearly fails to capture those gains on C1O and TIM, both with and without auxiliary training data, but does reach comparable level of performance to the ensemble on C100. Curiously, EnD+ AGE performs worse than the individual models on C10. This is in contrast to results from (Englesson & Azizpour 2019; Li & Hoiem| 2019). but the setups, datasets and evaluation considered there are very different to ours. On the other hand, Ensemble Distribution Distillation is generally able to reproduce the OOD detection performance of the ensemble. When auxiliary training data is used, EnD² is able to perform on par with the ensemble, indicating that it has successfully learned how the distribution of the ensemble behaves unfamiliar data. These results suggest that not only is EnD² able to preserve information about the diversity of an ensemble, unlike standard EnD, but that this information is important to achieving good OOD detection performance. Notably, on the CIFAR-10 dataset PNs yield the best performance. However, due to generally inappropriate choice of OOD (auxiliary) training data, PNs show inferior (with one exception) OOD detection performance on 110 and TIM.
Curiously, the ensemble sometimes displays better OOD detection performance using measures of total uncertainty. This is partly a property of the in-domain dataset- if it contains a small amount of data uncertainty, then OOD detection performance using total uncertainty and knowledge uncertainty should be almost the same (Malinin & Gales) 2018 Malinin 2019). Unlike the toy dataset considered in the previous section, where significant data uncertainty was added, the image datasets considered here do have a low degree of data uncertainty. It is important to note that more challenging tasks, which naturally exhibit higher level of data uncertainty, we would expect that the decomposition would be more beneficial. Additionally, it is also possible that if different architectures, training regimes and ensembling techniques are considered, an ensemble with a better estimate of knowledge uncertainty with auxiliary training data, tends to have better OOD detection performance using measures of knowledge uncertainty. This may be due to the use of Dirichlet output distribution, which may not be able to fully capture the details of the ensemble's ' behaviour. Furthermore, as EnD is (implicitly) trained by minimizing the forward KL-divergence, which is zero-avoiding (Murphy) 2012). it likely that the distribution learned by the model is 'wider' than the empirical distribution. This effect is explored in the next subsection and in appendix C
be obtained. Despite the behaviour of the ensemble, EnD?, especially
5.1 APPROPRIATENESS OF DIRICHLET DISTRIBUTION
Throughout this work, Prior Network that parametrizes Dirichlet was used for distribution- distilling ensembles of models. However, the output distributions of an ensemble for the same input not necessarily triichle-diitributee especially in regions where the ensemble is diverse. In the previous section we saw that EnD² models tend to have higher NLL than EnD models, and while EnD2 achieves good OOD detection performance, it doesn't 't fully replicate the ensemble's behaviour. Thus, in this section, we investigate how well a model which parameterizes a Dirichlet distribution is able to capture the exact behaviour of an ensemble of models, both in-domain and outto--distribution.
Figure 4 shows histograms of total uncertainty, data uncertainty and total uncertainty yielded by an ensemble, EnD² and EnD" Aux models trained on the CIFAR-10 dataset. The top row shows the uncertainty histogram for ID data, and the bottom for test OOD data (a concatenation of LSUN and TIM). On in in-domain data, EnD² is seemingly able to emulate the uncertainty metrics of the ensemble well, though does have longer tail of high-uncertainty examples. This is expected, as on in-domain examples the ensemble will be highly concentrated around the mean. This behaviour can be adequately modelled by a Dirichlet. On the other hand, there is a noticeable mismatch between the ensemble and EnD2 in the uncertainties they yield on OOD data. Here, EnD² consistently yields higher uncertainty predictions than the original ensemble. Notably, adding auxiliary training data makes the model yield even higher estimates of total uncertainty and data uncertainty. At the same time, by using auxiliary training data, the model's distribution of knowledge uncertainty starts to look more like the ensembles, but shifted to the right (higher).
9

---

Published as a conference paper at ICLR 2020
Creanile
0.7
Dep bell
066
0.4
87
011
80
055
500
2.5
80
0.5
50
1.5
2.
80
65
100
20
(a) Total Uncertainty ID
(b) Data Uncertainty
(c) Knowledge Uncertainty ID
0.12
Driambe
Enartbe
61
Enerde
010
0.10
0.19
004
040
0.0
0.0
0.0
(d) Total Uncertainty OOD
(c) Data Uncertainty OOD
(1) Knowledge Uncertainty OOD
Figure 4: Histograms of uncertainty of the CIFAR-10 ensemble, EnD² and EnD?, +Aux on in-domain (ID) and test out-of-domain (OOD) data.
Altogether, this suggests that samples from the ensemble are diverse in a way that's different from Dirichlet distribution. For instance, the distribution could be multi-modal or crescent-shaped. Thus, as a consequence of this, combined with forward KL-divergence between the model and the empirical distribution of the ensemble being zero-avoiding, the model over-estimates the support of the empirical distribution, yielding an output which is both more diverse and higher entropy than the original ensemble. This does not seem adversely impact OOD detection performance as measures of uncertainty for ID and OOD data are further spread apart and the rank ordering of ID and OOD data is either maintained or improved, which is supported by results from section [ However, this does prevent the EnD² models from fully retaining the calibration quality of the ensemble. It is possible that the ensemble could be better modelled by a different output distribution, such as a mixture of Dirichlet distributions or a Logistic-normal distribution.
6 CONCLUSION
Ensemble Distillation approaches have become popular, as they allow a single model to achieve classification performance comparable to that of an ensemble at a lower computational cost. This work proposes the novel task Ensemble Distribution Distillation (EnD?)- - an ensemble into a single model, such that it exhibits both the improved classification performance the ensemble and retains information about its diversity. An approach to EnD" based using Prior Network models is considered in this work. Experiments described in sections 4 and 5Jshow that on both artificial data and image classification tasks it is possible to distribution distill an ensemble into single model such that it retains the classification performance of the ensemble. Furthermore, measures of uncertainty provided by EnD? models match the behaviour of an ensemble of models artificial data, and EnD" models are able to differentiate between different types of uncertainty. However, this may require obtaining auxiliary training data on which the ensemble is more diverse in order allow the distribution- -distilled model to learn appropriate out-of-domain behaviour. On image classification tasks measures of uncertainty derived from EnD² models allow them to outperform both single NNs and EnD models on the tasks of misclassification and out- out-of-distribution input detection. These results are promising, and show that Ensemble Distribution Distillation enables a single model to capture more useful properties of an ensemble than standard Ensemble Distillation. Future work should further investigate properties of temperature annealing, investigate ways to enhance the diversity of an ensemble, consider different sources of ensembles and model architectures, and examine more flexible output distributions, such as mixtures of Dirichlets. Furthermore, while this work considered Ensemble Distribution Distillation only for classification problems, it can and should also be investigated for regression tasks. Finally, it may be interesting to explore combining Ensemble Distribution Distillation with standard Prior Network training.
10

---

Published as a conference paper at ICLR 2020
ACKNOWLEDGEMENTS
This research was partly funded by Cambridge Assessment English via the ALTA Institute, Cambridge University.
REFERENCES
Babak Alipanahi, Andrew Delong, Matthew T. Weirauch, and Brendan Frey. Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning. Nature Biotechnology. 33(8):831-838, July 2015. ISSN 1087-0156. doi: 10. 10.1038/mbt.3300.
//o:://xxx.a doi
org/10. nbt. 3300
Dario Amodei, Chris Olah, Jacob Steinhardr, Paul F. Christiano, John Schulman, and Dan Mané. Concrete problems in Al safety. hnttp://arxivv.rgg
06565) 2016. arXiv:
1606.06565.
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizzi, Model compression. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD *06, pp. 535-541, New York, NY, USA, 2006. ACM. ISBN 1-59593-339-5. doi: 10.1145/ 1550402.1150466 URL htttt:///ooo acm
111.1..111111111 1150444
Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Marc Sturm, and Noemie Elhadad. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proc. 2/th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '1, pp. 1721-1730, New York, NY, USA, 2015. ACM. ISBN 978- 978-1-44503-3664-22 doi: 10.1145/2783258. 2788613. URL httt:///oo...ca..cor///..1111111
acm.
Stanford CS23IN. Tiny
hww.ttttp..hkk.ppp.... com/ 2017.
Jeffrey De Fauw, Joseph R Ledsam, Bernardino Romera-Paredes, Stanislav Nikolov, Nenad Tomasev, Sam Blackwell, Harry Askham, Xavier Glorot, Brendan O' D'Donoghue, Daniel Visentin, et al. Clinically applicable deep learning for diagnosis and referral in retinal disease. Nature medicine, 24(9):1342, 2018.
Stefan Depeweg, José Miguel Hernández-Lobato, Finale Doshi-Velez, and Steffen Udluft. Decompo- sition of uncertainty for active learning and reliable reinforcement learning in stochastic systems. stat, 1050: 1, 2017a.
Stefan Depeweg, José Miguel Hernández-Lobato. Finale Doshi-Velez, and Steffen Udluft. Decom- position of uncertainty bayesian deep learning for efficient and risk-sensitive learning. arXiv preprint
2017b.
Erik Englesson and Hossein Azizpour. Efficient evaluation- time uncertainty estimation by improved distillation. arXiv preprini arXiv: 2019.
Yarin Gal. Uncertainty in Deep Learning. PhD thesis, University of Cambridge, 2016.
Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. In Proc. 33rd International Conference Machine Learning (ICML-16), 2016.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianche and R. Gar- nett (eds.). Advances in Neural Information Processing Systems 31, pp. 8803- 8812. Curran Associates, Inc., 2018. URL http:/ htap:://paperr... ceaapers.niss nips cc/paper/
8095 loss - -arf-ueuraaaeeeee eirracessmddee. mode- connect ivity- fast -etemmeneemmm ing of dnns. pdf
Ross Girshick. Fast R-CNN. In Proc. 2015 IEEE International Conference on Computer Vision (ICCV), pp. 1440-1448, 2015.
11

---

Published as a conference paper at ICLR 2020
Awni Y. Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, and Andrew Y. Ng. Deep speech: Scaling up end-to-end speech recognition, 2014. URL http://arxiv. wwwpariiiiorrgg/aa. wwt.:///gg/ab//////../ 5567 arXiv:1412.5567.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Dan Hendrycks and Kevin Gimpel. A Baseline for Detecting Misclassified and Out-of- Distribution Examples in Neural Networks. http://axxiv.org/aaapp 020.00/0//0100001133 2016.
arXiv:1610.02136
Geoffrey Hinton, Li Deng, Dong Yu, George Dahl, Abdel rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara Sainath, and Brian Kingsbury. Deep neural networks for acoustic modeling in speech recognition. Signal Processing Magazine, 2012.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. In NIPS Deep Learning and Representation Learning Workshop. 2015. URL http://arxi::
org/ org/abs/ 503. 02531
Diederik P. Kingma and Jimmy Ba. Adam: Method for Stochastic Optimization. In Proc. 3rd International Conference Learning Representations (ICLR), 2015.
Anoop Korattikara Balan, Vivek Rathod, Kevin Murphy, and Max Welling. Bayesian dark knowledge. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems 28, pp. 1438-34466, Curran Associates, Inc., 2015. URL
n nips cc/ wwwpcpppppp....eec.......eeeeeee / 5965 bayes an- r know sadgee.dd.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009. URL https
www cs. toronto. wdw//krrii learni lon features- Seartun--eattress 2009-TR, TR..FFF....... pdf.
B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. In Proc. Conference on Neural Information Processing Systems (NIPS), 2017.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(755)):366, 2015.
Zhizhong Li and Derek Hoiem. Reducing overconfident errors outside the known distribution, 2019.
URL httphhtts///peeeeiieeeee htt.://wwwwtttooommmmeeaaaaorroo
Wesley Maddox, Timur Garipov, Pavel Izmailov, Dmitry Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning. arXiv preprint arXiv: 2019.
A. Malinin, A. Ragni, M.J.F. Gales, and K.M. Knill. Incorporating Uncertainty into Deep Learning for Spoken Language Assessment. In Proc. 55th Annual Meeting of the Association for Computational Linguistics (ACL), 2017.
Andrey Malinin. Uncertainty Estimation in Deep Learning with application to Spoken Language Assessment. PhD thesis, University of Cambridge, 2019.
Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. In Advances in Neural Information Processing Systems, pp. 7047-7058, 2018.
Andrey Malinin and Mark Gales. Reverse kl-divergence training of prior networks: Improved uncertainty and adversarial robustness. Advances in Neural Information Processing Systems, 2019.
Tomas Mikolov, Martin Karafiát, Lukás Burget, Jan Cernocký, and Sanjeev Khudanpur. Recurrent Neural Network Based Language Model. In Proc. INTERSPEECH, 2010.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient Estimation of Word Representations in Vector Space, 2013a. URL httppp//aaxxx..oo//abb////////// 3781 arXiv:1301.378..
12

---

Published as a conference paper at ICLR 2020
Tomas Mikolov et al. Linguistic Regularities in Continuous Space Word Representations. In Proc. NAACL-HLT, 2013b.
Kevin P. Murphy. Machine Learning. The MIT Press, 2012.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep explo- ration via bootstrapped dqn. In D. D. Lee, M. Sugiyama, U. V. Luxburg. I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 4026 4034. Curran Associates, Inc., 2016. URL httt::/papers. nwp.. cttssper/ppprrr// 6501
dqp. dqn.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D Sculley, Sebastian Nowozin, Joshua V Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. arXiv preprint arwiv:
2019.
George Papamakarios. Distilling model knowledge. arXiv preprint arXiv:1510.024333 2015.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, 2017.
Joaquin Quiñonero-Camdee Candell.. Dataset Shift in Machine Learning. The MIT Press, 2009.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 815-823, 2015.
Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. Proc. International Conference on Learning Representations (ICLR), 2015.
L. Smith and Y. Gal. Understanding Mcasures Uncertainty for Adversarial Example Detection. In UAI, 2018.
Ruben Villegas, Jimei Yang, Yuliang Zou, Sungryull Sohn, Xunyu Lin, and Honglak Lee. Learning to Generate Long-ten term-term Future via Hicrarchical Prediction. In Proc. International Conference on Machine Learning (ICML), 2017.
Y. Wang, M. Wong, M. Gales, K. Knill, A. Ragni. Sequence teacher-student training of acoustic models for automatic free speaking language assessment. In 2018 IEEE Spoken Language Technology Workshop (SLT), pp. 994-1000, 2018. doi: 10.1 1091S1.T..2.1...400 8.6395577.
Max Welling and Yee Whye Teh. Bayesian Learning via Stochastic Gradient Langevin Dynamics. In Proc. International Conference on Machine Learning (ICML), 2011.
H. M. Wong and M. F. Gales. Multi-task ensembles with teacher-student training. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 84-90, 2017. doi: 10. 10910/SUUU200...800 2017..0.....0
Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. LSUN: construction of
large-scale image dataset using deep learning with humans in the loop, 2015. URL h // //rxivv.. / 03365 arXiv:1506.03365.
13

---

Published as a conference paper at ICLR 2020
APPENDIX A DATASETS, MODEL ARCHITECTURE AND TRAINING
Table 5: Description datasets used the experiments in terms number of images and classes.
Dataset
Train
Valid
Test
Classes
CIFAR-10
50000
10000
CIFAR-100
50000
10000
100
Tinylmagenet
100000
10000
200
LSUN (evaluation only)
10000
10
All models considered in this work were implemented in Pytorch (Paszke al. 2017) using variant of the VGG16 (Simonyan & Zisserman 2015) architecture for image classification. DNN and EnD models were trained using the negative log-likelihood loss of the labels and the mean ensemble predictions respectively. EnD" models were trained using the negative log-likelihood of the ensemble's 's output categorical distributions. All models 2015) optimizer, with 1-cycle learning rate policy and dropout regularization. For all ensembles, models were trained using different random seed initialization, and using different seeds for shuffling the data. In addition, data augmentation was applied via random left-right flips, random shifts to +44 pixels and random rotations by up to 15 degrees. Tables details the training configurations for all models. Furthermore, batch normalisation was used for both Ensemble Distillation and Ensemble Distribution Distillation, but not for Prior Networks.
trained using the Adam (Kingma Ba
Table 6: Training Configurations. 7u the initial learning rate, To is the initial temperature and 'Annealing' refers to whether a temperature annealing schedule was used. The batch size for all models was 128. Dropout rate is quoted in terms of probability of not dropping out unit.
Training Dataset
General
Distillation
Model
Epochs Cycle len. Dropout
To Annealing AUX data
DNN
45
30
0.5
EnD
0.7
No
EnD AUX
10
90
60
0.7
2.5
No
CIFAR-100
CIFAR-10
EnD2
90
60
0.7
10
Yes
EnD AUX
90
60
0.7
10
Yes
CIFAR-100
PN
5x 10
45
30
0.7
No
CIFAR-100
DNN
100
150
0.5
EnD
200
150
0.9
2.5
No
EnD AUX
10 10-3
200
150
0.9
2.5
No
CIFAR-10
CIFAR-100
EnD2
200
150
0.9
10
Yes
EnD
200
150
0.9
10
Yes
CIFAR-10
PN
5x 10
100
70
0.7
No
CIFAR-10
DNN
10
100
70
0.5
EnD
5 10
200
150
0.8
2.5
No
EnD AUX
5x
200
150
0.8
2.5
No
CIFAR-10
TinyImageNet
EnD
5x 10
200
150
0.8
10
Yes
E AUX
5x
200
150
0.8
10
Yes
CIFAR-10
PN
5x
100
70
0.7
No
CIFAR-10
To create the transfer D ensembles were evaluated the unaugmented CIFAR-10 and CIFAR-100 training examples. During distillation (both EnD and EnD?). models were trained on the augmented examples with the ensemble predictions the corresponding unaugmented inputs.
14

---

Published as a conference paper at ICLR 2020
A.1 NUMERICAL ISSUES WITH MAXIMUM LIKELIHOOD DIRICHLET TRAINING
As shown in equation & ensemble distribution distillation into prior network via likelihood maximi- sation is equivalent to minimising the loss function:
N Σ ( Dans) = N
K Σ In maa
M Σ m=
K
In
Σ M
(a
(1I)
If, due to numerical precision of the implementation, one of the ensemble member sample terms (im)
Te gets rounded to 0.0, the log term In me ((im) in the above equation cannot be computed. To avoid
this, we apply a small amount of central smoothing to the ensemble predictions:
(im) Te smootheed = (1 -1-30 T. (ém)
+
(12)
K
Where the smoothing parameter parameter y has been set to 1 x 10-4 for all EnD² experiments. Note that after applying central smoothing, each ensemble member's predictions still sum up to 1.0.
A.2 TEMPERATURE ANNEALING SCHEDULE
fixed temperature of 2.5 was used for Ensemble Distillation as recommended in (Hinton al. 2015), and was found to yield the best classification performance out of {1, 2.2, 5, 10). Temperature annealing resulted in worse classification performance for Ensemble Distillation, and hence was not used in the experiments. For the temperature annealing schedule, the temperature was kept fixed to initial temperature To for the first half-cycle. For the second half-eycle we linearly decayed the initial temperature To down to 1.0. Then, the temperature was kept constant 1.0 for the remainder of the epochs. For Ensemble Distribution Distillation, we found that an initial temperature of 10 performed best out of {5. 10, 20). The choice of the particular annealing schedule used has not been tested extensively, and is possible that other schedules that lead to faster training exist. However, must point that it was necessary use temperature scaling to train models the CIFAR-10, datasets.
CIFAR- and
APPENDIX B ASSESSING MISCLASSIFICATTIO DETECTION PERFORMANCE
In this work measures of uncertainty are used for two practical applications of uncertainty mis- classification detection and out-of-distribution sample detection. Both can be seen as an outlier detection task based on measures of uncertainty, where misclassifications are one form of outlier and -iit----iitiiittionn inputs are another form of outlier. These tasks can be formulated as threshold- based binary classification (Hendrycks & Gimpel, 2016). Here, detector Ir(x) assigns the label I (uncertain prediction) if an uncertainty measure H ((x) is above threshold T, and label 0 (confident prediction) otherwise. This uncertainty measure be any of the measures discussed in sections and
[1, H(x) > T
(x)x)
(13)
0, H(x) <T
Given a set of true positive examples D, = {-/ ش and set of true negative examples D. = ()) No, (an ji=1 the performance of such a detection scheme can be evaluated at a particular threshold value T using the true positive rate (T) and the false positive rate (T)):
(1) Nr
N₂ = N
N₂
1
((T)
(1)
SO(T) = N j=1
2 (x(
(14)
The range of trade-off's between the true positive and the false positive rates can be visualized using a Srceivvr-Operattng Characteristic (ROC) and the quality of the possible trade-offs can be summarized using the under the ROC curve (AUROC) (Murphy 2012). If there are significantly more negatives than positives, however, this measure will over-estimate the performance of the model and yield high AUROC value (Murphy 2012). In this situation it is better to calculate the precision
15

---

Published as a conference paper at ICLR 2020
and recall of this detection scheme every threshold value and plot them against each other Precision-Recal (PR) curve (Muphhy 2012). The recall R(T) is equal to the true positive rate ty(T), while precision measures the number of true positives among all samples labelled positive:
Σ IT (app') ΣN IT( Σラ IT(A))
N Np
1
P(T)
R(T)
(T
(15)
The quality of the trade-offs can again be summarized via the area under the PR curve (AUPR). For both the ROC and the PR curves an ideal detection scheme will achieve AUC of 100%. A completely random detection scheme will have an AUROC of 50% and the AUPR will be the ratio of the number of positive examples to the total size of the dataset (positive and negative) (Murphy 2012). Thus, the recall is given by the error rate of the classifier. This makes it difficult to compare different models with different base error rates, as AUPR can increase both due to better misclassification detection and worse eттоr rates.
Below we provide AUPR numbers for all models and datasets in order to illustrate how it can challenging to compare models using this metric. The difference in AUPR between models within dataset typically similar to the difference in classification performance. Thus, is challenging to assesses whether higher AUPR is due to better misclassification detection or higher error rate. Additionally, this table illustrate that confidence of the prediction a better measure of uncertainty than entropy for this task. At the same, knowledge uncertainty yield much worse misclassification detection performance. These results are consistent with (Malinin & Gales 2018 Malinin 2019).
In this work consider alternative to using AUPR to assess misclassification detection per- formance proposed in (Malinin, 2019). Consider the task of misclassification detection - ideally would like to detect all of the inputs which the model has misclassified based on measure of
Table 7: Mean Misclassification detection using AUPR using different measures of uncertainty for C10/C100/TIM across three models 2:
Total Uncertainty Confidence
Knowledge Uncertainty
Dataset
Model
% Error
Entropy
DNN
48.2 +2.7
47.0 +3.4
8.0
ENS
43.9 MM
41.1 = NA.
36.8 = NA
6.2 : NA
EnD
44.6 +3.3
44.1 +3.9
37.6 +2.7
6.7 40.3
C10
EnD
46.8 +1/2
46.1 +1.0
43.6 20.9
7.3 40.2
EnD+AUX
44.5 4:2.0
43.8 +1.77
37.7 41.1
6.7 40.2
EnD EnD+ +AUX
46.5 13.9
44.3 13.5
39.4 43.0
6.9 40.2
PN-RKL
40.5 +3.0
38.5 +2.7
35.0 +2.3
7.5 40.3
DNN
69.8 +1.5
69.4 +1.4
30.4 +0.3
ENS
67.2 NA
64.0 NA
57.6
26.3
EnD
68.1 10.8
67.6 11.1
62.2 11.8
28.0 10.4
C100
EnD
68.3 11.6
66.8 +1.5
63.4 11.2
27.9 10.3
EnD +AUX
69.3 10.3
66.9 10.2
58.0 10.2
28.2 40.3
EnD2 +AUX
68.9 20.4
66.7 49.2
62.0 11.0
28.0 #0.5
PN-RKL.
60.1 +1.8
58.0 +1.8
52.8 12.2
28.0 +0.7
DNN
77.9 41.3
78.3 41.2
41.8
ENS
76.5 NA
74.8 NA
71.8 +: NA
36.6 ANA
EnD
76.6 +o.5
77.3 40.8
70.5 +1.1
38.3 +0.2
TIM
2
77.0 +1.0
76.4 40.9
74.8 +1.3
37.6 #0.2
EnD +AUX
76.9 +1.8
77.1 42.1
69.9 11.2
38.5 10.3
EnD +AUX
76.4 +1.7
75.8 41.6
74.3 42.1
37.3
PN-RKL
73.0 41.2
71.1 41.1
60.7 41.3
40.0 4:0.6
16

---

Published as a conference paper at ICLR 2020
uncertainty. Then, the model can either choose to not provide any prediction for these inputs, they can be passed over or 'rejected' to an oracle (ie: human) to obtain the correct prediction. The latter process be visualized using rejection curve depicted in figure where the predictions of the model are replaced with predictions provided by an oracle in some particular order based on estimates of uncertainty. If the estimates of uncertainty would be a straight line from base eтrоr rate to the lower right corner. However, if the estimates of uncertainty are 'perfect' and always bigger for misclassification than for correct classification, then they would produce the 'oracle' rejection curve. The 'oracle' curve will go down linearly to 0% classification error the percentage of rejected examples equal the number of misclassifications. A rejection curve produced by estimates of uncertainty which are not perfect, but still informative, will sit between the 'random' and 'oracle' curves.
'useless', then, in expectation, the rejection curve
Oracle
Oracle
Classification Error (%)
Uncertainty
Classification Error (%)
Uncertainty
Random
Random
0
20
40
60
80
100
20
40
60
80
100
Percentage of examples rejected to oracle
Percentage examples rejected to oracle
Shaded ARere-
(b) Shaded is ARuss
Figure 5: Prediction Rejection Curves
The quality of the rejection curve can be assessed by considering the ratio of the area between the "uncertainty' and 'random' curves AR (orange in figure and the area between the 'oracle' and 'random' curves ARore (blue in figure 5 This yields the prediction rejection area ratio PRR:
ARuus
PRR
(16)
ARore
rejection area ratio 1.0 indicates optimal rejection, a ratio of 0.0 indicates 'random' rejection. A negative rejection ratio indicates that the estimates of uncertainty are 'perverse' they are higher for accurate predictions than for misclassifications. An important property of this performance metric that it is independent of classification performance, unlike AUPR, and thus it is possible compare models with different base eтrоr rates. Note, that similar approaches to assessing misclassification detection were considered in (Lakshminarayanan al. 2017 Malinin al. 2017)
APPENDIX C APPROPRIATENESS OF DIRICHLET DISTRIBUTION
Section 5.1 explored the behaviour measures of uncertainty predicted by models trained on the CIFAR- 1O dataset. In this appendix we provide the same for models trained on CIFAR- 100 and TinylmageNNe. general, the trends detailed in 5.1 for these datasets well EnD2 consistently yield higher uncertainties than the original ensemble, likely as consequence of the limitations of the Dirichlet distribution.
In addition, we have also provided histograms of total uncertainty for EnD and EnD models trained on CIFAR-10, CIFAR-100 and TinyImageNet relative to the ensemble for both in-domain and OOD data. Figure shows that while all EnD models match the uncertainty of the ensemble on in-domain data for all datasets, EnD consistently under-estimates the uncertainty for OOD data. The only exception is EnD+ This generally agrees with the results from section where EnD Aux is able to almost match the ensemble's OOD detection performance.
CIFAR- 100, where it matches the ensemble's 's predictions well.
17

---

Published as a conference paper at ICLR 2020
003
0.55
Exsrdd
0.30
0.25
De
in bl
0.35
0.5
0.25
0.08
0.23
0.20
0.11
0.19
046
0.05
045
0.00
000
0.0
(a) C100 Total Uncert. ID
(b) C100 Data Uncert. - ID
(c) C100 Knowledge Uncert. - -OOD
0.11
0.14
0.12
0.13
0.12
000
0.4
010
000
0.44
0.44
0.00
040
0.00
000
(d) C100 Total Uncert. 00D
(c) C100 Data Uncert. OOD
(f) C100 Knowledge Uncert. 0OD
0.33
0.30
0.56
Emerde
bol
Entlac
0.00
bel
0.2
022
0.55
0.12
0.0
0.19
0.12
0.09
0.0
0.0
(g) TIM Total Uncert. ID
(h) TIM Data Uncert. ID
ii) TIM Knowledge Uncert. OOD
Exvent
0.11
0.170
0.190
0.12
01/05
0.000
040
0.0
064
0.4
0:20
002
060
0.000
0.00
(i) TIM Total Uncert. OOD
(k) TIM Data Uncert. - 00D
(1) TIM Knowledge Uncert. - 0OD
Figure 6: Histograms of measures of uncertainty derived from ensemble, EnD" and EnD on in-domain (ID) and test out-of-domall (OOD) data from CIFAR-100 and TinylmageNet.
APPENDIX D ABLATION STUDIES
In this paper we made several important design choices. Firstly, we chose to use rich ensembles of 100 models, in order to generate good estimate of the empirical distribution of the ensemble on the simplex and assess whether the Dirichlet distribution was flexible enough to model it. Secondly, we used a temperature annealing schedule with an initial temperature of 10. In this section we conduct two ablation studies. Firstly, we assess whether Ensemble Distribution Distillation is possible when using an ensemble of 20, 50 and 100 models. Secondly, we investigate a range of initial temperatures (1, 2, 10, and 20) for temperature annealing for the models trained on the CIFAR-10 10 and CIFAR-100 datasets. Here, only the EnD? x
x model is considered. Results are presented in All figures depict the mean across 3 models (for each configuration) + standard
figures 8 3
deviation.
Overall, the results show two important trends. Firstly, using using 20 models does better than using 5 models, but there are no conclusive gains for ensemble distribution distillation from using more than 20 models. This suggests that it is sufficient to use ensemble of fewer models for Ensemble Distribution Distillation with models which parameterize the Dirichlet Distribution. This is beneficial in terms computation and memory savings. It is possible, however, that if more flexible distribution
18

---

Published as a conference paper at ICLR 2020
010
0.4
Saard
0.33
0.2
0.2
0.0
0.11
0.2
0.1
006
045
00
000
0.0
(a) C10 Total Uncertainty ID
(b) C100 Total Uncertainty - ID
(c) TIM Total Uncertainty ID
0.12
0.12
Emeedile
Eourde
41B
0.08
066
0.4
0.0
040
0.0
010
0.09
(d) C10 Total Uncertainty O0D
(e) C100 Total Uncertainty 00D
(1) TIM Total Uncertainty OOD
Figure 7: Histograms of measures of total uncertainty derived from ensemble EnD, and EnD +Aux on in-domain (ID) and test out-of-domate (OOD) data.
is used, such as a Mixture of Dirichlets, then it might be possible to derive further gains from larger ensemble. The second trend that it necessary to use temperature of at least in order to successfully distrilutionddistlll the ensemble. Using initial temperatures of 10 and 20 did not result in any significant further increase in performance. These results show that the temperature annealing process is important for Ensemble Distribution Distillation to work well.
Ensemble
EnD
0.24
Negative Liklllllloodd
0.23
Enor
Ensemble
EnD
0.11
0.20
0.19
20
40
-00
100
20
40
60
80
100
Number Models
Number Models
(a) % Classification Error
(b) Negative Log-likelihood
24
Ensemble
Expeceed Calbration Eiror
22
87
EnD
20
Prediction Rejection
88.50
86 25
1.4
86 18.00
12
Ensemble
85.75
1.0
EnDaAux
85.50
08
20
Number Models
(c) Expected Calibration Error
(d) Prediction Rejection Ratio
Figure 8: CIFAR-10 Model Ablation
19

---

Published as a conference paper at ICLR 2020
29.5
Ensemble EnD
1.15
23.0
1.05
Ensemble
EnD
1.00
Negafive
27.0
0.95
28.5
0.00
Number Models
Number Models
(a) % Classification Error
(b) Negative Log-likelihoce
Ensemble EnDŽAux
Expected Calibraiin Enror
Prodiction Rejection Railo
74.5
74.0
73.5
73.0
72.5
Ensemble
EnDeAur
72.0
100
Number of Models
Number Models
(c) Expected Calibration Error
(d) Prediction Rejection Ratio
Figure 9: CIFAR-100 Model Ablation
95.0
95.0
AUC-ROC
82.5
AUC-ROC
82.5
90.0
90.0
Ensemble LSUN
Ensemble LSUN
Ensemble TIM
Ensemble
EnDiaux LSUN
EnDaaux LSUN
EnDaAux
EnDaAux TIM
80.0
80.0
40
60
100
60
80
100
Number of Models
Number of Models
(a) C10 Total Uncertainty
(b) C10 Knowledge Uncertainty
90
90
AUC-ROC
85
85
75
Ensemble LSUN
Ensemble LSUN
Ensemble TIM
Ensemble
70
EnD]aux
70
EnDlaux
65
EnDi
85
EnDi TIM
Number of Models
Number of Models
(c) C100 Total Uncertainty
(d) C100 Knowledge Uncertainty
Figure 10: CIFAR-IO and CIFAR-100 Model Ablation Uncertaintics
20

---

Published as a conference paper at ICLR 2020
1.2
35
1.0
Etror
20
15
10
50
10.0 Temperature
12.5
17.5
20.0
7.5
10.0
12.5
15.0
20.0
Temperature
(a) % Classification Error
(b) Negative log-likelihood
20.0
Expected Caltrrltion Eiror
17.5
80
15.0
12.5
75
10.0
Prediction
70
7.5
5.0
25
5.0
7.5
12.5
15.0
25
50
7.5
10.0
12.5
15.0
17.5
20.0
(c) Expected Calibration Error
(d) Prediction Rejection Ratio
Figure 11: CIFAR-10 Temperature Ablation
21

---

Published as a conference paper at ICLR 2020
% Error
40
1.8
35
Negalive
30
2.5
50
7.5
10.0
12.5
15.0
17.5
20.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
Temperature
Temperature
% Classification Error
Negative Log-likelihood
Expected Calbralion Error
72.5
70.0
67.
63.0
62.5
60.0
57.5
Temperature
Temperature
(c) Expected Calibration Error
(d) Prediction Rejection Ratio
Figure 12: CIFAR-100 Temperature Ablation
96
85
Prediction Rejecton Ratti
80
Prediction Rejection
85
Total Unoertainty- - LSUN
Total Uncertainty LSUN
Knowledge Uncertainty
Knowledge Uncertainty Total Uncertainty TIM
Total Uncertainty- TIM
Knowledge Uncertainty
80
Uncertainty
70
25
5.0
7.5
10.0
12.5
15.0
17.5
20.0
25
50
7.5
10.0
12.5
15.0
17.5
20.0
Temperature
Temperature
(a) CIFAR-10 Uncertainties
(b) CIFAR- 100 Uncertainties
Figure 13: CIFAR-10 10 and CIFAR-100 Temperature Ablation Uncertainties
22


Reviews:
[{"review_id": "rygXFaj3tB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n\n========= Post Rebuttal ========= \n\n\nI appreciate the authors' effort in addressing the raised issues. I think the revised paper has higher quality in that the additional ablation studies are very useful to understand the effectiveness of the method and the importance of the hyperparameters. It now also has higher clarity in that the presentation of the work is considerably more detailed in the last revision. The outstanding issue is that the final results are not consistent across datasets, baselines, and metrics which is concerning. But, I think, overall, the paper is pushing on an interesting line of research which is relevant and educational for ICLR audience. Th concerns on consistency and conclusiveness of the results can hopefully be addressed in a journal version of the work as suggested by the authors.\n\n========= Summary ========= \n \nDistilling an ensemble of deep networks into a single student network is a common approach to reduce the inference-time computational complexity. In those cases, the paper poses the question of whether it is useful for the student to capture the diversity of ensemble members’ predictions on top of the mean distribution (as in the standard distillation). \nThe main motivation is that this captured diversity will enable the student network to decompose the total (predictive) uncertainty into data (aleatoric/irreducible) and knowledge (epistemic/model) uncertainty components. \nIn order to distill the diversity, it proposes to use “prior networks [Malinin&Gales 2018]” to output parameters of a  Dirichlet distribution over the simplex of possible predictive categorical distributions.\nIt uses one toy dataset as well as 3 small image classification datasets to compare the performance of the proposed method (EnD^2) with a) standard ensemble distillation (EnD), b) ensemble of deep networks, as well as c) an individual deep network. The performance is measured in terms of classification accuracy, and the quality of the predictive uncertainty for in-distribution (ID) and out-of-distribution (OOD) samples.\nIt shows that for OOD sample detection, the proposed EnD^2 outperforms the standard EnD and bridges the gap to the full ensemble. The classification accuracy and ID uncertainty quality is similar to the standard EnD.\n \n \n========= Strengths and Weaknesses ========= \n \n+ the paper studies the interesting problem of decoupling data/knowledge uncertainty in the commonly-used framework of knowledge distillation. It proposes a simple solution to the problem. So, it is highly relevant for the field.\n+ Results in Table 3 suggest significant improvements over the standard distillation (EnD). \n+ Figure 3 is pedagogical and intuitive for the data vs knowledge uncertainty decomposition and the effectiveness of auxiliary dataset for this toy problem.\n \n(I) General concerns:\n- The technical novelty is limited to the combination of knowledge distillation and prior networks.\n- Two closely-related works are not cited:\n[Li&Hoiem, “Reducing Over-confident Errors outside the Known Distribution” 2019] uses auxiliary dataset for ensemble distillation.\n[Englesson&Azizpour, “Efficient Evaluation-Time Uncertainty Estimation by Improved Distillation”, 2019] addresses the problem of efficient uncertainty estimation using knowledge distillation.\n \n(II) Concerns regarding the experiments\n- As also found curious by the authors, it is concerning that the EnD_Aux results are so low for OOD detection. Both [Li&Hoiem 2019] and [Englesson&Azizpour 2019] suggest that the standard distillation can perform similarly to (or even outperform) the individual model and the ensemble using an auxiliary dataset [Li&Hoiem 2019] or teacher label for augmented samples [Englesson&Azizpour 2019]. It should be mentioned that the OOD dataset setup is different across these works.\n- Why only 100-ensembles are used for the real-world experiments? Does a “successful” training of the prior network require a large number of samples? This would be important since prior ensemble works (e.g. [Lakshminarayanan et al. 2017] ) hardly improve beyond 5-15 networks. Training 100 networks is very costly which would be an important limitation of this work in case it’s necessary. An ablation study on the number of networks in the ensemble is required for investigating this trade-off.\n- Hyperparameter optimization: Appendix A provides the final values for the hyperparameters of each method. However, it is not clear how these were optimized? Was grid search used? What was the range for all the hyperparameters? Is there a validation set or cross validation is used and in each case how is the split done? Is it exactly the same hyper-parameter optimization that is done for EnD and EnD^2? \n \n(III) Missing from experiments\n- The main goal/motivation of the paper is to be able to decompose the total uncertainty when distilling an ensemble into a single model. In that respect, richer and more experiments are required to evaluate this ability. Currently, the experiments are focused on showing that EnD^2 outperforms EnD. The toy dataset, qualitatively, evaluates the decomposition quality with interesting results but is not enough to make conclusions for real-world datasets. For instance, as an additional experiment, plots/statistics can be given on data vs knowledge uncertainty for ID vs OOD samples.\n- Prior network [Malinin&Gales 2018] in its standard form is an important single-network baseline that should be included for both ID and OOD experiments. \n- Temperature scaling is mentioned to be a vital part of the model. As such, it requires a thorough ablation study to see the results with or without it as well as when changing the temperature and the annealing factor. It should be studied from two aspects: accuracy and convergence failure (as mentioned by the paper)\n- P.7: “Prediction Rejection Ratio” (PRR) is a measure proposed in this work but it’s only defined in the appendix. That is the only metric that measures the quality of uncertainty for ID samples (Table 3). As such, I believe it’s important to define PRR in the main paper and also further include more standard metrics such as NLL, AUROC, AUPRcurve in table 3 so that some context is given to the newly-proposed PRR.\n- Along the same line, it seems NLL is consistently and “more significantly” worse for EnD^2 compared to EnD.\n \n(IV) Missing training and implementation details\n- More details should be provided for the training of the student prior network including it’s loss function given a dataset and Dirichlet distribution. This can be obtained from [Malinin&Gales 2018] but it’s important for this work to be self-contained.\n- the loss function in equation 8 has log(p(\\pi|x;\\theta)); is there any numerical issue regarding the Dirichlet distribution and/or the log? If so, how significant and what measures are accordingly taken? Could it be that the temperature scaling is more a way of alleviating these issues as opposed to the shared support of distributions for KL divergence?  \n- the details of the annealing algorithm is entirely missing.\n \n \n========= Final Decision ========= \n \nThe paper addresses a highly relevant problem in a simple (and potentially effective) way. This is great. However, there are several concerns as listed above which altogether makes me lean towards an initial “weak reject” rating. (II) and (III) are more central to this initial rating. I will carefully read the authors rebuttal as well as other reviewers’ comments before I finalize my rating.\n \n \n========= Minor points ========= \n \ngeneral:\n- P.2: “[...] limitation of ensembles is that the computational cost of training and inference [...] One solution is to distill [...]“ -> this only remedies the computational complexity of inference and in fact increases the training time.\n- P.3: [Malinin&Gales 2018] should be cited for equation 4 and the discussion around it.\n- P.5: “Ensemble Distillation (EnD) is able to recover the classification performance [...] with only very minor degradation in performance”. Table 1 does not show any degradation for EnD. It shows some degradation for EnD^2 when 100-ensemble is used.\n- P.7: “Note, that on C100 and TIM, EnD2 yields better classification performance than EnD”: almost all of the classification improvements are well within one-std. In the specific case of C100, it’s a stretch to call it a “better classification performance”.\n- P.7: how many runs are used to obtain the mean and std reported in table 3 and 4.\n- P.7: is the PRR in table 3 calculated using the total or knowledge uncertainty for ensemble and EnD2?\n\nTypo:\n- P.2: “. Consider an ensemble of models {P[…]” --> a closing parenthesis is missing.\n- P.3: “Each of the models P([...]” --> a closing parenthesis is missing.\n- P.3: “ the entropy of the expected and“ -> “the entropy of the expectation”\n- P.4: “=\\hat{p}(x,\\pi)” → “\\sim\\hat{p}(x,\\pi)”\n- P3-4.: hat and star seem to have been arbitrarily used for input x, parameters theta, pi and p with most of them undefined.\n- P.5: script MI is used for mutual information while in P.4 script I is used.\n- P.6: “may require the use additional training” -> use of additional\n- P.6: Results of EnD in table 2 does not match table 1.\n \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "r1xzMickcB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper notes that ensemble distillation (EnD) loses the distributional information from the original ensemble, which prevents users of the distilled model from being able to obtain estimates of its knowledge uncertainty. It proposes the challenge of distilling the ensemble in a way that preserves information about the ensemble distribution, so that knowledge uncertainty can still be extracted from the distilled model. It names this task \"Ensemble Distribution Distillation (EnD^2)\". It then proposes using Prior Networks (introduced in Malinin & Gales 2018) as a solution, and proceeds to evaluate it with a series of experiments -- first obtaining some intuition from spiral dataset, then more rigorously on benchmark image datasets (CIFAR10/100/TinyImageNet).\n\nFirst, it trains ensembles of 10 and 100 NNs on a spiral dataset, distills them using the regular approach (EnD) and Prior Networks (EnD^2), compares their performance, and notes that the Prior Networks approach has comparable performance. Next, it visualizes over the input space of the spiral dataset the total uncertainty, data uncertainty, and knowledge uncertainty estimates, which are extracted directly from the original ensemble and from the EnD^2 distilled model (Figure 3). It notes that while the original ensemble is able to correctly estimate the knowledge uncertainty in regions that are far away from the training distribution, the EnD^2 model fails at this task (Figure 3f). It then proposes to augment the training set with out-of-distribution data, and demonstrates that this improves the estimation of knowledge uncertainty (Figure 3i). It also proposes a new metric for evaluating the Prediction Rejection Ratio (PRR), uses it to compare how the EnD^2 model compares to the original ensemble and the EnD model. Finally, it demonstrates using a series of benchmark image classification tasks that the EnD^2 model is able to identify out-of-distribution samples with comparable performance to the original ensemble.\n\nDecision: Leaning-to-Accept. Distillation is a well-established technique, and adapting it so that the same NN can perform both predictions and knowledge uncertainty estimates is impactful. This work proposes using Prior Networks as a way to distill ensembles of NNs in a way that preserves the knowledge uncertainty estimates, and evaluated this claim with a sequence of experiments. This work also proposes a new evaluation metric (Prediction Rejection Ratio), and can be used to evaluate future models that are able to simultaneously perform prediction and knowledge uncertainty estimation. However, the way that the paper is organized around the proposal of \"Ensemble Distribution Distillation\" as a novel machine learning task does not seem very well motivated, as the distribution was solely used to provide uncertainty estimates.\n\nStrengths:\n- The visualizations in Figure 3 helped to provide intuition to the reader.\n- Experiments have a clear logical flow. Spiral experiments provide intuition, motivate out-of-distribution data augmentation, then image data experiments provide evidence for the applicability of the method. \n- Motivates and explains the newly proposed evaluation metric (prediction rejection ratio) in the appendix.\n- The out-of-distribution detection experiments are quite comprehensive.\n- The training procedures are clearly detailed in the appendix.\n- Investigates the appropriateness of the Dirichlet distribution in the appendix.\n\nWeaknesses:\n- The proposal of the novel machine learning task of \"Ensemble Distribution Distillation\" does not seem very well motivated. In this paper, the distribution distillation was solely used to obtain a knowledge uncertainty estimation. Besides that, what else would the distribution be used for? It was also initially unclear to me what this paper contributes on top of \"Predictive Uncertainty Estimation via Prior Networks (Malinin & Gales, 2018)\". A suggestion is to rewrite the summary of contributions to emphasize that the use of Prior Networks to produce a single model that can both perform predictions and provide uncertainty estimates as an extension of ensemble distillation is novel, and that a more comprehensive set of experiments on more difficult image datasets were done in this paper.\n\nMinor comments:\n- page 2, expression right before equation 2, and in the first sentence on page 3 is missing closing parentheses.\n- page 3, figure 1. It wasn't initially obvious to me that the triangle represents the simplex of the softmax output, and each black dot represents the output of one model of the ensemble.\n- page 4, equation 9. Add some space to the right of the equality sign.\n- Use backticks`   instead of single quotation mark ' to open quotation marks in LaTeX.\n\n\nRebuttal response:\nI acknowledge the authors' point about the importance of EnDD in addition to knowledge uncertainty estimation, and maintain my rating.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "BkxPqUmUtH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nEnsembles of probabilistic models (e.g. for classification) provide measures of both data uncertainty (i.e. how uncertain each model is on average) and model uncertainty (i.e. how much the models disagree in their predictions). When naively distilling an ensemble to a single model, the ability to decompose total uncertainty into data uncertainty and model uncertainty is lost. This paper describes a method for ensemble distillation that retains both types of uncertainty in the case of classification. The idea is to train a prior network (i.e. a conditional Dirichlet distribution) to model the distribution of categorical probabilities within the ensemble. Both total uncertainty and data uncertainty can then be computed analytically from the prior network.\n\nPros:\n\nThe paper considers an interesting problem, and makes a clear contribution towards addressing it. The paper motivates the problem well, and explains the contribution and the method's limitations clearly.\n\nThe proposed method is simple, but well motivated, sound, and well explained.\n\nThe paper is very well written and easy to read. I particularly appreciated the toy experiment in section 4 and the visualization in figure 3, which showcase clearly what the method does.\n\nThe experiments are thorough, and the results are discussed fairly and objectively.\n\nCons:\n\nThe scope of the paper and the method is limited to the problem of probabilistic classification. However, one could have a more general ensemble of conditional or unconditional distributions. The method could in principle be applied to this setting, by having a hypernetwork learn the distribution of the distributional parameters of the models in the ensemble (the method presented in the paper is a special case of this, where the hypernetwork is a prior network and the distributional parameters are categorical probabilities). However, it's not clear that the method would scale to models with arbitrary distributional parameters. I would suggest that the paper make it clear from the beginning that the scope is probabilistic classification, and at the end discuss the extent to which this is a limitation, and how the method could potentially be extended to other kinds of ensembles.\n\nThe prior network used is essentially a conditional Dirichlet distribution, which (as the paper clearly acknowledges) may not always be flexible enough. A more flexible prior network could be a mixture of Dirichlet distributions, where the mixing coefficients and the parameters of each mixture component would be functions of the input, similarly to mixture density networks (http://publications.aston.ac.uk/id/eprint/373/) but with Dirichets instead of Gaussians. I believe that equations (9) and (10) would still be tractable in that case, as it's tractable to compute expectations under mixtures if the expectations under mixture components are tractable.\n\nOne limitation of the approach is that the prior network may not give accurate predictions for inputs it hasn't been trained on (as the paper discusses and section 4 clearly demonstrates). It's not clear how this problem can be overcome in general, and further research may be needed in that direction.\n\nSome of the results in Table 4 are puzzling (as the paper also acknowledges). In particular, the EnD model should be able to retain the performance of the ensemble when using total uncertainty but it doesn't. Also, using knowledge uncertainty doesn't always seem to be better than using total uncertainty, which to some extent defeats the purpose of the method (at least in this particular example). It would be good to investigate further these results. In any case, I appreciate that the paper acknowledges these results, but avoids unjustified speculation about what may be causing them.\n\nDecision:\n\nOverall, this is good work and I'm happy to recommend acceptance. There are some limitations to the method, but these can be seen as motivation for future work.\n\nMinor suggestions for improvement:\n\nThis older work is relevant and could be cited (but there is no obligation to do so).\nOn compressing ensembles:\n- Model compression, https://dl.acm.org/citation.cfm?id=1150464\n- Compact approximations to Bayesian predictive distributions, https://dl.acm.org/citation.cfm?id=1102457\n- Distilling model knowledge, https://arxiv.org/abs/1510.02437\nOn information-theoretic measures for decomposing total uncertainty as in eq. (4):\n- Decomposition of uncertainty in Bayesian deep learning for efficient and risk-sensitive learning, https://arxiv.org/abs/1710.07283\n\n\"[Knowledge uncertainty] arises when the test input comes from a different distribution than the one that generated the training data\"\nThis is only one way knowledge uncertainty can arise, it could also arise when the test input comes from the same distribution that generated training data, but there aren't enough training data available.\n\nSome parentheses are dangling at the bottom of page 2, top of page 3 and middle of page 4.\n\nIn figure 1, it'd be good to make clear in the caption that the figure is illustrating the simplex of categorical probabilities.\n\nThe last paragraph of section 2 uses the term \"posterior\" repeatedly to refer to P(y|x, \\theta) which is confusing. I would call P(y|x, \\theta) the \"model prediction\" or something like that.\n\nEq. (5) should be without the negative sign I think.\n\nIn section 3, I would use a different symbol (e.g. \\phi) to denote the parameters of the prior network, to clearly distinguish them from the parameters of the models in the ensemble.\n\n\"Optimization of the KL-divergence between distributions with limited non-zero common support is particularly difficult\"\nSome more evidence in support of this claim is needed I think; either explain why or provide a reference that does.\n\nIn section 4, the text says that EnD has \"a minor degradation in performance\" but table 1 seems to show otherwise. Also, the results of EnD in table 1 and table 2 are different, which makes me think there may be a typo somewhere.\n\nMaking the references have consistent format and correct capitalization (e.g. DNNs, Bayesian) would make the paper look even more polished.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper investigates how to distill an ensemble effectively (using a prior network) in order to reap the benefits of uncertainty estimation provided by ensembling (in addition to the accuracy gains provided by ensembling). 

Overall, the paper is nicely written, and makes a valuable contribution. The authors also addressed most of the initial concerns raised by the reviewers. I recommend the paper for acceptance, and encourage the authors to take into account the reviewer feedback when preparing the final version.
Decision: Accept (Poster)