Self-Distribution Distillation: Efficient Uncertainty Estimation
Authors: Yassir Fathullah
Mark Gales
Venue: ICLR 2022 Submitted (2022)
Keywords: distillation
self-distillation
distribution distillation
uncertainty
robustness

Abstract: Deep learning is increasingly being applied in safety-critical domains. For these scenarios it is important to know the level of uncertainty in a model’s prediction to ensure that appropriate decisions are made by a system. Deep ensembles are the de-facto standard approach to obtaining various measures of uncertainty. However, ensembles normally significantly increase the resources required in both the training and deployment phases. Approaches have been developed that typically address the costs in one of these phases. In this work we propose a novel training approach, self-distribution distillation (S2D), which is able to efficiently, both in time and memory, train a single model that can estimate uncertainties in an integrated training phase. Furthermore it is possible to build ensembles of these models and apply ensemble distillation approaches, hierarchical distribution distillation, in cases where one is less limited by computational resources in the training phase, but still requires efficiency in the deployment phase. Experiments on CIFAR-100 showed that S2D models outperformed standard models and Monte-Carlo dropout. Additional out-of-distribution detection experiments on LSUN, Tiny ImageNet, SVHN showed that even a standard deep ensemble can be outperformed using S2D based ensembles and novel distilled models.


Under review as a conference paper at ICLR 2022
SELF-DISTRIBUTION DISTILLATION: EFFICIENT UN- CERTAINTY ESTIMATION
Anonymous authors Paper under double-blind review
ABSTRACT
Deep learning is increasingly being applied in cafeta-ritical domains. For these scenarios it important to know the level of uncertainty in model's predic- tion to ensure that appropriate decisions are made by a system. Deep ensembles are the de-facto standard approach to obtaining various measures of uncertainty. However, ensembles normally significantly increase the resources required in both the training and deployment phases. Approaches have been developed that typi- cally address the costs in of these phases. In this work we propose a novel training approach, self-distribution distillation (S2D), which is able to efficiently, both in time and memory, train single model that can estimate uncertainties in an integrated training phase. Furthermore is possible to build ensembles of these models and apply ensemble distillation approaches, hierarchical distribu- tion distillation, in cases where one is less limited by computational resources in the training phase, but still requires efficiency the deployment phase. Exper- iments on CIFAR-100 showed that S2D) models outperformed standard models and Monte-Carlo dropout. Additional -uito--distribution detection experiments on LSUN, Tiny ImageNet, SVHN showed that even standard deep ensemble can outperformed using S2D based ensembles and novel distilled models.
INTRODUCTION
Neural networks (NNs) have enjoyed much success recent years achieving state-of-the-ard per- formance on large number of tasks within domains such as natural language processing [Vawwani ct al. 2017). speech recognition (Hinton et al. 2012) and computer vision (Krizhevsky al. 2012) Unfortunately, despite the prediction performance of NNs they are known to yield poor estimates of the uncertainties in their predictions-in knowing what they do not know (Lakshminarayanan et al.)
2017 Guo al. 2017). With the increasing application of neural network based systems in per- forming safety-critica tasks such as biometric identification (Schroff et al. 2015), medical diagnosis (De Fauw al. 2018) fully autonomous driving (Kendall al. 2019). it becomes increasingly important to be able to robustly estimate the uncertainty in a model's 's prediction. By having access to accurate measures of prediction uncertainty, system can act in more safe and informed manner.
Ensemble methods, and related schemes, have become the standard approach for uncertainty esti- mation. Lakshminarayanan et (2017) proposed generating a deep (random-seed) ensemble by training each member model with different initialisation and stochastic gradient descent (SGD). Not only does this ensemble perform significantly better than standard trained NN, it also displays better predictive uncertainty estimates. Although simple to implement, training and deploying an ensemble results in linear increase in the computational cost. Alternatively Gal & Ghahramani (016) introduced the Monte Carlo (dropout) ensemble (MC ensemble) which at test time estimates predictive uncertainty by sampling members of an ensemble using dropout. Though this approach generally does not perform as well deep ensemble (given the same computational power and ne- glecting memory) (Lakshminarayanan et al. 2017). it is significantly cheaper to train as it integrates the ensemble generation method into training.
Despite ensemble generation methods being computationally more expensive, they have an impor- tant ability to decompose predictive (total) uncertainty into data and knowledge uncertainty (De- peweg et al. 2018 Gal & Ghahramani 2016). Knowledge or epistemic uncertainty refers to the lack of knowledge ignorance about the most optimal choice of model (parameters) (Hüllermeier

---

Under review as a conference paper at ICLR 2022
As additional data is collected, the uncertainty in model parameters should decrease. This form of uncertainty becomes important whenever the model is tasked with mak- ing predictions for out-of-distribution data-points. For in-distribution inputs, it is expected that the trained model can return reliable predictions. On the other hand data or aleatoric uncertainty, repre- sents inherent noise in the data being modelled, for example from overlapping classes. Even if more data is collected, this type of noise is inherent to the process and cannot be avoided or reduced (Ma-) linin Gales 2018 Gal & Ghahramani] 2016 Ovadia al., 2019) The ability to decompose and distinguish between these sources of uncertainty is important as it allows the cause of uncertainty in the prediction be known and how it should be used in downstream tasks (Houlsby et al. 2
& Waegeman 2021
Kirsch et al. 2019).
Summary of contributions: In this work we make two important contributions to NN classifier train- ing and uncertainty prediction. First we introduce self-distribution distillation (S2D), new general training approach that an integrated, simultaneous fashion, trains a teacher ensemble and distribu- tion distils the knowledge to student. This integrated training allows the user to bypass training separate expensive teacher ensemble while distribution distillation (Malinin et al. 2020) allows the student to capture the diversity and model distribution tionally, distribution distillation would give the student the ability to estimate both data and knowl- edge uncertainty in single forward pass unlike standard NNs which inherently can not decompose predictive uncertainty, and unlike ensemble methods which can not perform the decomposition in single pass. Second, we train an ensemble of these newly introduced models and investigate dif- ferent distribution distillation techniques giving rise to hierarchical distributions over predictions for uncertainty. This approach is useful when there are no, or few, computational constraints in the training phase but still require robust uncertainties and efficiency in the deployment stage.
ensemble member predictions. Addi-
2 BACKGROUND AND RELATED WORK
This section describes two techniques for uncertainty estimation. First, ensemble methods for pre- dictive uncertainty estimation will viewed from a Bayesian viewpoint. Second, a specific form of distillation for efficient uncertainty estimation will be discussed.
2.1 ENSEMBLE METHODS
From a Bayesian perspective the parameters, 0, of a neural net are treated as random variables with some prior distribution p(0). Together with the training data D, this allows the posterior distribution p(0 (D) to be derived. To obtain the predictive distribution over all classes y (for some input x²) marginalisation over 0 is required:
Plyl=", D) Eprorp) |(ulkx 00] 0
Since finding the true posterior is intractable variational approximation p D ~ q()) is made
(Jordan al., 1999] Blundell et al., 2015 Graves 2011] Maddox al. 2019). Furthermore, marginalising over all weight values remains intractable leading to a sampling ensemble, approxi- mation method (Gal Ghahramani] 2016 Lakshminarayanan al. (017):
P(y|=" D) M
1
Σ P(yla", e(()), ((m) (())
Here, an ensemble generation method is required to obtain the predictive distribution and uncer- tainty. Two previously mentioned approaches to generate an ensemble are deep (naive) random- seed and MC-dropout ensemblef] Deep ensembles are based on training M models on the same data but with different initialisations leading to functionally different solutions. On the other hand, a MC-dropout ensemble explicitly defines variational approximation through the of dropout (Srivastava et al.) 2014) (used during training), allowing for straightforward sampling of model parameters. Another approach, SWA-Gaussian (Maddox et al.) 2019), finds Gaussian approximation based on the first two moments of stochastic gradient descent iterates. Unlike the deep ensemble approach, and similar MC-dropout, this method allows for simple and efficient sampling but suffers from higher memory consumption. Even a diagonal Gaussian approximation
In-depth comparisons of ensemble methods were conducted in Ovadia al. 2019 Ashukha al. 2020
2

---

Under review as a conference paper at ICLR 2022
requires twice the memory of standard network. There also exists alternative memory and/or com- pute efficient ensemble approaches such as BatchEnsembles (Wen et al, 2020) and MIMO (Havasi et al., 2021). While the former approach is parameter efficient it requires multiple forward passes at test time similar to MC ensembles. The latter avoids this issue by the use of independent subnet- works within single deep model leading to both efficient training and testing in terms of compu- tational and memory costs. However, MIMO still requires multiple output heads at test time; this is an issue when scaling large scale classification tasks where the output layer has tens hundreds of thousands classes.
Given an ensemble, the goal is to estimate and decompose the predictive uncertainty. First, the en- tropy of the predictive distribution P(y 20 D) can be seen as measure of total uncertainty. Second,
this can be decomposed (Depeweg et al. 2018 Kendall & Gal 2017 as:
H ((((vl2', ,D]
I I[y. IV..0x".D) +
Ep(a (D) [MP(V x00 00]]
(3)
Total Uncertainty
Knowledge Uncertointy
Data Uncertainty
where I is mutual information and H represents entropy. This specific decomposition allows total uncertainty to be decomposed into separate estimates of knowledge and data uncertainty. Further- more, the conditional mutual information can be rephrased as:
I yy 0 ; DD P)p(r))) [L(()() 0) P(yla>", D))
For an in-domain sample x" the mutual information should be low as appropriately trained models P(y x=" _z=,0)) 0) should be close the predictive distribution. High predictive uncertainty will only oceur if the input exists in region of high data uncertainty, for example when an input has significant class overlap. When the input x" is cat-of-distrrbution of the training data, one should expect inconsistent, different, predictions P(yla=" (0) leading to a much higher knowledge uncertainty estimate.
2.2 ENSEMBLE (DISTRIBUTION) DISTILLATION
Ensemble methods have generally shown superior performance on a range of tasks but suffer from
being computationally expensive. To tackle this issue, a technique called knowledge distillation
(KD) and its variants were developed for transferring the knowledge of an ensemble (teacher) into a
single (student) model while maintaining good performance (Hinton et al. 2014| Kim & Rush [016) Guo et al. 2020). This is generally achieved by minimising the KI-diiergence between the student prediction and the predictive distribution of the teacher ensemble. In essence, KD trains new student model to predict the average prediction of its teacher model. However from the perspective of uncertainty estimation the student model no longer has any information about the diversity of various ensemble member predictions; it was only trained to model the average prediction. Hence, it is no longer possible to decompose the total uncertainty into different sources, only the total uncertainty can be obtained from the student. To tackle this issue ensemble distribution distillation (En2D) was developed (Malinin et al. 2020).
Let π signify a categorical distribution, that is n. = P(y w.(T). The goal is to directly model the space of categorical predictions (T(m) = ((m)) made by the ensemble. In work developed by |Malinin et al. (2020) this is done by letting a student model (with weights Φ) predict the parameters of Dirichlet, which is a continuous distribution over categorical distributions:
(5) The key idea in this concept is that we are not directly interested in the posterior p(0(D) but how predictions π for particular inputs behave when induced by this posterior. Therefore, it is possible to replace p(O(D) with a trained distribution p( T [x",, Φ). It is now necessary to train the student given the information from the teacher which is straightforwardly done using negative log-likelihood:
P(T|2= x +) = a)) a = ())
M
In (( = M m==1
Σ
A decomposable estimate of total uncertainty is then possible by using conditional mutual informa- tion between the class and prediction π instead of 0 Malinin & Gales (2018):
H(P(yl=* Φ) I[y π T|="
+ Data Unceriainty
Eppoiae .)) ((P(()T))
Total Uncertuinty
Knowledge Uncertainty
3

---

Under review as a conference paper at ICLR 2022
This decomposition has similar interpretation to eq. (3). Using Dirichlet model, these uncertain- ties can found using a single forward pass, achieving much higher level of efficiency compared to ensemble. Assuming this distillation technique is successful, the distribution distilled stu- dent should be able to closely emulate the ensemble and be able to estimate similar high quality uncertainties both IDD and OOD data.
However, ensemble distribution distillation is only applicable and useful when the ensemble mem- bers are not overconfident and display diversity in their predictions-there no need in capturing diversity when there is none. It is often the case that, for example, convolutional neural networks are paeaeearamttriseed, display severe overconfidence and can essentially achieve perfect training accuracy restricting the effectiveness of this method (Guo et al.) 2017; Seo et al. [2019] Ryabinin al., 202T). Furthermore, this method can only be used when an ensemble available, leading to high training cost.
SELF- DILL-DISTRIBUTIN DISTILLATION
In this section propose self-distribution distillation (S2D) for efficient training and uncertainty estimation, bypassing the need for a separate teacher ensemble. This combines:
parameter sharing: allowing the teacher and student to share a common feature extraction base would accelerate training significantly, each will branch off and have their head; stochastic regularisation: the teacher can generate multiple predictions efficiently by for- ward propagating input through its head (with a stochastic regulariser) several times, emulating the behaviour an ensemble;
distribution distillation: while the teacher branch is trained on cross-entropy, student taught to predict distribution over teacher predictions capturing the diversity compactly,
This process is summarised in Fig. 1 This approach can take many specific forms with regards to
Teacher Branch
XM
Feature Extraction
v
Student Branch
Figure 1: General structure of a self-distribution distilled model. M stochastic teacher branch forward
propagations are trained on cross-entropy and simultaneously distribution distilled to the student.
the type of feature extraction module, stochastic regulariser, teacher branch and student modelling choice. For example, the teacher could entail a much larger branch capturing complex patterns the data, while the student could consist of smaller branch used for compressing teacher knowledge into more efficient form, at test time. On the other end, training efficiency can be achieved by forcing the teacher and student share the same branch parameters.
In this work choose particular model configuration that is highly efficient, shown in Fig, The main functional difference between the teacher and the student branches is the use of logit values, 2: for the teacher branch probability is predicted; whereas the student uses the logits for Dirichlet distribution. Furthermore the teacher uses stochastic regularisation techniques (SRTs) in generating multiple teacher predictions, analogous to an ensemble. In this work multiplicative Gaussian noise (Gaussian dropout) with unit mean and uniformly random standard deviation is used. This form
rppited
(p/m)) j...
Sof tnax
(17 (m))))
M
Festure Extraction
Linesar Lager
Exp
Figure 2: Dirichlet S2D model during training. Only the black part of the network is retained during prediction time, matching the behaviour of standard model.
4

---

Under review as a conference paper at ICLR 2022
was chosen due simplicity of sampling and possible ensemble diversity. There is wide range of other choices regarding what SRTs to use, from Bernoulli dropout, additive Gaussian noise to deciding which teacher branch layers this should be introduced. Furthermore, since the Dirichlet distribution has bounded ability to represent diverse ensemble predictions (Malinin et al. 2020), simply generating multiple teacher prediction by propagating through the last layer will not limit the ability of the model. To further improve the memory efficiency of the model, a single final linear layer shared by both student and teacher branches used. This parameter sharing makes the S2D model efficient even when the number of classes is large, and does not use any more parameters compared to a standard model. Note any NN classifier can cast into a self-distribution distillation format by inserting stochasticity prior to the final linear layer and can easily be combined with many other approaches such as MIMO (Havasi et al. 2021 and SWAG (Maddox et al. 2019).
This choice of integrating ensemble teacher training and distribution distillation into a single entity utilising parameter tying also serves as a regulariser (optimising two objectives using the same set of weights) and allows for inexpensive training. The only added training cost is from multiple forward passes through the final linear layer, a process which can be parallelised. Additionally, the restricted form of Fig. brings some numerical stability. As noted by Malinin et al. (2020), optimising student to predict a Dirichlet distribution can be unstable when there is a lack of common support between prediction and extremely sharp teacher outputs. However, note that teacher predictions are closely related to the expected student prediction:
Epir(Taa))
Softmax(z), )(/) Softmax(z(")) (")),, E (m)]
leading to increased common support. Additionally, stochasticity in the teacher forces the outputs to have some diversity, mildly limiting overconfidence. Now train the teacher branch using cross- entropy, and simultaneously, use the teacher predictions to train the student branch. Let the weights of this model be denoted by & and say we have some input-target pair (ær, y). The teacher loss (for a single sample) is then:
1 M Σ m=1
(Ø) = M
E8(y, In (m)
where 5 the indicator function. The student branch could be trained using log-likelihood as eq. (6) but it has been found that this approach could be unstable (Fathullah et al. [202] Ryabinin et al. 202II Instead we use the teacher categorical predictions in estimating proxy teacher Dirichlet a using maximum log- likeliikeihood. The resulting student loss is KL-divergence based:
Lat (Φ) = KL (ir((T; a) || Dir((::))
ā = arg max > (m)
(10)
& m
The proxy Dirichlet is estimated using a numerical approach developed by Minka 2000). The overall training loss becomes L(Φ) = Lex(0) + (Ø) with a small constant Jr.
Deep learning models often overfit on training data leading to less informative outputs. To alleviate these issues we integrate temperature scaling in the student branch loss. While training the teacher branch predictions on cross-entropy we temperature scale the same predictions and use the resulting ones in estimating proxy teacher Dirichlet. The student branch will repeatedly be taught to predict a smoother/wider Dirichlet distribution, while the teacher branch's objective to maximise the probability of the correct class resulting in middle ground.
4 SELF-DDSTTIBUTION DISTILLED ENSEMBLE APPROACHES
computational resources during the training phase are not constrained it would open up the pos- sibility for self-distribution distilled ensembles and various hierarchical distillation approaches of such models. First it can be noted that the ensemble generation methods mentioned in previous sections can easily be used with the S2D models in the previous section. The predictive distribution of such an ensemble would take the following form:
Ply = we|w" D) /ʃ P(y we ))p(T( ()p(p(D)dddd M
M
(m)
Σ
(11)
(m)
a
Furthermore, an ensemble of Dirichlet models can be used to estimate similar uncertainty measures as previously described:
5

---

Under review as a conference paper at ICLR 2022
H ( D)} I π lrl + Ep(elD) [ [H((())))]
(12)
This is a generalisation of eq. ( since specific weights & have been replaced with conditioning on the dataset D. Computing these uncertainties requires only a few modifications compared the standard ensemble in eq. (3).
Next, the most natural step transfer the knowledge of S2D (Dirichlet) ensemble into single model. A choice needs to be made regarding the hierarchy of student modelling: should the student predict a categorica Dirichlet, or distribution over Dirichlets herebyy given the family name hierarchical distribution distillation (H2D). Initially we start by training student model to predict single Dirichlet identical eq. (5). However, since the S2D ensemble provides, for an input x", of Dirichlets {a (m) f(x"; ) a modified distillation criterion is needed:
Σ KL (I(DIT(T: ((Ø) M m=1
1
Dir(T;
- f=x";
(13)
This KL-ddveereence based loss also allows the reverse KL criterion to be used (Malinin & Gales, 2019) if desired. One criticism of this form of model, Dirichlet H2D (H2D-Dir), is that the diversity across ensemble members is lost, similar to the drawback in standard distillation. Therefore, we seek distribution over Dirichlets to capture this higher level of diversity.
To model the space of Dirichlets we need to define a distribution over the parameters. Here we are faced with a choice: (I) model the parameters a E IR directly (restricted to the non-negative real space) or (2) apply transformation to simplify the modelling. Here a logarithmic transformation & E R applied and a simple distribution over the Dirichlet parameters, a diagonal Gaussian, be used (see Appendix C for justification for this modelling choice). With these building blocks, the goal of H2D is to train a student model, with weights X, predict the parameters of diagonal Gaussian ()) (H2D-Gauss):
P(In ox/=" A) N(Inoa II M(noaeee -
(14)
By sampling from this Gaussian, one can obtain multiple Dirichlet distributions similar to, but cheaper than, an S2D ensemble. Clearly, the flexibility of such model can easily be extended by allowing the model to predict fully specified covariance, however due to computational tractability only diagonal covariance models are used in this work. secondary head is required for such model, see Fig. In similar fashion to previous approaches, this model can be trained using
Linsur
Layer
Feature Extraction
Linear
Exp
Layer
Figure 3: The architecture of diagonal H2D-Gauss student.
negative log- ligeliied or by estimating proxy teacher Gaussian and use KL-divergence, In this work we have adopted the proxy approach, see Appendix A.I for details
EXPERIMENTAL EVALUATION
This section investigates the self-distribution distillation approach on classifying image data. First, this approach is compared to standard trained models and established ensemble based methods (deep ensembles and MC-dropout) well as the diagonal version of SWAG (SWAG-Diag) and MIMO. Second, self-distribution distillation is combined with all above mentioned approaches. Finally, knowledge distillation is compared hierarchical distribution distillation of Dirichlet ensembles.
This comparison is based on two sets experiments. The first set compares the performance of all baselines and proposed models in terms of image classification performance and calibration on
SSince transferring knowledge from Dirichlet ensemble into student predicting a categorical critically loses information about diversity, this method will not he investigated.

---

Under review as a conference paper at ICLR 2022
CIFAR-100 100 (Krizhevsky & Hinton 2009) without (C100) and with (C100+) data augmentation. The second set of experiments compares the out-of- of-distributton (OOD) detection performance using the LSUN Yu et al. 2015), Tiny ImageNet (CS23IN 2017) and SVHN (Netzer et al. 201T) datasets.
All experiments are based on training the DenseNet- (k = 12) with a depth of 100 (Huang et al, 2017). For ensemble generation methods M = 5 models were sampled (in the case of MC-dropout ensembles and SWAG) or trained (in the case deep ensembles). For MIMO we use two output heads (M 2) due limited capacity in the chosen model (Havasi et 2021). Note that for this choice of model it was not possible to use ensemble distribution distillation since DenseNet-BC models display high confidence on the training data of CIFAR- 100 causing instability in distillation.
All single model training runs were repeated 5 times; mean + 2 standard deviations are reported. The experimental setup and additional experiments are described in Appendix D
5.1 CIFAR-100 CLASSTFICATION PERFORMANCE EXPERIMENTS
The first batch of experiments show the classification performance using a range of metrics such as accuracy, negative log-likelihood (NLL) and expected calibration error (ECE), Table [ Per- haps the most noteworthy result is the improvement in all metrics and datasets of a self-distribution distilled model compared its standard counterpart. The improvement is more than standard deviations. A similar picture can be observed for the S2D versions of SWAG-Dias DiGgg and MC-dropout which, without any notable cost of training and inference. improve upon their equivalent standard counterparts in all metries notably. Regarding MIMO small gain can still be observed when switch- ing to the self-distribution distillation framework but this boost smaller. Finally for the deep en- semble approach, the S2D version only shows marginal improvement in accuracy and NLL but notable increase ECE. In fact, it is observed that ensembling standard and S2D models reduces and increases ECE respectively. This trend is associated with the level of ensemble calibration. Un- like a standard deep ensemble, the members of the S2D counterpart are close to being calibrated, displaying little to overconfidence. Ensembling these calibrated models lead to confidet average predictions hence, the increased calibration error. Note, calibration error and negative log- likelihood easily be reduced post-training by temperature scaling predictions.
The next set of comparisons regard distilled models, the final block of Table [ As expected they all perform in between the performance of individual model and deep ensemble. While standard ensemble distillation (knowledge distillation) was found to achieve better accuracy than other dis- tillation methods, this success was highly dependent on the value of temperature scaling used. A sub-optimal choice of temperature can drastically reduce performance. On the other hand, when distilling an S2D ensemble, no additional hyper-parameters are needed. observe that while both
Table Test performance std) and train/test cost. Dropout regularisation was only used for C100. Inference times (per input) were estimated using an NVIDIA V100 GPU. *SWAG inference speeds not take into account the time to update batch norm statistics.
Dataset Model
C100 NLL
C100+ NLL
Computational Cost
Acc
RECE
Acc
EECE
Params Inference
Individual
74.6 0.5
1.11 + 007
11.95 + 168
77.5 0.2
1.01 0
10.84 2.32
0.80M
2.3ms
S2D Individual
75.7
0.87 +
2.54
78.1
0.81
4.35
MIMO
75.2
1.05 +
10.51 +
77.6
0.89
8.23
+
0.83M
2.3ms
S2D MIMO
75.4 0.1.1
0.90 = 0.88
5.77 1.63
78.1 =
0.80 +: 0.07
4.07
: 0.43
SWAG-Diag
74.8 + 1.0
1.08 = 0.05
10.73 + 1.31
77.7 = 0.9
0.98 +: 0.00
9.60
+: 3.25
1.60M
11.6ms
S2D SWAG-Diag
75.9 066
0.85 : 0.03
3.87 0.5
78.2 = 1.3
0.79 + 0.07
3.65 :: 0.62
MC ensemble
75.6 0:
0.94 +: 0.04
6.67
+ 1.18
0.80M
11.5ms
S2D MC ensemble
76.6
0.83 0.02
2.57
Deep ensemble
79.3
0.76
1.44
82.1
0.66
1.61
4,00M
11.5ms
S2D Deep ensemble
79.7
0.73
5.48
82.1
0.64
3.79
EnD
77.9
0.91
10.36
81.2
0.81
9,51
0.80M
2.3ms
H2D-Dir
77.7
0.84
3.24
80.9
0.71
3.42
112D-Gauss
77.5
0.77
1.39
80.5
0.68
2.41
0.83M
2.4ms
7

---

Under review as a conference paper at ICLR 2022
H2D-Dir and H2D-Gaur obtained higher NLL they also achieved better calibration than their S2D ensemble teacher. Lastly, one can observe that H2D-Dir and H2D-Gauss both outperform the standard SWAG-Diag and MC-dropout ensemble using only single forward pass. Although these distilled models involve an expensive training phase (a teacher ensemble is required) they are able to, test time, achieve much higher computational efficiency and estimate and decompose uncertainty.
5.2 OUT-OF-DISTRRBUTION DETECTION EXPERIMENTS
The second batch of experiments investigate the out-of-distribution detection performance of mod- els. The goal to differentiate between types of data, negative in-distribution (ID, sampled from the same source as the training data) and positive out-of-distribttinn (OOD) data.
In all experiments the models were trained C100. The ID data was always set the test set of C100 and OOD data was the test set of LSUNTTMM/VHHN Both LSUN and TIM examples had be resized or randomly cropped as preprocessing before being fed to the model. The detection was done using four uncertainty estimates: confidence, total uncertainty (TU), data or alcatoric uncertainty (DU) and knowledge or epistemic uncertainty (KU). Performance was measured using the threshold independent AUROC (Manning & Schütze 1999) and AUPR (Fawcett) 2006) metrics.
Due to limited space, some LSUN and TIM experiments have been moved to Appendix B.1
First, there is not single case in Tables and 3 where an individual model, MIMO, SWAG-Diag single S2D model. This statement holds for all the analysed uncertainties apart from confidence where both MIMO and SWAG-Diag are insignificantly better. When comparing to deep ensemble, the S2D model outperformed in many cases. The general trend is that the ensemble is able to output marginally higher quality confidence and total uncertainty estimates in most datasets, but that S2D sometimes outperforms the ensemble when using data uncertainty (as in Table 3).
or MC-dropout ensemble able to outperform the detection performance
Interestingly, the MC ensemble seems to degrade the quality of confidence and total uncertainty when compared to its standard individual counterpart. However, since MC-dropout ensemble can estimate data uncertainty, it is able to outperform the standard model overall. Similarly, the S2D MC ensemble generally has inferior detection performance compared its single deterministic model equivalent. The only exception is in detecting SVHN where the ensemble has marginally better data uncertainty estimates. Regarding SWAG-Diag and MIMO they both gain from being cast into a diff-distrinttion distillation viewpoint drastically increasing their detection performance without additional cost at inference.
Although the S2D deep ensemble, when compared to its vanilla counterpart, wasn't able to show any noticeable accuracy boost (on CIFAR-100) does outperform in this detection task. The only case where the S2D ensemble was not able to outshine the vanilla ensemble is when both use knowl-
Table 2: OOD detection results (LSUN resize) trained on C100. Best in column and best overall.
OOD %AUROC
OOD %AUPR
Model
Conf.
TU
DU
KU
Conf.
TU
DU
KU
Individual
77.3 09
79.8 =
74,2 =
76.9 1.2
S2D Individual
78.4 23
80.7 1.2
80.8
80.0 4:2
75.4 +
78.3 3.5
79.5 135
75.5 +
MIMO
78.5 12
80.5 + 1.4
80.6 = 1.4
75.0 2.
75.0 +
78.0 1.6
78.1 16
67.0 + 3.5
S2D MIMO
80.6 4.1
81.4 1 44
81.4
81.3 1.42
76.6 5.2
78.8
80.3 1
77.7 5.3.
SWAG-Diag
78.5 1.0
80.5
80.6 1.3
75.2 ox
75.0 +
78.1 1.7
78.3 +
67.1 +
S2D SWAG-Diag
78.7
80.9
81.1
80.9 3.
75.4 1 2.7
78.4 :
79.7 1:32
76.2 4:1.
MC ensemble
76.6 ax
78.3 08
78.9
72.4
72.2
74.6 1.6
75.6
64.2
S2D MC ensemble
77.7
79.8
80.5
78. 2.29
73.7
76.1
78.6
72.0 3:2
Deep ensemble
81.1
82.9
83.4
79.2
77.7
80.4
81.2
73.6
S2D Deep Ensemble
82.4
84.8
85.0
83.5
79.5
82.5
83.9
78.7
EnD
79.4
81.0
75.8
78.2
H2D-DDir
80.3
83.2
83.4
86.4
77.9
81.9
81.9
83.4
H2D-Gauss
80.8
83.9
85.7
80.7
78.2
82.0
85.8
76.0
8

---

Under review as a conference paper at ICLR 2022
Table 3: OOD detection results (SVHN) trained C100. Best in column and best overall.
Individual
79.7 56
81.8 60
88.3 +
89.6 39
S2D Individual
83.0
86.0 1:
87.7 =
81.2 3
90.6
92.0 1.6
94.4 1.11
86.1 3:33
MIMO
81.8 4.1
84.3 = 4.5
84.3
80.9 s3
89.9 = 2.5
91.4 2.8
91.4 2
88.2 #:
S2D MIMO
84.1 23
87.2 # 21
87.4 =
83.7
89.6
92.9 1.6
93.2 #
90.4 +
SWAG-Diag
$1.4
83.5 366
83.5
80.5 4
89.2 :
90.2
90.2 13
88.3 +: 366
S2D SWAG-Diag
83.2 2.7
86.3
87.7
82.7 4.3
90.7 =
92.3 1.8
94.3 +
87.3 #
MC ensemble
79.0
81.6
83.1
68.3 3.0
88.1
89.3
90.7
77.4 +
S2D MC ensemble
82.3 + 4.3
85.9
88.4 =
79.7 6.1
90.5 =
92.1 2.
95.0 1.7
85.4 -
Deep ensemble
84.5
87.2
86.8
85.0
91.3
92.5
92.2
91.5
S2D Deep ensemble
86.5
89.9
91.7
85.1
92.6
94.1
96.2
88.4
EnD
78.0
79.8
87.0
87.9
H2D-Dir 84.6
88.4
88.5
87.6
91.7
93.6
91.7
90.6
H2D-Gauss
81.2
85.3
90.1
74.5
90.0
91.4
95.9
81.7
edge uncertainty to detect SVHN examples using the AUPR metric. Generally, S2D based systems outperform their standard counterparts.
Regarding distillation based approaches, it is observed that knowledge ensemble distillation, EnD, able to outperform the standard model in all cases except SVHN detection, and no case is able to reach the deep ensemble performance, which it was distilled from. On the other hand, both the H2D-Dir and H2D-Gauss models outperform the distilled model and are able to decompose pre- dictive uncertainty. Specifically discover that H2D-Dir is able generate the highest quality knowledge uncertainty estimates almost all cases, and able to outperform its S2D ensemble teacher using this uncertainty. The H2D-Gauss model however, was not able to boast similar high quality knowledge uncertainty. Instead, this model displayed the generally best performing data un- certainty estimates, able to outperform the vanilla deep ensemble in all cases, and the S2D equivalent in all but SVHN detection.
6 CONCLUSION
Uncertainty estimation within deep learning is becoming increasingly importance, with deep ensem- bles being the standard for estimating various sources of uncertainty. However, ensembles suffer from significantly higher computational requirements. This work proposes self-distribution distil- lation (S2D), a novel collection of approaches for directly training models able to estimate and de- compose predictive uncertainty, without explicitly training an ensemble, and can seamlessly be com- bined with other approaches. Additionally, if one is not resource restricted during the training phase, novel approach, hierarchical distribution distillation (H2D), is described for transferiig/ddsttllin the knowledge of S2D style ensembles into a single flexible and robust student model. It is shown that S2D models able outperform standard models and rival MC ensembles the CIFAR-100 test set. Additionally, S2D) is able to estimate higher quality uncertainty estimates compared to stan- dard models and MC ensembles and in most cases, able better detect out-of-distribution images from the LSUN, SVHN and TIM datasets. Combination of S2D with other promising approaches such MIMO and SWAG also show additional gains in accuracy and detection performance. S2D is also able to rival the deep ensemble in certain cases even though it only requires a single forward pass. Furthermore, S2D deep ensembles and H2D derived student models are shown to notably out- perform the deep ensemble in almost all detection problems. These promising results show that the efficient self-distribution and novel hierarchical distribution distillation approaches have the poten- tial to train robust uncertainty estimating models able to outperform deep ensembles. Future work should further investigate self-distribution distillation in other domains such as natural language processing and speech recognition. The need for more efficient uncertainty estimation is especially useful for these areas as they often utilise large-scale models. Furthermore, one could also analyse variations of S2D such as utilising less weight sharing, generating more diverse teacher predictions or changing the student modelling choices.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, and Dmitry Vetrov. Pitfalls of in-domain uncertainty estimation and ensembling in deep learning. International Conference on Learning Representations, 2020.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. In International Conference on Machine Learning, 2015.
CS231N. Tiny imagenet. herokuapp. com/
Stanford University, 2017. https://tiny hagpss//etin--mmanee.... imagenet.
Jeffrey De Fauw, Joseph R. Ledsam, Bernardino Romera-Paredes, Stanislav Nikolov, Nenad Toma- sev, Sam Blackwell, Harry Askham, Xavier Glorot, Brendan O'Donoghue, Daniel Visentin, George van den Driessche, Balaji Laksbminarayanan, Clemens Meyer, Faith Mackinder, Simon Bouton, and Kareem Ayoub, et al. Clinically applicable deep learning for diagnosis and referral in retinal disease. Nature Medicine, 2018.
Stefan Depeweg, José Miguel Hernández-Lobato, Finale Doshi- and Steffen Udluft. De- composition of uncertainty bayesian deep learning for efficient and risk-sensitive learning. In International Conference on Machine Learning, 2018.
Yassir Fathullah, Andrey Malinin, and Mark J. F. Gales. Ensemble distillation approaches for gram- matical error correction. In International Conference on Acoustics, Speech and Signal Processing, 2021.
Tom Fawcett. An introduction roc analysis. In Pattern Recognition Letters, 2006.
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning, 2016.
Alex Graves. Practical variational inference for neural networks. In Advances in Neural Information Processing Systems, 2011.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, 2017.
Qiushan Guo, Xinjiang Wang, Yichao Wu, Zhipeng Yu, Ding Liang, Xiaolin Hu, and Ping Luo. Online knowledge distillation via collaborative learning. In Conference Computer Vision and Pattern Recognition, 2020.
Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snoek, Balaji Lakshmi- narayanan, Andrew Mingbo Dai. and Tran Dustin. Training independent subnetworks for robust prediction. In International Conference on Machine Learning, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Conference on Computer Vision and Pattern Recognition, 2016.
Geoffrey E. Hinton, Deng Li, Dong Yu, George E. Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N. Sainath, and Brian Kingsbury, Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. In IEEE Signal Processing Magazine, 2012.
Geoffrey E. Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. In Conference Neural Information Processing Systems, 2014.
Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. In arXiv preprint arXiv: 2011.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. European Conference on Computer Vision, 2016.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks, In Conference Computer Vision and Pattern Recognition, 2017.
10

---

Under review as a conference paper at ICLR 2022
Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. In Machine Learning. 2021.
Michael I. Jordan, Zoubin Ghahramani, Tommi S. Jaakkola, and Lawrence K. Saul. An introduction to variational methods for graphical models. In Machine Learning, 1999.
Alex Kendall and Yarin Gal. What uncertainties do we need bayesian deep learning for computer vision? In Conference on Neural Information Processing Systems, 2017.
Alex Kendall, Jeffrey Hawke, David Janz, Przemyslaw Mazur, Daniele Reda, John-Mark Allen, Vinh-Dieu Lam, Alex Bewley, and Amar Shah. Learning to drive in a day. In International Conference Robotics and Automation, 2019.
Kim and A. M Rush. Sequence-level knowledge distillation. In Conference on Empirical Methods in Natural Language Processing, 2016.
Andreas Kirsch, Joost van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acqui- sition for deep bayesian active learning. In Neural Information Processing Systems, 2019.
Alex Krizhevsky and Geoffrey E. Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. Technical Report.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton, Imagenet classification with deep convo- lutional neural networks. In Conference on Neural Information Processing Systems, 2012.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Conference on Neural Information Processing Systems, 2017.
Wesley J. Maddox, Timur Garipov, Pavel Izmailov, Dmitry Vetrov, and Andrew G. Wilson. A simple baseline for bayesian uncertainty in deep learning. In Conference on Neural Information Processing Systems, 2019.
Andrey Malinin and M. Gales. Predictive uncertainty estimation via prior networks. In Confer- on Neural Information Processing Systems, 2018.
Andrey Malinin and Mark Gales. Reverse kl-divergence training of prior networks: Improved un- certainty and adversarial robustness. In Conference on Neural Information Processing Systems, 2019.
Andrey Malinin and Mark F. Gales. Uncertainty estimation in autoregressive structured prediction. In International Conference on Learning Representations, 2021.
Andrey Malinin, Bruno Mlodozeniec, and Mark J. F Gales. Ensemble distribution distillation. In International Conference on Learning Representations, 2020.
Chris Manning and Hinrich Schütze. Foundations of Statistical Natural Language Processing. MIT Press, 1999.
Thomas Minka. Estimating a dirichlet distribution. Technical report, Massachusetts Institute of Technology, 2000. Technical Report.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In Neural Information Processing Systems Workshop Deep Learning and Unsupervised Feature Learning, 2011. http://
uflal wtw...ssanforddeee edw.orrddddu/aaeeo www..e//w.....uubb........
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua V. Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In Conference on Neural Information Pro- cessing Systems, 2019.
Max Ryabinin, Andrey Malinin, and Mark J. F. Gales. Scaling ensemble distribution distillation to many classes with proxy targets. In arXiv preprint arXiv:2001.10995, 2021.

---

Under review as a conference paper at ICLR 2022
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. Conference on Computer Vision and Pattern Recognition, 2015.
Seonguk Seo, Paul Hongsuck Seo, and Bohyung Han. Learning for single-shot confidence calibra- tion in deep neural networks through stochastic inferences. In Conference on Computer Vision and Pattern Recognition, 2019.
Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way prevent neural networks from overfitting In Journal of Machine Learning Research, 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Conference on Neural In- formation Processing Systems, 2017.
Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: an alternative approach efficient ensemble and lifelong learning. In International Conference on Machine Learning, 2020.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint 50.00366.,,, 2015.
Sergey Zagonuyko and Nikos Komodakis. Wide residual networks. In The British Machine Vision Conference, 2016.
12

---

Under review as a conference paper at ICLR 2022
A
EXPERIMENTAL CONFIGURATION
Table Description of datasets used in training and evaluating models.
Dataset
Train
Test
Classes
CIFAR-100
50000
10000
100
LSUN
10000
10
SVHN
26032
10
Tiny ImageNet
10000
200
All models were trained on the CIFAR-100 dataset, with and without data augmentation. The aug- mentation scheme involves randomly mirroring and shifting images following He et al. (20T6); Huang et al. (2016). Remaining datasets were used as out-of-distribution samples in the detection task.
All individual models, and ensemble members were based off DenseNet-BC (k = 12, 100 layers) architecture and trained according to Huang et al. (2017). SWAG-Diag was obtained by checkpointing the weights of the last 20 epochs with a reduced learning rate of = 1.0 x 10 MIMO with two output heads trained using the same setup for the standard model. To keep training costs comparable to (S2D) individual models, no batch or input repetition was used (Havasi al. 202T). Similarly all delf-distribution distilled equivalents were trained with identical training recipes with the addition of a student loss (μ- = 1.28 x 10 10-+).
Regarding distilled based models, the EnD baseline was trained using negative log-likelihood using the average temperature scaled prediction of the teacher ensemble, with 7 E {1.0, 2.0, 3.0, 4.0, 5.0). For the hierarchical distribution distillation approaches the students were first initialised with the weights of an S2D model trained for 150 epochs, for increased stability. Thereafter, each student was trained using the appropriate H2D criteria with a significantly reduced learning rate. H2D-Dir was trained using = 5.0 x 10-5 for an additional 1550 epochs. H2D-Gauss required an initial learning rate of r- = 5.0 x 10-3 which was reduced by factor of after 75 and 150 epochs. was trained for 170 epochs. Additionally, uncertainties were computed by generating 50 samples from each Gaussian prediction, since this modelling choice does not result in closed form expressions.
A.1 PROXY TARGET TRAINING
Since the use of negative log-likelihood can be unstable in training S2D and distilling H2D models we utilise proxy targets and KL-divergence. It has already been mentioned that the proxy target in S2D follows:
ā = arg max (i ()), π (m) = Sof (m)
(15)
&
Each categorical prediction will be temperature scaled, with 1.5, to mitigate overconfident predictions. While H2D-Dir does not require any proxy targets, the Gaussian equivalent does. The proxy diagonal Gaussian, estimated according to maximum log-likelihood, has a closed-form ex- pression:
M Σ In M
M Σ
-²=
02
a1 (m)
-i)²
(16)
M
m=1
where 2² = ④ represents an element-wise multiplication. This is then used in KI.-divergence based loss, training the student with prediction μt, a according to:
KL (M(2;; [( = -
(Mc - p2.2
(17)
20
Note however, that the proxy targets are detached from any back gradient propagation calculations. This is to simulate typical teacher-student knowledge transfer where teacher weights are kept fixed during student training.
13

---

Under review as a conference paper at ICLR 2022
B OUT-OF-DISTRIBUTION DETECTION
This section covers remaining out-of-distribution detection experiments. First, we cover the LSUN and Tiny ImageNet detection problem for all models considered in section 5.2 Thereafter, additional experiments will be run on ensembles of various sizes. This is to investigate if the low quality of knowledge uncertainty estimates caused by limited number of ensemble members.
B.1 TINY IMAGENET EXPERIMENTS
Similar to the results section 5.2 the S2D Deep ensemble and H2D-Gauss outperformed all other models, see Table and 7 The only exception the use confidence on resized TIM with the AU- ROC metric where the Deep ensemble marginally outperforms the S2D equivalent. However, unlike previous results, knowledge uncertainty seems to perform on par with or outperform confidence. The exception is the MC ensemble.
Table 5: OOD detection results (LSUN random crop) trained on C100. Best in column and best overall.
OOD AAUROC
OOD AAUPR
Model
Conf.
TU
DU
KU
Conf.
TU
DU
KU
Individual
83.2
85.7 +
79.4 5.6
83.0 5.9
S2D Individual
+
88.9 4:11
90.3 = 40
84.1 + 4.8
81.9 6.62
= 5.7
90.3 = 50
76.0 =
MIMO
83.3 + 3.9
86,2 + 4.2
86.3 4.3
80.9 + 1.6
79.6 6.6
83.8 6.8
83.8 69
72.4 +
S2D MIMO
2.5
89.5 +:
90.7 2.8
4: 2.8
78.0 3.4
84.8 3.5
89.4
3:3.3
SWAG-Diag
84.3 2.
87.1 3:1.1
87.1 3.1
80.8 1:
80.8 1:400
84.5 : 3.8
84.6 +: 3.8
73.4 14.2
S2D SWAG-Diag
85.6
89.1
90.4
85.3 +
81.8
86.5
90.2
76.4 3.6
MC ensemble
+ 3.5
84.4 + 4.0
86.4 388
63.0 +:
77.0 + 3.6
84.9 + 4.0
+ 3.1
S2D MC ensemble
83.3 2.3
86.9 +
90.0
77.7 +
79.3 3:2
83.8
90.1
69.8 1::
Deep ensemble
85.9
89.1
90.9
80.4
82.0
86.3
89.1
72.5
S2D Deep ensemble
86.8
90.5
93.7
81.5
83.0
87.9
93.9
73.4
EnD
84.7
87.4
81.1
84.9
H2D-Dir
85.3
88.9
88.8
91.7
82.5
87.4
87.6
87.1
H2D-Gauss
86.9
90.6
95.1
76.0
82.9
88.0
95.7
67.0
Table 6: OOD detection results (TIM resize) trained C100. Best in column and best overall.
Individual
77.6
79.5 +
74.2 -
77.1
S2D Individual
78.0 08
80.1 0:07
79.6 0.8
78. 1:04
75.3 1 0.9
77.7 1.9
76.6 1:
76.3 0.55
MIMO S2D MIMO
78.1 0.
79.9
79.9
76.3 1.
74.6
77.3
77.4
69.6 2.00
80.1 12
80.7 + 1.2
80.7 + 1.2
80.4 + 12
77.3 + 1.6
77.8 + 1.6
77.7 + 15
77.5 + 1.6
SWAG-Diag
77.7 0.7
79.6
79.6 0.6
76.4 +
74.2
77.0 0.8
77.1 a:08
70.0 + 0.7
S2D SWAG-Diag
78.6 0.
80.5 + 0.6
80.1 = 0.7
79.2 0.0.
75.6 + 0.9
78.1
77.1 # 10
76.5 =
MC ensemble
78.5 0.55
80.6 = 0.
80.8 0.4
76.6 +
75.2 - 0.5
78.1 0.6
78.4 a.
70.9 + 1.1
S2D MC ensemble
79.3 0.5
4:
81.1 à: 0.5
0.6
76.4 +
78.5 0.8
4:
77.1 H:: 0.7
Deep ensemble
81.7
83.6
83.5
78.9
81.6
76.6
S2D Deep Ensemble
81.5
84.2
82.8
82.8
79.1
82.0
79.9
80.0
EnD
78.7
80.4
75.4
78.0
H2D-Dir
77.3
79.8
79.6
81.6
74.5
77.9
77.7
79.2
H2D-Gauss
80.5
82.6
83.7
82.8
78.8
81.4
82.5
80.1
14

---

Under review as a conference paper at ICLR 2022
Table 7: OOD detection results (TIM random crop) trained on C100. Best in column and best overall.
Individual
76.7
79.2
74.7
78.5
S2D Individual
80.2 = 59
85.4 62
84.5 59
86.4 6.
79.3 # 6.3
83.3 6.7
81.9 166
83.1 =
MIMO S2D MIMO
79.4
81,9 5.3
81.9
79.8 4.6
77.1
80.9
80.8 53
74.9 =
80.3 :: 8.6
86.5 + 8.5
86.5 =: 8.5
86.9 8.6
80.0 d: 6.5
82,9 6.4
83.0 + 6.4
84.9 4: 6.5
SWAG- Diag
78.4 1: 3.5
80,9 4:
80.9 =
78.6 2.2
76.0 3.3
79.8
79.7 1:3.7
73.7 3:5
SWAG-Diag
80.5 + 6.0
84.8 6.5
83.8 = 6.3
86.6 666
79.4 # 5.5
83.4 6.1
+ 622
83.4 +:
MC ensemble
75.8 + 4.5
78.8 4.8
79.7 = 4,9
69.3 3.7
74.3 # 40
78.5 4.5
80.0 +
60.8 d:
S2D MC ensemble
78.8 +:
82.1 6.4
82.6 =
82.0 6.1
77.1 + s.2
81.1
81.8 + 5.1
79.8 4:
Deep ensemble
80.9
84.2
83.5
82.3
79.3
83.9
79.8
S2D Deep ensemble
84.8
88.5
86.4
89.7
82.8
87.3
84.4
87.7
EnD
72.7
74.8
75.0
H2D-Dir
74.7
78.2
77.9
84.2
73.2
77.7
77.5
81.7
H2D-Gauss
83.2
88.0
88.0
88.5
81.0
86.0
87.2
84.1
B.2 ENSEMBLE SIZE EXPERIMENTS
Knowledge uncertainty was found to have underwhelming performance (especially for MC and Deep ensembles) did not show similar trends to prior work (Malinin Gales) 2018 2021 Malinin et al.) [0200) To possibly mitigate this, the ensemble size was increased as a smaller number of models could lead to inaccurate measures diversity and knowledge uncertainty. Results are compiled in Tables 3
Performance on the CIFAR-100 test set is shown in Table g Increasing ensemble size leads to improved accuracy and lower negative log- likeliioisss as would be expected. The MC ensemble also becomes better calibrated. The Deep ensemble on the other hand has increasing calibration with the number of members. This is due the ensemble prediction becoming under-confident when averaging over a large number members.
Out-of-distributio detection performance LSUN, SVHN and TIM compiled in Tables B3 Although the MC ensemble enjoys improved accuracy when increased in size, it seems to remain relatively unaffected in terms of OOD detection using any uncertainty metrie. In detecting LSUN using random crops, the performance of KU interestingly deteriorates notably. Overall this points MC ensembles' lacking ability in utilising new information from additional ensemble member draws/samples for better uncertainty estimation. Regarding the Deep ensemble, it generally im- proves with increasing size with any metric, however with diminishing returns. this case un- certainties improve with ensemble size, not only knowledge uncertainty. Therefore it seems that the cause for confidence, total and data outperforming knowledge uncertainty is not due to the ensemble size being limited to five members.
Table 8: Test performance of various ensembles and sizes (1: 2 std). All models are trained on C100.
Ensemble Type
Ensemble Size (M)
Acc.
NLL
%ECE
75.6 + 09
0.94 0.04
6.67 + 1.18
MC
10
75.8 + 0.9
0.92
6.11 + 1.11
20
76.0 +
0.91
0.04
5.81 + 1.12
79.3
0.76
1.44
Deep
10
80.1
0.71
1.91
80.3
0.68
2.19
15

---

Under review as a conference paper at ICLR 2022
Table 9: OOD detection results (LSUN resize) trained C100.
OOD AAUROC
OOD
%AUP
Type
M
Conf.
TU
DU
Conf.
TU
DU
KU
76.6 0.8
78.3 + 0.8
78.9 0.
72.4 + 1.2
72.2 1
74.6 + 16
75.6 + 1.7.
64.2 + 20
MC
10
76.7
78.3
79.1 4.09
72.6
72.3 +
74.6
75.9
64.3
20
76.8 0.7
78.4 + 0.8
79.2 + 08
72.7 = 1.3
72.4 1
74.6 + 1.6
76.0 + 1.7
64.3 +
81.1
82.9
83.4
79.2
77.7
80,4
81.2
73.6
Deep
10
82.0
83.9
84.8
80.3
79.1
81.8
83.4
74.9
20
82.2
84.0
85.1
80.9
79.4
81.8
83.6
75.7
Table 10: OOD detection results (LSUN random crop) trained C100.
81.0
84.4
86.4
63.0
77.0
81.7
84.9
53.1
MC
10
81.0
= 3.5
84.4 3.9
86.7 + 3.2
61.6 + 3.9
77.0 30
81.8 + 4.0
85.4 # 4.0
52.2 + 3.0
20
80.8
84.1
86.6
60.9
76.7 39
81.3
85.3
$1.7
85.9
89.1
90.9
80.4
82.0
86.3
89.1
72.5
Deep
10
85.7
89.3
91.3
81.3
81.8
86.4
89.9
73.1
20
86.2
89.8
92.2
82.0
82.1
86.8
91.0
73.1
Table 11: OOD detection results (SVHN) trained on C100.
79.0 4.3
81.6 + 4.7
83.1 + 4.6
68.3 + 3.0
88.1 2.
89.3 4:33
90.7 #
77.4 + 18
MC
10
78.9
81.5 4.7
83.3 4.77
67.5
88.0 277
89.3 13
90.9
76.6 20
20
78.9 44
81.5 4.7
83.3 1:47
67.1 = 3.
88.1 23
89.2 1 33
90.9 2.1
76.3 22
84.5
87.2
86,8
85.0
91.3
92.5
92.2
91.5
Deep
10
84.1
87.0
87.5
83.9
91.2
92.4
93.1
90.3
20
83.7
86.6
87.2
84.1
91.0
92.2
92.9
90.6
Table 12: OOD detection results (TIM resize) trained C100.
5
78.5
80.6 0.3
80.8
76.6
75.2 0.5
78.1 106
78.4 0.5
70.9
MC
10
78.7 = 0.6
80.8 = 04
81.0 o.5
77.4 = 0.7
75.4 0:
78.4 106
78.7 = 0.5
72.2 +
20
78.8
80.9 044
81.2 1:44
77.9
75.6 0:
78.4 1
78.8
72.9 1
81.7
83.6
83.5
81.0
78.9
81.6
81.5
76.6
Deep
10
82.3
84.1
84,2
82.4
79.8
82.2
82,4
78.7
20
82.6
84.4
84.5
83.0
80.1
82.4
82.8
79.6
Table 13: OOD detection results (TIM random crop) trained on C100.
75.8 = 45
78.8 =
79.7 1:99
69.3 = 37
74.3 40
78.5 1 4.5
80.0 = 4.3
60.8 1: 37
MC
10
75.7
78.7 5.1
79.7 5.52
69.1
74.2 42
78.5 4:4
80.2
60.7 38
20
75.7 = 47
78.6 = 5.
79.7 5.5.
69.0 = 4.1
74.3 4.1
78.4 44
80.3 =
60.6 1: 4.4
80.9
84.2
83.5
82.3
79.3
83,9
83.2
79.8
Deep
10
82.8
86.5
85.7
85.5
81.0
85.8
85.0
83.7
20
83.4
87.1
86.1
86.8
81.6
86.4
85.4
C BEHAVIOUR OF UNCERTAINTIES
This section investigates how the uncertaintics produced from a vanilla Deep ensemble differ from self-distribution distilled derived systems, and how well hierarchical distribution distillation captures the behaviour of its teacher. The comparison will be made between the in-domain CIFAR-100 and, out of simplicity, only the oft-of-domain SVHN test set.
the contrast of various uncertainties between an CIFAR- 100 (ID) and SVHN (OOD) sets. Clearly, the S2D systems output ID uncertainties in consistent manner, matching the
16

---

Under review as a conference paper at ICLR 2022
140-
120-
sharte NOT Cans
2.0
K25
820
415-
115
105
500
36
26
12
24
(a) ID: Total uncertainty
(b) Data uncertainty
(c) ID: Knowledge uncertainty
107
Sibiee erants
800
35 4
030
14
1.5
35
(d) OOD: Total uncertainty
(c) OOD: Data uncertainty
(1) OOD: Knowledge uncertainty
Figure 4: Histograms of various uncertainties produced by Deep ensemble, S2D, S2D Deep ensemble and systems. -uitr--diitiiuution data was generated from the SVHN test set.
conceptually different Deep ensemble. Observe that S2D integrates temperature scaling (smoothing predictions) into the training of models; total and data uncertaintie esstimated by these models will naturally have larger entropy than Deep ensembles. While it is expected that the Deep ensemble would have different behaviour on the SVHN OOD set, it is surprising to observe how well H2D- Gauss aligns with its S2D Deep ensemble teacher. An individual S2D model was also able to generate closely related total and data uncertainty estimates, but suffers significantly in producing consistent knowledge uncertainties. These results raise the question if a Gaussian student could capture the diversity in vanilla Deep ensemble by modelling the logits, in similar fashion to how H2D-Gauss models its teacher-a a possible avenue for future work.
D ADDITIONAL EXPERIMENTS: WIDERESNET
Following the DenseNet-BC experiments in section 5 we repeated them with different architecture. In this section we focus on a significantly larger WideResNet (Zagoruyko & Komodakis) (016)) model with depth of 28 and widening factor of 10. The standard and S2D models were both trained as described in Zagoruyko & Komodakis (0116), with the S2D specific parameters being the same previously described. The only difference is that teacher predictions were generated using multiplicative Gaussian noise with a fixed standard deviation of 0.10.
The
model was also trained in a different manner. First, it was initialised from S2D model trained for 150 epochs. Thereafter it was trained for an additional 80 epochs with a starting learning rate of r = 2 10-3 which was reduced by factor of after 60 epochs. For this section, EnD and H2D-Dir were not investigated.
Table 14 shows test set performance. Unlike previous experiments, S2D was not able to outperform an individual model by more than two standard deviations, in this case achieving around one standard deviation improvement in accuracy. Interestingly, the MC approach has worse accuracy for both the standard and S2D case, however this could be due to the small number of drawn samples (M = 5). Furthermore, both Deep ensembles significantly outperform their individual equivalents with the S2D version being slightly better in all measured performance metrics. The notable result in this table is the high performance of m2DsGa able to outperform the Deep ensemble in C100 and achieve near ensemble performance in C100+.
In the OOD detection task we observe that both versions the MC ensemble struggle to outperform their individual counterparts. There also seems to be a disparity in performance when comparing re- size and random cropped LSUN and TIM. With random crops, all S2D systems notably outperform
knowledge uncertainty does necessarily increase with temperature.
17

---

Under review as a conference paper at ICLR 2022
Table 14: Test performance (= 2 std).
Dataset
C100
C100+
Model
Acc.
NLL
%BCE
Acc.
NLL
&ECE
Individual
73.9 0.
1.05 0.02
5.26 + 0.78
81.1 00
0.76
5.21
S2D Individual
74.2 0.
1.06 0,0
5.48 + 2.25
81.3 2.3
0.74 + nol
4.24 0.74
MC ensemble
73.6 0.5
1.05 0.03
4.70 + 0.88
81.0 0.5
0.74 + nol
3.29 0.36
S2D MC ensemble
73.8
1.03 0.04
2.95 + 1.01
81.0
0.73 +
1.99
Deep ensemble
77.1
0.88
5.08
83.4
0.63
2.27
S2D Deep ensemble
77.9
0.86
4.52
83.6
0.63
1.84
H2D-Gauss
77.4
0.95
5.19
82.8
0.71
2.45
their standard counterparts. this case both S2D Individual and H2D-Gauss were able to outper- form the Deep ensemble using any uncertainty metric. In the other case of resizing LSUN and TIM images and SVHN the detection performance difference smaller but the S2D Deep ensemble still remains the best model with both DausGauus and Deep ensemble performing similarly.
Table 15: LSUN (resize) OOD detection results. Best in column and best overall.
OOD SAUROC
OOD AAUPR
Model
Conf.
TU
DU
KU
Conf.
DU
Individual
76.3 aos
76.7 1
70.7 1
71.1 0.
S2D Individual
76.0 + 1.1
76.5 i 1.5
76.7 =
75.7 1.6
71.4 1: 1.8
72.0 277
72.8 1:37
69.7 : 2.0
MC ensemble
75.8
76.2 0.7
76.4
65.2 1.7
70.3
70.5
70.8
56.2
MC ensemble
75.7 + 1.0
76.4 1.7
77.0 = 1.6
75.2 21
71.0 # 1.6
71.6 2.7
73.1 4
69.6 +: 2.6
Deep ensemble
77.6
78.0
78.4
68.0
72.3
72.6
73.1
58.8
S2D Deep ensemble
77.7
78.5
79.3
76.8
73.2
74.1
75.9
71.3
H2D-Gaue
77.1
77.2
77.8
77.5
72.0
71.8
71.9
72.3
Table 16: LSUN (random crop) OOD detection results. Best in column and best overall.
Individual
72.4 5.0
73.9 5.4
67.0 1:
68.7 3.1
S2D Individual
75.8 3.44
77.6 4.3
77.9 =
76.5 1:4
70.5 1 3.9
72.6 49
1.4.
71.4 ::
MC ensemble
68.9 5.6
70.3 6.0
70.9 = 6.2
50.8 43
64.0 + 3.0
65.2 35
66. + 36
45.7 + 1.5
S2D MC ensemble
72.7 s2
74.5 + 41
75.9
72.0 444
67.7 3.3
69.7 4.6
73.4 4.4
65.7 +
Deep ensemble
72.1
74.2
75.2
60.6
67.2
69.2
70.5
51.6
S2D Deep ensemble
75.5
78.4
80.0
75.4
73.9
77.2
69.0
H2D-Gauss
76.0
77.6
77.8
76.4
69.6
71.5
74.1
70.9
Table 17: SVHN OOD detection results. Best in column and best overall.
Individual
80.1 +: 4.6
81.6 44
4: 2.4
89.0 23
S2D Individual
80.1 4:
81.6 4.4
81.9 =
81.4 1:54
88.6 a: 2.3
89.2 25
90.1 2:25
87.8 ::
MC ensemble
77.6 49
79.1 4.5
79.7
56.6 2.
86.9 2.3
87.5
88.0 2:
70.2
MC ensemble
77.3 4.7
79.0 48
80.1 = 466
77.3 + 5.6
+ 2.5
87.7 2.7
89.6 + 2.5
85.7 +
Deep ensemble
81.5
83.4
84.0
68.3
89.2
89.9
90.4
77.9
S2D Deep ensemble
81.5
83.7
84.6
81.8
89.6
90.5
92.0
88.1
H2D -aussss
81.5
82.1
83.2
80.6
88.6
88.4
90.5
87.1
18

---

Under review as a conference paper at ICLR 2022
Table 18: TIM (resize) OOD detection results. Best in column and best overall.
Individual
79.7 + 0.4
80.5 0.4
75,9
76.9
S2D Individual
79.2 +
0.5
80.2 +
80.2 0.4
#
77.1 1.0
76.7 +
MC ensemble
79.8 = 0.4
80.6 = 0.3
80.7 = 04
68.3 1.7
07
77.0 0.6
77.1 :06
59.5 +: 1.6
S2D MC ensemble
79.4
80.3
80.2
80.1 +
75.9 09
77.1
77.2 1
76.8 =
Deep ensemble
81.8
82.7
82.7
72.5
78.4
79.3
79.2
64.1
S2D Deep ensemble
81.9
82.9
82.9
82.5
79.0
80.2
79.6
M2D-Gauss
80.9
81.4
81.4
81.5
77.4
79.0
78.9
78.0
Table 19: TIM (random crop) OOD detection results. Best in column and best overall.
Individual
71.2 3.38
72,8 4.0
68.9 3.5
70.9 4.0
S2D Individual
73.1 1:30
74,9 1.6
76.3 :
75.9 3.44
71.4
73.7 22
1:24
73.4 ::
MC ensemble
70.1 +s
71.8 +
72.1 = 3.7
$7.1 1.
# 3.6
70.2 = 39
4
50.4 +
S2D MC ensemble
71.7 2:7
73.8 + 32
74.2
73.7 +
70.0
72.6
73.3 + 13
71.9 1.6
Deep ensemble
72.2
74.5
74.7
65.2
70.3
72.9
73.0
58.1
S2D Deep ensemble
74.3
77.0
77.3
77.1
72.6
75.9
76.2
75.5
H2D-Gauss
75.2
76.9
77.3
76.4
72.0
74.0
74.5
73.5
19


Reviews:
[{"review_id": "s2M7eu4FpHn", "reviewer": "Reviewer_MHeN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- The paper is pretty solid in its technical idea contributions. It explores the (likely previously unexplored) space of modeling uncertainty with distillation/self-distillation very thoroughly, although using Dirichlet-based output for modeling uncertainty is not new.\n- The writing is generally good. The good story flow in the paper makes it easy to read and follow.\n- The proposed methods achieve good results compared to the baselines.\n\nWeaknesses\n- The paper mentions that ensembles suffer from high computational requirements and they propose S2D as an alternative to that. However, S2D Individual still has some pretty large performance gaps to Deep Ensemble, especially in Table 1 and 2. The methods that outperform Deep Ensemble still need heavy computations.\n- The paper only includes EnD, traditional MC, and Deep Ensembles as the methods for experimental comparison. There are many more existing works that do uncertainty estimation. One example is `Learning for single-shot confidence calibration in deep neural networks through stochastic inferences. CVPR 2019.` that distills confidence/uncertainty knowledge to a student that does single-shot confidence estimation.\n- En2D is introduced as an existing method that can distill the distribution knowledge to the student in Sec. 2.2 but it is not included in experimental comparison. Is EnD actually En2D?\n- The tables do not show the computational (memory, inference time) requirements of each method/row. This makes it hard to make a good comparison between the methods.\n- What are the differences between \"Best in category\" and \"Best overall\"?\n- In the Introduction section, there is this statement \"inputs it is expected that the trained model parameters can return reliable predictions.\" that does not make any sense.", "summary_of_the_paper": "This paper introduces several distillation-based uncertainty estimation methods with distribution-based outputs and losses for the student network: (1) distilling from an ensemble of point-estimation/regular sub-network to a distribution-based Dirichlet sub-network (S2D); (2) distilling from an ensemble of Dirichlet networks to a single network (H2D) that either predicts a Dirichlet distribution (H2D-Dir) or a Gaussian distribution over multiple Dirichlet parameters (H2D-Gauss). Experimental results show that the proposed methods outperform conventional MC and deep ensemble methods on several tasks.  ", "summary_of_the_review": "This paper introduces some nice ideas for uncertainty estimation with distillation, but it fails to make comparison to many more related existing works and it has some clarity issues. It is hard to know how it stands when it sufficiently considers the existing methods for comparison.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "aeQau0nHd4", "reviewer": "Reviewer_EMdF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Strengths*\n\n* It is a simple model that requires only one network and two \"heads\" (linear layers at the end of the network) for ensemble distillation. \n* The method is able to match the deep ensemble's performance for Cifar-100 (classification, calibration), and LSUN and SVHN (OOD deteection).\n\n*Weaknesses*\n\n* Clarity: The notion could be improved and does not contribute to understanding the work. E.g., what's $M$, $m$, $\\alpha$, $\\delta$?,  There are several typos and missing words (e.g., \"viewed from a Bayesian.\", \"over categoricals:\", \"There are a wide range of\"). Please proofread the paper and fix these.\n* Significance: The distillation itself is very similar to [1] and the authors could make a better job in detailing the difference of it. The difference is about using the self-distribution distillation, however, the authors fail to compare with existing methods to compare whether their methods outperform other distillation or ensembling methods.\n* Insufficient evaluation: There are NO (!) comparisons to other distillation approaches or state-of-the-art (efficient) ensembling methods. Among others, works like [1, 2, 3, 4] can be compared to. Please look into these works for more models that can be compared to. Further, it is also good to not just compare the performance of the distilled model but also compare its efficiency (e.g., number of parameters, FLOPs).\n\n*References*\n\n[1] Malinin, A., Mlodozeniec, B. and Gales, M., 2019. Ensemble distribution distillation. arXiv preprint arXiv:1905.00076.\n\n[2] Wenzel, F., Snoek, J., Tran, D. and Jenatton, R., 2020. Hyperparameter ensembles for robustness and uncertainty quantification. arXiv preprint arXiv:2006.13570.\n\n[3] Havasi, M., Jenatton, R., Fort, S., Liu, J.Z., Snoek, J., Lakshminarayanan, B., Dai, A.M. and Tran, D., 2020. Training independent subnetworks for robust prediction. arXiv preprint arXiv:2010.06610.", "summary_of_the_paper": "The paper proposes an ensemble distillation approach, in which one network is used as feature extractor and \"two heads\" (two networks representing teacher and student) are added to the network for self-distillation. The multiple teacher predictions can be generated through by adding multiplicative Gaussian noise. The distillation approach follows Malinin et al. [1] who proposes to model the predictive distribution with a Dirichlet and predicts the parameter of the Dirichlet distribution (instead of predicting the Categorial distribution). This approach can be both used for model distillation and ensemble distillation. The authors evaluated their models on CIFAR-100, LSUN, SVHN w.r.t. classification performance, calibration and out-of-distribution detection.", "summary_of_the_review": "I do think this is an interesting approach of using just one model for ensembling and distillation. However, at this stage, I found the paper lacks in clarity, significance and thorough evaluation (see weaknesses above for details). Thus I am currently recommending a weak reject.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "O_1rGSNcBX6", "reviewer": "Reviewer_MF93", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Although this paper is a creative attack on a significant problem, I cannot recommend acceptance at this time.\nMy main reservations are to do with how the paper situates itself with respect to prior work and the theoretical representation of the ideas.\n\nThe prior work section could be improved.\n1) Ensemble methods, variational inference, and SWAG are distinct methods. Ensemble methods (e.g., Lakshminarayanan et al. 2017) are as you describe. Monte Carlo dropout (MCDO) is a variational inference (VI) method, which makes a variational approximation. The predictive distribution of an MCDO variational approximation can be efficiently computed with a Monte Carlo expectation, which has some resemblances to an ensemble, but has a different principled justification. SWAG is an ammortized stochastic gradient Markov-chain Monte Carlo method, which is neither an ensemble nor a variational method.\n2) You write that MCDO \"generally does not perform as well\" as deep ensembles, which should be supported by citation (and is a complicated claim which depends a lot on the resource constraints and problem definition).\n3) Generally, the claim that different measures of uncertainty are grounded in different measures is best supported by some evaluation of where the noise comes from and what it might have to do with the prior which was chosen.\n4) Minor points: Jordan et al. 1999 Introduction to Variational Inference is a better citation for variational inference; the Gal, 2016 thesis which you cite for uncertainty decomposition is less appropriate than Kendall and Gal 2017 \"What Uncertainties Do We Need for Bayesian Deep Learning for Computer Vision\"; the Amodei et al. 2016 paper you cite isn't really about fully autonomous driving and you'd be better off citing a paper that is in the opening paragraph\n\nIf I understand correctly, the main difference between your proposal and Ensemble Distribution Distillation is the choice to tie all parameters before the last layer?\nYou mention that this may have a beneficial regularizing effect.\nAre you able to construct experiments that show that this is the source of the benefit?\nAre there other elements of your contribution that might extend here also?\n\nI would like to see more of a theoretical investigation of what kinds of uncertainty are actually represented by the estimates.\nFor example, it seems like the network is not able to express any uncertainty that corresponds to the fixed feature representation.\n\nReference results for DenseNet-BC are ~82% accuracy on CIFAR-10, which is significantly greater than any of the results you report.\nThis makes me wonder if there is a problem with tuning the baselines, which might make the rest of the results less reliable?\nDo you know what might be going on there?", "summary_of_the_paper": "The authors propose a way to construct and train neural networks which represent uncertainty which is efficient with respect to compute and memory at execution-time.\nThey propose learning a shared feature representation with an ensemble-head which is distilled into a single head during training, with the ensemble-head thrown away for deployment.", "summary_of_the_review": "I am not able to recommend acceptance. I think the paper could use a more careful grounding in prior work which investigates the contributions and their theoretical motivations more carefully. I also have questions about the implementation of the baselines and whether there might be a reason the performance of baselines falls below those reported for the architecture.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Dqr0gmiaOop", "reviewer": "Reviewer_Wumr", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThis paper is clearly presented.\n\nThe proposed method is easy to be integrated into other methods and Experiments are done on different tasks to validate effectiveness.\n\nweakness:\n\nmy only concern is the novelty of this method. S2D combines parameter sharing, stochastic regularization and distribution distillation.  These components exist in the literature. For example, parameter sharing is popular in online knowledge distillation methods like [1]; Also, in experiments, the authors did not compare with other distillation methods and did not show the model size and flops comparison. \n[1] Online Knowledge Distillation via Collaborative Learning", "summary_of_the_paper": "This paper contributes to neural network classifier training and uncertainty prediction.  It proposes a self-distribution distillation method that can train a single model to estimate uncertainties in an integrated training phase.  Also, it is flexible to be extended to build ensembles of models in the training phase and efficiently deployed in the test phase. Experiments are done on both classification and out-of-distribution detection tasks to show effectiveness.", "summary_of_the_review": "This paper proposed an efficient uncertainty prediction. Experiments are done on two tasks to do evaluations. The paper is well written. However, since the proposed method is like a combination of existing techniques. The main concern is the novelty of the proposed method and whether it is fairly compared with SOTA. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This article presents novel distillation-based methods for neural network training and uncertainty estimation. While the idea is interesting, there is a general agreement amongst reviewers that the paper lacks clarity, adequate discussion of the relevant literature  and comparisons to existing work. Although the revision uploaded by the authors goes in the right direction by adding some experiments and clarifying some of the issues raised by the reviewers, further work is needed to make the submission stronger.
Decision: Reject