Diversity Matters When Learning From Ensembles
Authors: Giung Nam
Jongmin Yoon
Yoonho Lee
Juho Lee
Venue: NeurIPS 2021 Poster (2021)
Keywords: deep ensembles
knowledge distillation
output diversified sampling
batch ensemble
calibration

Abstract: Deep ensembles excel in large-scale image classification tasks both in terms of prediction accuracy and calibration. Despite being simple to train, the computation and memory cost of deep ensembles limits their practicability. While some recent works propose to distill an ensemble model into a single model to reduce such costs, there is still a performance gap between the ensemble and distilled models. We propose a simple approach for reducing this gap, i.e., making the distilled performance close to the full ensemble. Our key assumption is that a distilled model should absorb as much function diversity inside the ensemble as possible. We first empirically show that the typical distillation procedure does not effectively transfer such diversity, especially for complex models that achieve near-zero training error. To fix this, we propose a perturbation strategy for distillation that reveals diversity by seeking inputs for which ensemble member outputs disagree. We empirically show that a model distilled with such perturbed samples indeed exhibits enhanced diversity, leading to improved performance.

Diversity Matters When Learning From Ensembles
Giung Nam Jongmin Yoon'*, Yoonho Lee Juho Lee
KAIST', Daejeon, South Korea, AITRICS", Scoul, South Korea, Stanford University USA {giung, jm. juholee) @kaist........ ac.kr
Abstract
Deep ensembles excel in large-scale image classification tasks both in terms of prediction accuracy and calibration. Despite being simple to train, the computation and memory cost of deep ensembles limits their practicability. While some recent works propose distill an ensemble model into a single model to reduce such costs, there is still a performance gap between the ensemble and distilled models. We propose simple approach for reducing this gap, i.e., making the distilled performance close to the full ensemble. Our key assumption is that a distilled model should absorb as much function diversity inside the ensemble as possible. We first empirically show that the typical distillation procedure does not effectively transfer such diversity, especially for complex models that achieve near-zero training error. To fix this, we propose perturbation strategy for distillation that reveals diversity by seeking inputs for which ensemble member outputs disagree. We empirically show that model distilled with such perturbed samples indeed exhibits enhanced diversity, leading to improved performance.
Introduction
Deep Ensemble (DE) [Lakshminarayanan et al., 2017], a simple method to ensemble the same model trained multiple times with different random initializations, is considered to be competitive method for various tasks involving deep neural networks both in terms of prediction accuracy and uncertainty estimation. Several works have tried to reveal the secret behind DE's effectiveness. As stated by Duvenaud et al. [2016] and Wilson and Izmailov [2020], DE can be considered as an approximate Bayesian Model Average (BMA) procedure. Fort et al. [2019] studied the loss landscape of DES and showed that the effectiveness comes from the diverse modes reached by ensemble members, making it well suited for approximating BMA. It is quite frustrating that most sophisticated approximate Bayesian inference algorithms, especially the ones based on variational approximations, are not as effective DES in terms of exploring various modes in parameter spaces.
Despite being simple to train, DE incurs significant overhead inference time and memory usage. is therefore natural to develop a way to reduce such costs. An example of such method is Knowledge Distillation (KD) [Hinton et al., 2015], which transfers knowledge from large teacher network to a relatively smaller student network. The student network is trained with loss that encourages it to copy the outputs the teacher networks evaluated at the training data. With KD, there have been several works that learn single student network by distilling from DE teacher networks. A naive approach would be to directly distill ensembled outputs of DE teachers to the single student network. A better way proven to be more effective is to set up student network having multiple subnetworks (multiple heads [Tran et al., 2020] or rank-one factors [Mariet et al., 2020]) and distill the outputs of each ensemble member each subnetwork in one-to-one fashion. Nevertheless, the empirical performance of such distilled networks still far inferior to DE teachers.
Equal contribution
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Figure 1: It shows randomly selected train examples from CIFAR-10 and their corresponding predictions given by 8 ResNet-32 models. In the second row, the vertical axis denotes the ensemble members, and the horizontal axis denotes the class index.
There may be many reasons why the students are not doing as well as DE teachers. We argue that critical limitation of current distillation schemes is that they do not effectively transfer the diversity of DE teacher predictions to students. Consider an ensemble of deep neural networks trained for image classification task. Given sufficient network capacity, each ensemble member is likely to achieve mear-zero training error, meaning that the resulting outputs ensemble members evaluated the training set will be nearly identical, shown in Fig. 1. In such situation, reusing the training data during distillation will not encourage the student to produce diverse predictions. This is quite critical, considering various results establishing that the effectiveness of DE comes from averaging out its diverse predictions.
To this end, propose method that amplifies the diversity of students learning from DE teachers. Our idea simple: instead of using the same training set, use a perturbed training set for which the predictions of ensemble members are likely to be diversified. To implement this idea, we employ Output Diversified Sampling (ODS) [Tashiro et al., 2020], sampling scheme that finds small input perturbations that result in significant changes in outputs. Empirically, perturbed with ODS result in large deviations in the DE teacher outputs. We further justify our method show that distilling using training data
confirm that the inputs
by analyzing the role of ODS perturbation. Specifically, perturbed with ODS can be interpreted as an approximate Jacobian matching procedure where ODS improves the sample efficiency of the approximate Jacobian matching. Intuitively, by approximately matching Jacobians of teachers and students, we are transferring the outputs of teacher networks evaluated only the training data points but also on nearby points.
Using standard image classification benchmarks, we empirically validate that our distillation method promotes diversities in student network predictions, leading to improved performance, especially in terms of uncertainty estimation.
Backgrounds
2.1 Settings and notations
The focus of this paper is on the K- K-way classification problem taking D-dimensional inputs. We denote a student neural network as S(x)= IRP [0, 1] and teacher neural network as T(x), RD + [0, 1jK_ S(x) and T(x) outputs class probabilities, and we denote the logits before softmax S(x) and f(x). The ktt element of output is denoted ((R) (a:). For DE teachers with j ===,, 1, members, the jth ensemble member is denoted as (x)) For student network having M subnetworks, the jth subnetwork is denoted as (x)
2.2 Knowledge distillation
Knowledge distillation (KD) [Hinton et al., 2015] aims to train a student network S(æ:) by matching its outputs to the outputs of teacher network T(x). In addition to the standard cross-entropy loss between student outputs and the ground-truth labels, i.e., (CF(S(x)) y) - >k= ()()) log S(k) (a:, KD additionally encourages the student's 's output to be close possible to that of the teacher by also minimizing the KL-divergence between output class distributions of the student and the teacher:
T(x:
log softmax (4) S(xz)
(Sv(S(x), T(x):T)
softmax (k)
(1)

---

ODSdiivenfiiaation with
randomly selected (T...., T4)
OHR--O-GRR eusemble distillation
Ti (@)
(()))
EODS
T2
T2(@)
S2(Ø:)
ac
S
(a)
(a))
T
Ta(T)
(ar)
Figure 2: Overall structure of our method: (1) We first perturb train examples with respect to deep ensemble (DE) teachers, i.e,, Ø.. (2) DE teachers produce diverse predictions given perturbed train examples. (3) Since the diversity of teachers has been revealed, student can effectively learn its diversity in the knowledge distillation framework.
Here, T > is temperature parameter which controls the smoothness of the distributions. We scale the KL divergence by r² since the soft targets scale 1/r². As result, in the KD framework, the student imitates the teacher's outputs for training examples by minimizing L (1 - a)fcc aLKD- where a E (0, 1) is hyperparameter to be specified.
2.3 Batchensemble and one-to-one distillation
BatchEnsemble (BE) [Wen et al., 2020] is lightweight ensemble method that reduces the number of parameters by weight sharing. Specifically, each layer BE model consists of shared weights W and rank-one factors r for j -....,, M with M being number of subnetworks. Unlike the full ensemble that constructs full set of M weights, BE shares W across all subnetworks while the rank-one factors are kept specific to each subnetwork. The weight matrix of the jth subnetwork is computed as W o r and consequently BE requires far fewer parameters than full ensemble.
During training, BE simultaneously trains all subnetworks by feeding a duplicate of the current minibatch for each rank-one factor r,s; into the network. This procedure is easily vectorized because the computation graph for each subnetwork has the same structure, differing only in weight values. Mariet al. [2020] proposed to further improve the performance of BE by distilling from fully trained DE teachers instead of training from scratch. During distillation, the parameters of jth subnetwork Sy (x) trained by distilling from the jth DE teacher (x).. The corresponding objective function is then written as (c.f., Fig. 2 without perturbation)
M Σ (cs
Throughout the of the paper, will use the student networks and follow this one-to-one distillation framework.
2.4 Output diversified sampling (ODS)
ODS [Tassiro et al., 2020] is a sampling method that can maximize diversities in outputs of a given function; it is originally introduced to replace random input perturbation to enhance the performance of the adversarial attack. Let F R + R be a target function. ODS draws a perturbation vector guided from the gradient of the target function follows.
w ~ 11), Eons (2: F, = Va(w T F(x))
[V F(x))]l, R
(3)
The intuition behind this sampling strategy is to seek the direction in the input space that maximizes the similarity between the function output and a randomly sampled vector w. Following such direction can make diverse output shifts due to the randomness in w.

---

3 Learning from Ensembles with Output Diversification
3.1 One-to-one distillation with ODS
As discussed before, vanilla one-to-one distillation evaluates DE teachers on points where their outputs are nearly identical, the students trained using them tend to exhibit low diversity in outputs. To avoid this undesirable behavior, we propose to instead train students on points perturbed with ODS. Let (x, y) be training input pair. We first pick random teacher T uniformly from (Ti,.... Tm
and perturb x as
(4) where 1/ > 0 is a step-size. Ideally, we would like to evaluate ODS vectors for each teacher to build M perturbed versions of æ, but this would take too much time for gradient computation, especially when we have a large number of ensemble models. Instead, we just pick one of the teachers, use it to perturb the input, and share the perturbed input to train all students with the following loss function,
= x + 1)oDss (æ, softmax (T,(()))) w)
M L= ((a-
(S), (())))
where we are evaluating the KL-divergence between teacher outputs and student outputs on the perturbed input.
An implicit assumption here is that a diversifying direction computed a specific random teacher can bring diversities in the oulputs of all the teacher networks. This is based the transferubility assumption empirically justified in Tashiro et al. [2020]: :: ODS perturbations are computed on a surrogate model when we do not have access to the true model, assuming that the Jacobians computed from the surrogate model can approximate the true Jacobian to some extent. In our case, let Ti be randomly picked teacher with which an ODS perturbation is computed, and T2 be another teacher. Assuming the transferability of Jacobian, we can let V (xx) s RV (2(x) for some matrix R. Then the ODS perturbation computed from (x) is
EODS (z, Tı, w) x T Ti (R' w) TV) T2(x)) x cøns( T2, R w).
(6)
That is, the ODS perturbation computed from T1 acts as another ODS perturbation with different random guide vector R² w. Hence, the same perturbation drives the outputs of Ti and Tz towards different directions. Note that without transferability, the Jacobians (x) and 72 would be vastly different and
the directions w and R This may be good in terms of diversification, but the resulting perturbed input act as an adversarial example not very useful for distillation, making the teachers completely disagree (refer to Appendix A.2 for more details). Our experiments in Section 5.2 and Section 5.3 empirically re-confirm this transferability assumption.
We can also interpret the random teacher selection as a stochastic approximation. Specifically, to find a direction maximizing diversities between teachers, we can consider an objective
E Emy
Σ w Tar))
with Wi..... WM i.d. 11, That is, we want to find a direction in the input space = follow different guide vectors { 1 Picking a random teacher and
making the teachers (T computing the ODS perturbation can be understood as a stochastic approximation this objective. Again, thanks the transferability of the Jacobians, the variance of this stochastic gradient would be small, making it sufficient to use a single random teacher.
the gradient of
In addition, we empirically found that scaling the ODS updates according to confidence slightly improves performance. That is, we perturb an input æ as
(8) where Cmax(x, Ty, T) = max; softmax (k) (T. (x))T) denotes the maximum class probability. This scheme takes larger steps towards ODS directions for datapoints with high confidence. We call this modification ConFODS
T/Cnua (x, Tr,7) Tonss (x, (oftmax(7 w),
The training procedure with ODS perturbation is straightforward (Algorithm 1); we can just add the ODS computation procedure and replace the loss function from the original KD training procedure. Fig. 2 depicts our overall training procedure as diagram.

---

Algorithm Knowledge distillation from deep ensembles with ODS perturbations
Require: Training data D {( v,)),=
Require: Knowledge distillation weight a, temperature т, learning rate B, step-size 7)-
Require: Deep ensemble teacher networks (T, 1==1 where M denotes the size ensembles. Let S be student network which consists of M subnetworks (S, [5,))=1- Initialize parameters while not converged do
the student network S.
Sample (x, y) D, w Uni
and teacher index uniformly from (L....,M) M). Sample diversified inputs á = x + 7eoos (z, (fft)ax(TT(==)1)))
LCB < - En Ex=1 y log St (x)
CKD + - Σe k- softmax (T()) (S
(1 - 0 - 0 Ve
LCE + aCCD)
end while
3.2 Interpretation as approximate Jacobian matching
We can also interpret distillation strategy Srinivas and Fleuret [2018] shows that a KD procedure on inputs perturbed by small noise implicitly encourages matching the Jacobians of teacher and student. More specifically, let S and T be student and teacher models. The first-order Taylor expansion of the expected KD loss on perturbed inputs be written as!
approximate Jacobian matching procedure.
Ee Te[CkD(S(g) + +)), +)] (kD(S(x), +rtt::/---------1---------------
o(o2),
(9)
where 0² -EeeT E] and
K
S (ik
(m(S(Z), EE
(10)
While Srinivas and Fleuret [2018] suggested using isotropic Gaussian noise E o2 we argue that simple Gaussian noise is ineffective, especially for digeeniimnsional data that are likely to be embedded low-dimensional manifolds. In such case, most perturbation directions are orthogonal to the data manifold, and thus Gaussian noise becomes uninformative. On the other hand, ODS directly uses the Jacobian of the teacher to construct the perturbation direction. We conjecture that due to this fact, KD with ODS perturbation is a much more sample-efficien efficieff approximate Jacobian matching procedure compared to using Gaussian perturbations. We perform experiments to empirically validate this claim in Section 5.3.
The above analysis is for the usual KD framework with single teacher and single student. In case, we are matching M teachers to M student subnetworks, but with ODS perturbations computed from single teacher randomly sampled from M teachers. Again, we are operating under the assumption that the Jacobian computed from a specific teacher will transfer well to other teachers, so the approximate Jacobian matching using is still more effective compared to simply using Gaussian perturbations.
4 Related Works
Knowledge distillation KD is method for transferring the information inside large model into smaller one [Hinton et al., 2015] by minimizing the KL divergence between teacher and student networks. [Wang et al., 2020] finds that additionally augmenting the data results in further performance gains because such augmentations allow the KD loss to tap into teacher information outside of the training set. Furthermore, modifications to the standard KD objective allow one to distill the knowledge in an ensemble of teachers into one student network while preserving the benefits in uncertainty estimation of ensembles [Malinin et al., 2020, Tran et al., 2020]. KD can enhanced via gradient matching, and several existing methods [Srinivas and Fleuret, 2018, Czarnecki et al., 2017] propose sampling methods for knowledge distillation up to higher-order gradients. However, such existing methods suffer from high computational burden inefficient random perturbation.
We assume 1 for notational simplicity, but the argument applies equally well to any
5

---

Ensembles Ensemble methods [Hansen and Salamon, 1990, Dietterich, 2000], which construct set of learners and make predictions on new data points by a weighted average, have been studied extensively. Ensembles perform at least as well as each individual member [Krogh and Vedelsby, 1994], and achieve the best performance when each member makes errors independently. Aside from benefits in model accuracy, DE TLakshminarayanan et al., 2017, Ovadia et al., 2019] have recently shown to be simple and scalable alternative to Bayesian neural networks because of their superior uncertainty estimation performance. Wen et al. [2020] proposes network architecture that uses rank-one parameterization to train an ensemble with low computation and memory cost.
5 Experiments
In this section, we try to answer the following questions with empirical validation.
Do images perturbed by ODS increase diversity in DE teacher predictions? - Section 5.2.
Do ODS perturbations encourage Jacobian matching? Section 5.3.
Does enhanced diversity of students lead to improved performance in terms of prediction accuracy and uncertainty estimation? Section 5,4 and Appendix A.I.
5.1
Experimental setup
Datasets and networks We compared our methods on CIFAR-10/100 and TinylmageNer. We used ResNet-32 [He et al., 2016] for CIFAR-10, and WideResNet-28s. 10 [Zagoruyko and Komodakis, 2016] for CIFAR- and Imay Please refer to Appendix B. for detailed training settings.
Hyperparameter settings The important hyperparameters for KD are the pair (a, r); for CIFAR- 10, after a through hyperparameter sweep, we decided stay consistent with the convention of (Q,T) = (0.9, 4) for all methods [Hinton et al., 2015, Cho and Hariharan, 2019, Wang et al., 2020]. For CIFAR-100 and TinylmageNet, used the value (ax r) (0.9, 1) for all methods. We fix ODS step-size to 1/255 across all settings. See Appendix B. for more details.
Uncertainty metrics We measure uncertainty calibration performance with common metrics in- cluding Accuracy (ACC), Negative Log-Likelihood (NLL), Expected Calibration Error (ECE) and Brier Score (BS). We also report the Deep Ensemble Equivalent (DEE) score [Ashukha et al., 2020], which measures the effective number of models by comparing it to a full DE. For all methods, the metrics computed at optimal temperatures obtained from temperature scaling [Guo et al., 2017], as suggested by Ashukha et al. [2020]. See Appendix B.2 for the definition of each metric.
5.2 Impact of ODS on diversities
In order to assess the effect of ODS perturbation on the diversities of DE teachers or student subnet- works, we draw diversity plots in Fig. 3. To draw a diversity plot, we first collect the outputs of target models evaluated on specific dataset and binned the samples w.r.t. their confidence values (minimum confidence among M models). For each bin, we computed average pairwise KL-divergence between class output probabilities of ensemble members. We also report the representative value (mean-KLD), which is defined as average KL Kivdiveree values weighted by bin counts.
Fig. 3a shows the diversity plots of DE teachers on 100. Intuitively, the samples predicted with high confidences usually come with low diversities, and this is indeed depicted in the plots. For the DE evaluated on the training set, the mean- KL.D value is very small (the first box) compared to the one computed on the test set (the fourth box). While Gaussian perturbation does not affect the diversities (the second box), ODS drastically amplifies the diversities (the third box).
Fig. 3b shows the diversity plots of the BE students trained on CIFAR-10. Although evaluated at the test set, the BE distilled without any perturbation (the first box), the BE distilled with Gaussian perturbations (the second box), and the BE trained from scratch (the fourth box) exhibit very low diversities. On the other hand, the BE distilled with ODS perturbations show relatively high diversity, and will show later in Section 5.4, this indeed leads to improved performances.
In Section 3, we assumed that an ODS computed from a specific teacher can be transferred to other teachers. As empirical justification, we draw the diversity plot using ODS computed from an

---

DeepEns-4 Train
DeepEns-4
DeepEns-4 Train
(0DS perturbad)
DeepEns-4 on Test
255
Nean-D.D 0.002
Nemn-LD 6.055
Nem--L. 0.478
Nean-LD 0.25
dvwrdly (0
162
0.5
0.5
0.5
0.5
0.0
1c.4
0.6
0.6
1.0
0.6
08
1.0
0.6
0.8
1.0
0.6
0.8
1.0
mihimum confidence
mirimum confidence
mirimum
mihimum confidence
(a) Diversity plots of DE-4 teachers for ResNet-32 train examples of CIFAR-10.
BatchEns on
BatchEns-4 on (KD
BatchEns-4 on
BatchEns-4 on Test
Svandyy (01)
Meam-D 0.008
Mean-0 D 0.011
Kean-D B 0.004
NeanaD D 0.014
162
0.2
02
0.2
0.2
0.0
0.0
0.0
0.0
10.
0.6
0.8
1.0
0.
1.0
0.6
0.8
1.0
0.6
0.8
1.0
mihinum confidance
mirimum confidance
confidance
(b) Diversity plots of BE-4 students for ResNet-32 on test examples of CIFAR-IO.
Figure 3: Bar colors denote the density of the bins, ie., ratio of samples belonging to the bins.
external teacher network trained in the same way the teachers but not actually being used during distillation) in Fig. 6a. The mean-KI.D values are indeed increased compared to the original DE or DE evaluated on examples perturbed with Gaussian noise, confirming the validity our assumption.
5.3 ODS for Jacobian matching
Effectiveness We empirically validate our conjecture that ODS perturbations are more effective Jacobian matching. To this end, we computed the cosine similarities between the vec- torized Jacobians of ResNet-32 teacher and student networks trained with vanilla KD. We computed the same quantity us- ing student networks distilled with Gaussian perturbations and ODS perturbations. We draw Receiver Operating Characteristic (ROC) curves to compare the similarities of distilled students and the baseline trained from scratch. Specifically, we compare two distributions of the cosine similarities between the vector- ized Jacobians of teacher network and student network: (1) when the student network is trained by KD, and (2) when the student network is trained from scratch without any guide from the teacher network. Results in Fig. 5 clearly show that the Jacobians of the student distilled with ODS better match the Jacobians, while the student distilled with Gaussian perturbations did not significantly improve upon the student distilled with vanilla KD
-0.1
-02
-X- Gaussian
-03
-X-- ODS
1/256
2/256
3/255
4/255
slep-size
Figure 4: SNR values computed with Gaussian and ODS perturba- tions with varying step-sizes.
Next, we compute the gradient of the term related to the Jacobian matching, i.e.,
G( E) Ve (e(C s), T(x + +)) ((o((((),T(()))
(11)
Using both Gaussian perturbations and ODS perturbations, we collected multiple gradient samples and measured the Signal-to-Noie toise Ratio (SNR) of the gradient norms. Specifically, computed SNR(#, e) ||LE [vec(G(æx, e))][l / /II Var (vec(G(a, {<)]l>)]] where vec(-) is vectorization and Var[-] is dlement-wise variance, As shown in Fig. 4, the gradients computed with ODS perturbations exhibit higher SNR, indicating that learning with such gradients is more stable and efficient.
Transferability As in Section 5.2, we performed an experiment to verify our assumption that ODS perturbations for specific model are transferable to other models, for the purpose approximate Jacobian matching. We trained a BE student with ODS perturbations computed from an external teacher, and measured the cosine similarities between the student and teachers (not between the students and the external teacher). As one can see from Fig. 6b, the resulting student network copied the Jacobians of the teachers significantly better than the baselines.
7

---

D 0.177
0.5
0-2
15
0.2
6:7
67
06-
3.00
2.23
6,92
0.75
1.00
0.50
0.25
0.50
5.73
3,00
0.00
0.25
0.50
5.75
3,00
0.0
0.2
0.4
0.0
0.8
1.0
(C0S AUROCHO
500
Basdiin 220000.37. ALROC-
PRTATII COnEidence
Basdine (CDS=0,373 ALROCHS, 5021
Voni 1000-0..27, ALROCHO.
Panill (cope: MJ
Vanilla (CO5-3228, 1.70
Gazaion 1055-0..18, AUROC+B.2544
Casaaian (C25-e.220) AUROC-0
Cocoalan (CO5m2.223 AUROCHO
104
10-2
(a) Training set
(b) Validation set
(a) Diversity plot
(b) ROC-curve
Figure 5: ROC-curves with cosine similarities of Jacobians between a teacher and students. 'COS' denotes the cosine similarities between the teacher and student averaged over examples, and 'AU- ROC' denotes the area under the ROC-curves.
Figure 6: Transferability analysis using diversity
plot and ROC-curve. Here, BE student is trained with ODS perturbations computed from an exter- nal teacher, and it is denoted as 'ExIODS' in the ROC-curve for validation set.
Table 1: Knowledge distillation from DE-M into BE-M where M denotes the size of ensembles: ACC and calibruted metrics including NLL., BS, ECE, and DEE. All values for ResNet-32 and WideResNet- 28x 10 are averaged over four and three experiments, respectively.
BatchEns = DeepEns (ResNet-32 on CIEAR-10)
Method
# Params
ACC (1)
NI.L (1)
BS ())
ECE (
DEE (1)
T: DeepEns-4
1.86 M
94.42
0.167
0.082
0.008
S: BatchEEns-4
0.47 M
93.37.10.11
0.20411.0002
0.000
0. 0.008100.00
1.419
+ KD
93.98=a0.20
0.188
0.091=a.002
0. 0.009#0.002
2.019=0.174
+ KD + Gaussian
03.93.10.11 12 12....
0,187
0.090-..001
0. 0,00910 002
2.042
+ ODS
93.89+00.10
0.181
+0.002
0.090=0.001
4.006=o.000 001
2.486=0.100
ConfODS
94..01 +0:18
0. 0.180
0.089=0.00 wom
0.007:00000
2.524=0.000
T: DeepEns-8
3.71 M
94.78
0.157
0.077
0.005
BatchEEs-8 KD
0.48 M
93.47.10.14
0.20211.000
0.098 0.002
0.00611.000
1.494
14 13
0.18220.00
0.088=0.001
0.0 1.01000:001
2.39
KD Gaussian
94.09=0.00
0.1844
0.089=0.001
0.010+0.002
2.206=00115
KD ODS
14.13+0.00
0.175+0.000
0.086=00.01
0. 00.0600022
2.991 =0.322
KD ConFODS
94.18
0.17410.002
0.086 10.001
0. 1.000110.00 001
3.064
BatchEns DeepEns (WRN-28x10 on CIFAR R-10)
Method
# Params
ACC (1)
NLL ())
BS (L)
ECE ())
DEE (1)
T: DeepFns-4
146.15 M
82.52
0.661
0.247
0.022
S: BatchEns-4
36.62 M
80.3440.08
0.755+0.007
0.280=0.002
0.027=0.00:
1.449
+ KD
80.5110.22
10.04
0.274_a.001
0. 10.00s
1.58210.035
+ KD + Gaussian
80 30.3910.114
0.76110.00
0.277=0.000
0. 0.022210.00 00.000
1.379
KD + ODS
88 52
0.67420.016
0.257 0.008
0. 0.026 003
3.303
+ + ConFODS
$1.85+00.32 32
0.672+0.030
0.256=1..00m =0.00
0.024+0.000 000
3.333
5.4 Main results: image classification tasks
We validate whether the enhanced diversity of student networks helpful for actual prediction. We compared our methods against three baselines: BE trained from scratch [Wen et al., 2020], BE distilled without perturbation [Mariet et al., 2020], and BE distilled with Gaussian noises.
The results are presented in Table 1. For both datasets, BE distilled with ODS or ConFODS significantly outperforms baselines in terms of every metric we measured. The improvement is much more significant for the larger ensembles (M = 8 for CIFAR-10) and the larger dataset (CIFAR-100 100)..100). We note that using our diversification strategy improves performance both terms of raw accuracy and uncertainty metrics. An especially noteworthy result on CIFAR-100 is that BE distilled with ODS achieved DEE score over three. This is quite remarkable considering the BE model has fewer parameters than the full DE which, by definition, has expected DEE of four. Moreover, further uncertainty results in the Appendix A. show that our approach is superior with respect to both predictive uncertainty for out- out-of-distribution examples and calibration on corrupted datasets.

---

Table 2: Cross-architecture knowledge distillation for model compression on CIFAR-100 and Tiny ImageNet: BE-4 students with smaller networks (i.e., WideResNet-288x and WideResNet-28kx learn from the DE-4 teacher which consists of larger networks (iee, WideResNet-28x 10). values for CIFAR-100 and TinylmageNet are averaged over three and one experiments, respectively.
BatchEns-4 a DeepEns-4 (WRN-28x2 on CIFAR-100)
Method
Params
(T)
NL).
(L)
(4)
T: DecpEns-4
146.15 M
82.52
0.676
0,250
0.035
S: BatchEEns-4
1.50 M
75.17=0.27
1.245+0.0224
0.383=0.004
0.141
+ K.D
75. 75.19+0.38
1.207=0.021
0.377=0.007
0.136+0.000
+ K.D. + Gaussian
74_50=0.17
1.247=0.012,
0.389=0.00d
0.137 1..3000008,
+ KD ODS
76 03 10.22
0.899 20.018
0.333 0.0003
0 12072070000 0.0002
+ KD + ConTODS
76.01+0.16
0.901 + 0.000
1.333==0.002
0.02840.004
BatchEns-4 DeepEns-4 (WRN-28x5 on CIFAR-100)
Method
Params
ACC (t)
NLL (L)
BS (J)
ECE (L)
T: DeepEns-4
146.15 15 M
82.52
0.661
0.247
0.022
S: BatchEns-4
9.20 M
78.7520.11
00:012
0.297=0.000
0.02110.000
KD
78.8910.10
0.804
0.296_0.000
0.022 10.001
KD Gaussian
78.8010.41 0.41
0.815 p. 009
0.297
1297_00.000
0.0201...0
KD ODS
80.24.0.05
0.742
0.279=0-002
0.028-20.04
KD ConfODS
80. 20.25
0. 0.733 000007
0.275 20.005
0.027
BatchEns-4 DeepEns-4 (WRN-28x5 TinylmageNet)
Method
# Parums
ACC (t)
NLL (4)
BS ())
ECE (L)
T: DecpEns-4
146.40 40 M
69.90
1.242
0.403
0.016
8: BatchEns-1
9.23
64.86
1.455
0.464
0.022
KD
65.86
1.432
0.456
0.022
KD Gaussian
65.72
1.446
0.457
0.022
KD ODS
65.96
1.408
0.456
0.022
One great benefit the KD framework is that it can transfer knowledge to networks having different architectures, greatly reducing the number of parameters required to achieve certain level of performance. Using CIFAR-100 and TinylmageNet datasets, we compared our method to baselines with student networks having smaller architecture. The results displayed in Table 2 show that KD with ODS or ConfODS outperforms the baselines by a wide margin.
Conclusion
In this paper, we proposed simple yet effective training scheme for distilling knowledge from an ensemble of teachers. The key idea of our approach is to perturb training data in proper way to expose student networks the diverse outputs produced by ensemble teachers. We employed ODS to find such diversifying perturbation. Our distillation scheme with ODS perturbation can be interpreted as a sample-efficient approximate Jacobian matching procedure. We empirically verified that our method enhances the diversities of students, and that such enhanced diversities help to reduce the performance gap between student and teacher models in KD. interesting future work would to rigorously analyze the Jacobian matching aspect of our method and to provide theoretical justification and a better distillation method to harness it. Also, in this paper, we constrained ourselves to BE students, and applying our framework other student network architectures such as Bayesian neural networks would be interesting.
Limitations Although have shown promising results, our method still has several limitations. Due to its nature as a KD framework, the need to train M teachers involves a considerable com- putational cost. Also, method cannot improve the student beyond its capacity. For instance, in Table 2, WideRessNet28x22 distilled from WideResNet-28x 10 does not greatly benefit from our method. Arguably the most critical limitation is that method is heuristic and lacks any theoretical guarantee. In that sense, as mentioned above, rigorously analyzing the effect of ODS perturbation will be important future research.

---

Acknowledgement
This work was partly supported by Institute of Information communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2019-0-000)) Artificial Intelligence Graduate School Program(KAIST), No. 202 100-0002, Artificial Intelligence Innovation Hub), National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF- 2021R REDRRFIAAIE 1061655, NRF-200IM3355090250000 and Samsung Electronics Co., Ltd (IO201214- 08176-01).
References
A. Ashukha, A. Lyzhov, D. Molchanov, and D. Vetrov. Pitfalls of in-domain uncertainty estimation and ensembling in deep learning. International Conference on Learning Representations (ICIR), 2020.
H. Cho and Hariharan. On the efficacy of knowledge distillation. In IEEE/CVF International Conference Computer Vision (ICCV), 2019.
W. M. Czarnecki, S. Osindero, M. Jaderberg, G. Swirszez, and R. Pascanu. Sobolev training for neural networks. In Advances in Neural Information Processing Systems 30 (NIPS 2017), 2017.
G. Dietterich. Ensemble methods in machine learning. In Proceedings of the First International Workshop on Multiple Classifier Systems, 2000.
D. Duvenaud, D. Maclaurin, and R. Adams. Early stopping as nonparametric variational inference. In Proceedings of The 19th International Conference on Artificial Intelligence and Statistics (AISTATS 2016), 2016.
S. Fort, H. Hu, and B. Lakshminarayanan. Deep ensembles: A loss landscape perspective. arXiv:1912.02757, 2019.
C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. calibration of modern neural networks. In Proceedings of The 34th International Conference Machine Learning (ICML 2017), 2017.
L. Hansen and P. Salamon. Neural network ensembles. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1990.
M. Havasi, R. Jenatton, S. Fort, J. Z. Liu, J. Snoek, B. Lakshminarayanan, A. M. Dai, and D. Tran. Training independent subnetworks for robust prediction. In International Conference on Learning Representations (ICLR), 2021.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2016.
D. Hendrycks and T. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. International Conference on Learning Representations (ICLR), 2019.
G. E. Hinton, O. Vinyals, and Dean. Distilling the knowledge in a neural network. In Deep Learning and Representation Learning Workshop, NIPS 2014, 2015.
A. Krogh and J. Vedelsby. Neural network ensembles, cross validation and active learning. In Proceedings of the 7th International Conference on Neural Information Processing Systems, 1994.
B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems 30 (NIPS 2017), 2017.
A. Malinin, Mlodozeniec, and M. J. F. Gales. Ensemble distribution distillation. In International Conference Learning Representations (ICLR), 2020.
Mariet, Jenatton, F. Wenzel, and D. Tran. Distilling ensembles improves uncertainty estimates.
3rd Symposium Advances in Approximate Bayesian Inference, 2020.
10

---

Y. Ovadia, E. Fertig, J. Ren, Z. Nado, D. Sculley, S. Nowozin, J. V. Dillon, B. Lakshminarayanan. and J. Snock. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems 32 (NeurlPS 2019), 2019.
S. Srinivas and F. Fleuret. Knowledge transfer with jacobian matching. In Proceedings of The 35th International Conference on Machine Learning (ICML 2018), 2018.
Y. Tashiro, Y. Song, and Ermon. Diversity can be transferred: Output diversification for white- and black-box attacks. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 2020.
L. Tran, B. S. Veeling, K. Roth, J. Swiatkowski, J. V. Dillon, J. Snock. S. Mandt, T. Salimans, S. Nowozin, and R. Jenatton. Hydra: Preserving ensemble diversity for model distillation. arXiv:2001.04694, 2020.
H. Wang. S. Lohit, M. Jones, and Y. Fu. Knowledge distillation thrives on data augmentation. arXiv:2012.02909, 2020.
Y. Wen, D. Tran, and J. Ba. Batchensemble: an alternative approach to efficient ensemble and lifelong learning. In International Conference Learning Representations (ICLR), 2020.
A. G. Wilson and P. Izmailov, Bayesian deep learning and probabilistic perspective of generalization. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 2020.
S. Zagoruyko and N. Komodakis. Wide residual networks. In Proceedings of the British Machine Vision Conference (BMVC), 2016.


Reviews:
[{"review_id": "i-clwMQqIO", "reviewer": "Reviewer_z2qY", "summary": "This paper focuses on the topic of effectively distilling ensembles of models into a single model. Assuming that the distilled model should inherit the diversity of the ensemble, they attempt to make the distilled performance more comparable to the ensemble by using a (adversarial) perturbation strategy that directly identifies sources of diversity within the ensemble (i.e., where models within the ensemble disagree with each other). The authors demonstrate empirically that utilizing such a diversity-inducing objective within distillation improves the final performance of the distilled model in comparison to the ensemble on image classification experiments. They motivate this objective for distillation with intuitive descriptions of the importance of diversity within an ensemble, as well as by making connections to gradient matching between the student and teacher. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I begin my review by emphasizing that I am completely open to authors' comments and feedback in regard to my review. My final score will be mostly based upon discussion with authors in regard to my questions and initial thoughts on the manuscript.\n\nPros:\n- A reasonable explanation is provided for the gap in distilled network performance in comparison to the ensemble that is supported by evidence. I found this discussion to be intuitive and useful.\n- This method reduces the significant overhead (memory and computation) of ensembles by only using a single model, but narrows the gap in performance between the model and the ensemble.\n- Authors make a connection to gradient matching that is explained well and supported empirically. This is a nice interpretation/understanding of why the method works.\n- The methodology seems to consistently improve the performance of distilled models in comparison to distillation with gaussian noise or normal distillation procedures.\n- Paper is generally well-written, and it seems like the authors have spent time to polish it.\n\nCons:\n- There is not much performance improvement on CIFAR10. It would be good to replace this with a larger dataset (or simply add a larger dataset).\n- Only one method of input perturbation is attempted. It would be nice to have justification for why ODS is the best perturbation strategy and comparisons to other options in the experiments. I see that you show Gaussian perturbation does not induce diversity, but it seems that there should be numerous other perturbation strategies that would induce a similar result to ODS.\n- No large-scale classification experiments (e.g., ImageNet). I believe running experiments on ImageNet for this work is important, as it is unclear from experiments whether your approach will scale to datasets with a significantly increased number of classes (e.g., 1000 instead of 100). This is especially true because a significant performance difference is seen between 10 and 100 class cases. It seems like ODS has been tested on ImageNet by Tashiro et. al., however. \n- The experiments focus on one knowledge distillation strategy (one-to-one batch ensemble distillation). Is it reasonable to test some other methods of distillation with a similar ODS approach?\n\n\nQuestions:\n- Are adversarial perturbations the most appropriate kind to use within this methodology? I am not fully convinced that adversarial perturbations will capture diversity in the ensemble that has tangible, semantic meaning, but rather small/imperceptible changes that drastically modify network output. \n- Assumption that diversifying directions transfer between teachers is fully justified. (But, how do we choose which teacher to use? Does it matter?)\n\nSmall Comments:\n- You probably need a section on adversarial attacks on neural networks in the related work, as the perturbation strategy for this work is inherited from an adversarial attack paper. \n- It would be nice to provide some clearer explanation of why you choose the exact distillation strategy that you use. For me, it is not immediately clear whether using batch ensemble and one-to-one distillation is a reasonable choice (i.e., is this widely considered the SOTA approach?)\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I address limitations within my main review (authors also address limitations in the paper directly). There is no negative societal impact. ", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "Yvzs60eWV8C", "reviewer": "Reviewer_p6SF", "summary": "This paper proposes to improve Knowledge Distillation (KD) for ensemble models. The authors apply ODS [Tashiro et.al, 2020] to each teacher model in the ensemble to obtain perturbed data. Thereafter, the perturbed data is used to train student models, each corresponding to one teacher model. The authors argue that ODS can improve the diversity among student models in the ensemble.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main contribution is applying an existing technique, ODS [Tashiro et.al, 2020] to a non-trivial problem of ensemble model distillation. The abstract (line 12) promises to find input samples that teacher models' outputs disagree. Then I would expect one has to feed all teacher models' outputs into the process of seeking the perturbed inputs. But algorithm 1 treats each teacher model separately in the process. It seems to me that the algorithm 1 completely relies on the diversifying ability of ODS, and assumptions made in line 113-114. \n\nHowever, assumption in line 113-114 is quite counter-intuitive to me. Usually, a wise design of ensemble model is to include diversified member models. In this case, this means that the teacher models are very different from each other. If so, intuitively, a perturbation that maximumly change one teacher's decision does not necessarily change another's. For the similar reason, the assumption made on Jacobians (line 143) seems too strong, and most likely cannot hold in practice. This concern challenges the correctness of using one teacher model to derive perturbation vectors for all teacher models (line 110-111).\n\nThe paper has many clarity issues. Below is an incomplete list of them:\n\nline 96-97: It is not very clear how ODS works. Which one is the perturbation vector? $w$ or $v_{ODS}$? I assume it is $v_{ODS}$, after checking the seminal work [Tashiro et.al, 2020]. Please improve the presentation, e.g., including equation (3) from the seminal paper.\n\nBefore Equation (9), please define $\\epsilon$ as the perturbation in this specific context. Using multiple perturbation vectors, $v_{ODS}$ and $\\epsilon$, may be hard for readers to understand.\n\nEquation (10) is taken from equation (2) of [Srinivas and Fleuret, 2018]. It is therefore not this paper's contribution, and the authors should say that in the paper. Also, the notation is very informal. What does $\\frac{J_s(x)}{S(x)}$ mean? a matrix divide by a vector?\n\nLine 194-195: I cannot find a definition of \"confidence value\". Is it the predicted probability of the correct class label? Please define it first. Fig 3 seems to be very important, as it fulfills the promise made in abstract (line 12). However, the setup from line 192-198 is hard to read, which made it hard to judge the correctness and significance of fig.3.\n\nLine 224: It is not clear what the ROC curve means. Also how is it obtained? I cannot find any description in the main paper or supplementary material.\n\nLine 234-235, why higher SNR is more effective?\n\nOverall, the paper does not make it crystal clear why diversity among teacher models are signified/promoted.\n\nThrough multiple rounds of Q&A, I think correctness is now justified, and I appreciate the authors' efforts in clarification. Nevertheless, the current draft needs to include all the key points during the discussion process. For example, the draft itself needs heavy revision to manifest how diversity among ensemble members is achieved. I now raise the score to 5, due to the fact that such a revision may require another round of reviewing.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "Sst46rOoLYT", "reviewer": "Reviewer_1b8g", "summary": "The paper studies the question of ensemble function diversity and how to preserve it on ensemble distillation to a single model. The authors use input perturbations that make the models, especially well trained models that are near 0 loss on the training set, disagree on the input, and then use knowledge distillation from individual members of the ensemble to individual subnetworks of the distilled model. The authors show that without this approach, the distilled models suffer from lack of function diversity, and that their approach enhances it and in accordance with their theory enhances performance as well.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I like the paper. It is well written, deals with an important question that is of interest to the community, and presents its approach in an easy to understand way. A small caveat is that despite following the subfield pretty closely I might not be able to judge the originality of the approach to the full extent and will likely defer to other reviewers if they have substantial points on this.\n\nSTRONG ASPECTS:\n1) the paper is very clearly presented and was easy to read and understand\n2) the problem the authors are looking at and addressing is important and broadly practically relevant\n3) the technique is simple and could be implemented relatively easily (maybe apart from the gradient-informed input perturbation, which requires access to the models in the ensemble to be distilled)\n4) I like the gradient matching perspective on why this technique could be working well\n\nQUESTIONS AND CLARIFICATIONS:\n1) what other approach to making sure that function diversity is transferred to the distilled model are there? did you compare your technique to them?\n2) In Figure 2 and in the subsequent text it seems to be assumed that you are distilling an ensemble of models to a single model that still has a distinct number of subnetworks but is cheaper (for example the batch ensemble you are discussing). Could you approach work for actually distilling to a single model without this explicit multiple-subnetworks structure.\n3) What is the advantage of using the gradient-informed perturbations in Equation 5 over the uniform perturbations? You mention that the random perturbations will be bad if the underlying data is low dimension,which I understand. This paper https://openreview.net/forum?id=XMoyS8zm6GA seems to suggest that they are actually very high dimension, which might make the random perturbations similarly good?\n4) Could you perhaps use a MIMO-like architecture to be distilled into as well? I found this one by Havasi et al to be pretty high performance https://arxiv.org/abs/2010.06610 (only if you feel this might be a good idea, I know this is beyond the scope of your paper)\n5) Aren't the perturbations in Eq. 5 pretty similar to adversarial perturbations, just with a target \"label\" being the random vector w?\n\nWEAKER ASPECTS:\n1) the experimental verification would be more convincing if you also looked at ImageNet. I know that this can be though, and I won't be decreasing my score based on it, but it could help a lot with the future adoption of your technique.\n2) In Table 1 the models on CIFAR-100 reach only low 80% test accuracy. I think it would be good to show that your conclusions also work for more powerful SOTA models like the ViT that easily reach 90%+ test.\n\nOverall, I like the paper!\n\nUPDATE AFTER REBUTTAL:\nI am very happy with the authors' detailed response and would like to increase my score from 6 to 7 to reflect that. Thank you very much and great job on the paper!\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "GbtBsf_XIc", "reviewer": "Reviewer_7xWm", "summary": "This paper attempts to reduce the performance gap between an ensemble and a model that was distilled from it. The main idea is that, when distilling, the model should “absorb as much function diversity inside the ensemble as possible”. To do that, the paper proposes using ODS perturbation strategy, that samples diverse predictions for a given input form the ensemble, to use as label during distillation.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The ideas proposed are neat and well motivated by previous literature finding that prediction diversity is important for ensemble performance. I found particularly interesting the results that ODS sampling is approximately gradient matching, and that by doing gradient matching, the distilled model is encouraged to behave like the teachers on “nearby points”.\n\nMy main concern with the paper is that most of the experiments are performed on CIFAR-10 and CIFAR-100. I would like to see experiments on a more realistic benchmark, such as ImageNet, to assess the generality of the conclusions.\n\nI would also be interested in seeing other ensembling methods tested, such as EMA.\n\nOtherwise, I found the paper to be easy to read, well explained, and the analyses to be insightful.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a method for distilling an ensemble to a single model by training on inputs which are perturbed to make ensemble member outputs disagree. Most reviewers (1b8g, 7xWm, z2qY) seemed to agree that the method was sensible, elegant, and clearly-presented, and that the gradient-matching intuition was clever and well-supported. Reviewer p6SF raised significant concerns about correctness of the authors' transferability assumption, but after rebuttal and discussion these were mostly addressed. p6SF also raised concerns about clarity, but these too were partially addressed and not in my opinion sufficient to justify rejection. Another common concern (1b8g, 7xWm, z2qY) was the lack of large-scale experiments, but the authors address this with TinyImageNet experiments in rebuttal to the satisfaction of multiple reviewers (1b8g, z2qY). Therefore I recommend acceptance.
Decision: Accept (Poster)