Diversity Matters When Learning From Ensembles
Year: 2021

Abstract:
Deep ensembles excel in large-scale image classification tasks both in terms of prediction accuracy and calibration. Despite being simple to train, the computation and memory cost of deep ensembles limits their practicability. While some recent works propose to distill an ensemble model into a single model to reduce such costs, there is still a performance gap between the ensemble and distilled models. We propose a simple approach for reducing this gap, i.e., making the distilled performance close to the full ensemble. Our key assumption is that a distilled model should absorb as much function diversity inside the ensemble as possible. We first empirically show that the typical distillation procedure does not effectively transfer such diversity, especially for complex models that achieve near-zero training error. To fix this, we propose a perturbation strategy for distillation that reveals diversity by seeking inputs for which ensemble member outputs disagree. We empirically show that a model distilled with such perturbed samples indeed exhibits enhanced diversity, leading to improved performance.

heading: 1 Introduction
text: Deep Ensemble (DE) [Lakshminarayanan et al., 2017], a simple method to ensemble the same model trained multiple times with different random initializations, is considered to be a competitive method for various tasks involving deep neural networks both in terms of prediction accuracy and uncertainty estimation. Several works have tried to reveal the secret behind DE’s effectiveness. As stated by Duvenaud et al. [2016] and Wilson and Izmailov [2020], DE can be considered as an approximate Bayesian Model Average (BMA) procedure. Fort et al. [2019] studied the loss landscape of DEs and showed that the effectiveness comes from the diverse modes reached by ensemble members, making it well suited for approximating BMA. It is quite frustrating that most sophisticated approximate Bayesian inference algorithms, especially the ones based on variational approximations, are not as effective as DEs in terms of exploring various modes in parameter spaces.
Despite being simple to train, DE incurs significant overhead in inference time and memory usage. It is therefore natural to develop a way to reduce such costs. An example of such a method is Knowledge Distillation (KD) [Hinton et al., 2015], which transfers knowledge from a large teacher network to a relatively smaller student network. The student network is trained with a loss that encourages it to copy the outputs of the teacher networks evaluated at the training data. With KD, there have been several works that learn a single student network by distilling from DE teacher networks. A naïve approach would be to directly distill ensembled outputs of DE teachers to the single student network. A better way proven to be more effective is to set up a student network having multiple subnetworks (multiple heads [Tran et al., 2020] or rank-one factors [Mariet et al., 2020]) and distill the outputs of each ensemble member to each subnetwork in a one-to-one fashion. Nevertheless, the empirical performance of such distilled networks is still far inferior to DE teachers.
∗ Equal contribution
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
There may be many reasons why the students are not doing as well as DE teachers. We argue that a critical limitation of current distillation schemes is that they do not effectively transfer the diversity of DE teacher predictions to students. Consider an ensemble of deep neural networks trained for an image classification task. Given sufficient network capacity, each ensemble member is likely to achieve near-zero training error, meaning that the resulting outputs of ensemble members evaluated at the training set will be nearly identical, as shown in Fig. 1. In such a situation, reusing the training data during distillation will not encourage the student to produce diverse predictions. This is quite critical, considering various results establishing that the effectiveness of DE comes from averaging out its diverse predictions.
To this end, we propose a method that amplifies the diversity of students learning from DE teachers. Our idea is simple: instead of using the same training set, use a perturbed training set for which the predictions of ensemble members are likely to be diversified. To implement this idea, we employ Output Diversified Sampling (ODS) [Tashiro et al., 2020], a sampling scheme that finds small input perturbations that result in significant changes in outputs. Empirically, we confirm that the inputs perturbed with ODS result in large deviations in the DE teacher outputs. We further justify our method by analyzing the role of ODS perturbation. Specifically, we show that distilling using training data perturbed with ODS can be interpreted as an approximate Jacobian matching procedure where ODS improves the sample efficiency of the approximate Jacobian matching. Intuitively, by approximately matching Jacobians of teachers and students, we are transferring the outputs of teacher networks evaluated not only on the training data points but also on nearby points.
Using standard image classification benchmarks, we empirically validate that our distillation method promotes diversities in student network predictions, leading to improved performance, especially in terms of uncertainty estimation.

heading: 2 Backgrounds

heading: 2.1 Settings and notations
text: The focus of this paper is on the K-way classification problem taking D-dimensional inputs. We denote a student neural network as S(x) : RD → [0, 1]K and a teacher neural network as T (x) : RD → [0, 1]K . S(x) and T (x) outputs class probabilities, and we denote the logits before softmax as Ŝ(x) and T̂ (x). The kth element of an output is denoted as S(k)(x). For DE teachers with j = 1, . . . ,M members, the jth ensemble member is denoted as Tj(x). For a student network having M subnetworks, the jth subnetwork is denoted as Sj(x).

heading: 2.2 Knowledge distillation
text: Knowledge distillation (KD) [Hinton et al., 2015] aims to train a student network S(x) by matching its outputs to the outputs of a teacher network T (x). In addition to the standard cross-entropy loss between student outputs and the ground-truth labels, i.e., LCE(S(x),y) = − ∑K k=1 y
(k) logS(k)(x), KD additionally encourages the student’s output to be as close as possible to that of the teacher by also minimizing the KL-divergence between output class distributions of the student and the teacher:
LKD(S(x), T (x); τ) = −τ2 K∑ k=1 softmax(k) ( T̂ (x) τ ) log softmax(k) ( Ŝ(x) τ ) . (1)
Here, τ > 0 is a temperature parameter which controls the smoothness of the distributions. We scale the KL divergence by τ2 since the soft targets scale as 1/τ2. As a result, in the KD framework, the student imitates the teacher’s outputs for training examples by minimizing L = (1− α)LCE + αLKD, where α ∈ (0, 1) is a hyperparameter to be specified.

heading: 2.3 BatchEnsemble and one-to-one distillation
text: BatchEnsemble (BE) [Wen et al., 2020] is a lightweight ensemble method that reduces the number of parameters by weight sharing. Specifically, each layer of a BE model consists of shared weights W and rank-one factors rjs>j for j = 1, . . . ,M with M being number of subnetworks. Unlike the full ensemble that constructs full set of M weights, BE shares W across all subnetworks while the rank-one factors are kept specific to each subnetwork. The weight matrix of the jth subnetwork is computed as W ◦ rjs>j and consequently BE requires far fewer parameters than a full ensemble. During training, BE simultaneously trains all subnetworks by feeding a duplicate of the current minibatch for each rank-one factor rjs>j into the network. This procedure is easily vectorized because the computation graph for each subnetwork has the same structure, differing only in weight values. Mariet et al. [2020] proposed to further improve the performance of BE by distilling from fully trained DE teachers instead of training from scratch. During distillation, the parameters of jth subnetwork Sj(x) is trained by distilling from the jth DE teacher Tj(x). The corresponding objective function is then written as (c.f., Fig. 2 without perturbation)
L = M∑ j=1 { (1− α)LCE(Sj(x),y) + αLKD(Sj(x), Tj(x); τ) } . (2)
Throughout the rest of the paper, we will use the BE as student networks and follow this one-to-one distillation framework.

heading: 2.4 Output diversified sampling (ODS)
text: ODS [Tashiro et al., 2020] is a sampling method that can maximize diversities in outputs of a given function; it is originally introduced to replace random input perturbation to enhance the performance of the adversarial attack. Let F : RD → RK be a target function. ODS draws a perturbation vector guided from the gradient of the target function as follows.
w ∼ Unif([−1, 1])K , εODS(x,F ,w) = ∇x(w>F(x)) ‖∇x(w>F(x))‖2 ∈ RD. (3)
The intuition behind this sampling strategy is to seek the direction in the input space that maximizes the similarity between the function output and a randomly sampled vector w. Following such a direction can make diverse output shifts due to the randomness in w.

heading: 3 Learning from Ensembles with Output Diversification

heading: 3.1 One-to-one distillation with ODS
text: As discussed before, vanilla one-to-one distillation evaluates DE teachers on points where their outputs are nearly identical, so the students trained using them tend to exhibit low diversity in outputs. To avoid this undesirable behavior, we propose to instead train students on points perturbed with ODS. Let (x,y) be an training input pair. We first pick a random teacher Tr uniformly from {T1, . . . , TM} and perturb x as
x̃ = x+ ηεODS(x, softmax(T̂r(x)/τ),w), (4) where η > 0 is a step-size. Ideally, we would like to evaluate ODS vectors for each teacher to build M perturbed versions of x, but this would take too much time for gradient computation, especially when we have a large number of ensemble models. Instead, we just pick one of the teachers, use it to perturb the input, and share the perturbed input to train all students with the following loss function,
L = M∑ j=1 { (1− α)LCE(Sj(x),y) + αLKD(Sj(x̃), Tj(x̃); τ) } , (5)
where we are evaluating the KL-divergence between teacher outputs and student outputs on the perturbed input.
An implicit assumption here is that a diversifying direction computed from a specific random teacher can bring diversities in the outputs of all the teacher networks. This is based on the transferability assumption empirically justified in Tashiro et al. [2020]: ODS perturbations are computed on a surrogate model when we do not have access to the true model, assuming that the Jacobians computed from the surrogate model can approximate the true Jacobian to some extent. In our case, let T1 be a randomly picked teacher with which an ODS perturbation is computed, and T2 be another teacher. Assuming the transferability of Jacobian, we can let ∇xT1(x) ≈ R∇xT2(x) for some matrix R. Then the ODS perturbation computed from T1(x) is εODS(x, T1,w) ∝ w>∇xT1(x) ≈ (R>w)>∇xT2(x) ∝ εODS(x, T2,R>w). (6) That is, the ODS perturbation computed from T1 acts as another ODS perturbation with different random guide vector R>w. Hence, the same perturbation drives the outputs of T1 and T2 towards different directions. Note that without transferability, the Jacobians∇xT1(x) and∇xT2(x) would be vastly different and so are the directions w and R>w. This may be good in terms of diversification, but the resulting perturbed input act as an adversarial example not very useful for distillation, making the teachers completely disagree (refer to Appendix A.2 for more details). Our experiments in Section 5.2 and Section 5.3 empirically re-confirm this transferability assumption.
We can also interpret the random teacher selection as a stochastic approximation. Specifically, to find a direction maximizing diversities between teachers, we can consider an objective
Ew1,...,wM [ M∑ j=1 w>j Tj(x) ] , (7)
with w1, . . . ,wM i.i.d.∼ Unif([−1, 1])K . That is, we want to find a direction in the input space making the teachers {Tj}Mj=1 follow different guide vectors {wj}Mj=1. Picking a random teacher and computing the ODS perturbation can be understood as a stochastic approximation of the gradient of this objective. Again, thanks to the transferability of the Jacobians, the variance of this stochastic gradient would be small, making it sufficient to use a single random teacher.
In addition, we empirically found that scaling the ODS updates according to confidence slightly improves performance. That is, we perturb an input x as
x̃ = x+ ηCmax(x, Tr, τ)εODS(x, softmax(T̂r(x)/τ),w), (8) where Cmax(x, Tr, τ) = maxk softmax(k)(T̂r(x)/τ) denotes the maximum class probability. This scheme takes larger steps towards ODS directions for datapoints with high confidence. We call this modification ConfODS.
The training procedure with ODS perturbation is straightforward (Algorithm 1); we can just add the ODS computation procedure and replace the loss function from the original KD training procedure. Fig. 2 depicts our overall training procedure as a diagram.
Algorithm 1 Knowledge distillation from deep ensembles with ODS perturbations
Require: Training data D = {(xi,yi)}Ni=1. Require: Knowledge distillation weight α, temperature τ , learning rate β, ODS step-size η. Require: Deep ensemble teacher networks {Tj}Mj=1 where M denotes the size of ensembles.
Let S be a student network which consists of M subnetworks {Sj}Mj=1. Initialize parameters θ of the student network S. while not converged do
Sample (x,y) ∼ D,w ∼ Unif([−1, 1])K , and a teacher index r uniformly from {1, . . . ,M}. Sample diversified inputs x̃ = x+ ηεODS(x, softmax(T̂r(x)/τ),w) LCE ← − ∑M j=1 ∑K k=1 y
(k) logS(k)j (x) LKD ← −τ2 ∑M j=1 ∑K k=1 softmax
(k)(T̂ (x̃)/τ) log softmax(k)(Ŝj(x̃)/τ) θ ← θ − β∇θ((1− α)LCE + αLKD)
end while

heading: 3.2 Interpretation as approximate Jacobian matching
text: We can also interpret our distillation strategy as an approximate Jacobian matching procedure. Srinivas and Fleuret [2018] shows that a KD procedure on inputs perturbed by small noise implicitly encourages matching the Jacobians of a teacher and a student. More specifically, let S and T be student and teacher models. The first-order Taylor expansion of the expected KD loss on perturbed inputs can be written as1
Eε[LKD(S(x+ ε), T (x+ ε)] = LKD(S(x), T (x)) + LJM(S(x), T (x)) + o(σ2), (9)
where σ2 = E[ε>ε] and
LJM(S(x), T (x)) = −Eε [ K∑ k=1 ε>∇xT (k)(x)∇>xS(k)(x)ε S(k)(x) ] . (10)
While Srinivas and Fleuret [2018] suggested using isotropic Gaussian noise ε ∼ N (0, σ2ID), we argue that simple Gaussian noise is ineffective, especially for high-dimensional data that are likely to be embedded in low-dimensional manifolds. In such a case, most perturbation directions are orthogonal to the data manifold, and thus Gaussian noise becomes uninformative. On the other hand, ODS directly uses the Jacobian of the teacher to construct the perturbation direction. We conjecture that due to this fact, KD with ODS perturbation is a much more sample-efficient approximate Jacobian matching procedure compared to using Gaussian perturbations. We perform experiments to empirically validate this claim in Section 5.3.
The above analysis is for the usual KD framework with a single teacher and a single student. In our case, we are matching M teachers to M student subnetworks, but with ODS perturbations computed from a single teacher randomly sampled from M teachers. Again, we are operating under the assumption that the Jacobian computed from a specific teacher will transfer well to other teachers, so the approximate Jacobian matching using it is still more effective compared to simply using Gaussian perturbations.

heading: 4 Related Works
text: Knowledge distillation KD is a method for transferring the information inside a large model into a smaller one [Hinton et al., 2015] by minimizing the KL divergence between teacher and student networks. [Wang et al., 2020] finds that additionally augmenting the data results in further performance gains because such augmentations allow the KD loss to tap into teacher information outside of the training set. Furthermore, modifications to the standard KD objective allow one to distill the knowledge in an ensemble of teachers into one student network while preserving the benefits in uncertainty estimation of ensembles [Malinin et al., 2020, Tran et al., 2020]. KD can be enhanced via gradient matching, and several existing methods [Srinivas and Fleuret, 2018, Czarnecki et al., 2017] propose sampling methods for knowledge distillation up to higher-order gradients. However, such existing methods suffer from a high computational burden or inefficient random perturbation.
1We assume τ = 1 for notational simplicity, but the argument applies equally well to any τ .
Ensembles Ensemble methods [Hansen and Salamon, 1990, Dietterich, 2000], which construct a set of learners and make predictions on new data points by a weighted average, have been studied extensively. Ensembles perform at least as well as each individual member [Krogh and Vedelsby, 1994], and achieve the best performance when each member makes errors independently. Aside from benefits in model accuracy, DE [Lakshminarayanan et al., 2017, Ovadia et al., 2019] have recently shown to be a simple and scalable alternative to Bayesian neural networks because of their superior uncertainty estimation performance. Wen et al. [2020] proposes a network architecture that uses a rank-one parameterization to train an ensemble with low computation and memory cost.

heading: 5 Experiments
text: In this section, we try to answer the following questions with empirical validation.
• Do images perturbed by ODS increase diversity in DE teacher predictions? - Section 5.2. • Do ODS perturbations encourage Jacobian matching? - Section 5.3. • Does enhanced diversity of students lead to improved performance in terms of prediction
accuracy and uncertainty estimation? - Section 5.4 and Appendix A.1.

heading: 5.1 Experimental setup
text: Datasets and networks We compared our methods on CIFAR-10/100 and TinyImageNet. We used ResNet-32 [He et al., 2016] for CIFAR-10, and WideResNet-28x10 [Zagoruyko and Komodakis, 2016] for CIFAR-100 and TinyImageNet. Please refer to Appendix B.1 for detailed training settings.
Hyperparameter settings The important hyperparameters for KD are the pair (α, τ); for CIFAR10, after a through hyperparameter sweep, we decided to stay consistent with the convention of (α, τ) = (0.9, 4) for all methods [Hinton et al., 2015, Cho and Hariharan, 2019, Wang et al., 2020]. For CIFAR-100 and TinyImageNet, we used the value (α, τ) = (0.9, 1) for all methods. We fix ODS step-size η to 1/255 across all settings. See Appendix B.1 for more details.
Uncertainty metrics We measure uncertainty calibration performance with common metrics including Accuracy (ACC), Negative Log-Likelihood (NLL), Expected Calibration Error (ECE) and Brier Score (BS). We also report the Deep Ensemble Equivalent (DEE) score [Ashukha et al., 2020], which measures the effective number of models by comparing it to a full DE. For all methods, the metrics are computed at optimal temperatures obtained from temperature scaling [Guo et al., 2017], as suggested by Ashukha et al. [2020]. See Appendix B.2 for the definition of each metric.

heading: 5.2 Impact of ODS on diversities
text: In order to assess the effect of ODS perturbation on the diversities of DE teachers or student subnetworks, we draw diversity plots in Fig. 3. To draw a diversity plot, we first collect the outputs of target models evaluated on a specific dataset and binned the samples w.r.t. their confidence values (minimum confidence among M models). For each bin, we computed average pairwise KL-divergence between class output probabilities of ensemble members. We also report the representative value (mean-KLD), which is defined as average KL-divergence values weighted by bin counts.
Fig. 3a shows the diversity plots of DE teachers on CIFAR-10. Intuitively, the samples predicted with high confidences usually come with low diversities, and this is indeed depicted in the plots. For the DE evaluated on the training set, the mean-KLD value is very small (the first box) compared to the one computed on the test set (the fourth box). While Gaussian perturbation does not affect the diversities (the second box), ODS drastically amplifies the diversities (the third box).
Fig. 3b shows the diversity plots of the BE students trained on CIFAR-10. Although evaluated at the test set, the BE distilled without any perturbation (the first box), the BE distilled with Gaussian perturbations (the second box), and the BE trained from scratch (the fourth box) exhibit very low diversities. On the other hand, the BE distilled with ODS perturbations show relatively high diversity, and as we will show later in Section 5.4, this indeed leads to improved performances.
In Section 3, we assumed that an ODS computed from a specific teacher can be transferred to other teachers. As an empirical justification, we draw the diversity plot using ODS computed from an
external teacher (a network trained in the same way as the teachers but not actually being used during distillation) in Fig. 6a. The mean-KLD values are indeed increased compared to the original DE or DE evaluated on examples perturbed with Gaussian noise, confirming the validity of our assumption.
5.3 ODS for Jacobian matching
Effectiveness We empirically validate our conjecture that ODS perturbations are more effective in Jacobian matching. To this end, we computed the cosine similarities between the vectorized Jacobians of ResNet-32 teacher and student networks trained with vanilla KD. We computed the same quantity using student networks distilled with Gaussian perturbations and ODS perturbations. We draw Receiver Operating Characteristic (ROC) curves to compare the similarities of distilled students and the baseline trained from scratch. Specifically, we compare two distributions of the cosine similarities between the vector-
ized Jacobians of a teacher network and a student network: (1) when the student network is trained by KD, and (2) when the student network is trained from scratch without any guide from the teacher network. Results in Fig. 5 clearly show that the Jacobians of the student distilled with ODS better match the Jacobians, while the student distilled with Gaussian perturbations did not significantly improve upon the student distilled with vanilla KD.
Next, we compute the gradient of the term related to the Jacobian matching, i.e.,
G(x, ε) = ∇θ(LKD(S(x+ ε), T (x+ ε))− LKD(S(x), T (x))). (11)
Using both Gaussian perturbations and ODS perturbations, we collected multiple gradient samples and measured the Signal-to-Noise Ratio (SNR) of the gradient norms. Specifically, we computed SNR(x, ε) = ‖Eε [vec(G(x, ε))]‖2 / √ ‖Var [vec(G(x, ε))]‖2, where vec(·) is vectorization and Var[·] is element-wise variance. As shown in Fig. 4, the gradients computed with ODS perturbations exhibit higher SNR, indicating that learning with such gradients is more stable and efficient.
Transferability As in Section 5.2, we performed an experiment to verify our assumption that ODS perturbations for a specific model are transferable to other models, for the purpose approximate Jacobian matching. We trained a BE student with ODS perturbations computed from an external teacher, and measured the cosine similarities between the student and teachers (not between the students and the external teacher). As one can see from Fig. 6b, the resulting student network copied the Jacobians of the teachers significantly better than the baselines.

heading: 5.4 Main results: image classification tasks
text: We validate whether the enhanced diversity of student networks is helpful for actual prediction. We compared our methods against three baselines: BE trained from scratch [Wen et al., 2020], BE distilled without perturbation [Mariet et al., 2020], and BE distilled with Gaussian noises.
The results are presented in Table 1. For both datasets, BE distilled with ODS or ConfODS significantly outperforms baselines in terms of every metric we measured. The improvement is much more significant for the larger ensembles (M = 8 for CIFAR-10) and the larger dataset (CIFAR-100). We note that using our diversification strategy improves performance both in terms of raw accuracy and uncertainty metrics. An especially noteworthy result on CIFAR-100 is that BE distilled with ODS achieved a DEE score over three. This is quite remarkable considering the BE model has far fewer parameters than the full DE which, by definition, has an expected DEE of four. Moreover, further uncertainty results in the Appendix A.1 show that our approach is superior with respect to both predictive uncertainty for out-of-distribution examples and calibration on corrupted datasets.
One great benefit of the KD framework is that it can transfer knowledge to networks having different architectures, greatly reducing the number of parameters required to achieve a certain level of performance. Using CIFAR-100 and TinyImageNet datasets, we compared our method to baselines with student networks having smaller architecture. The results displayed in Table 2 show that KD with ODS or ConfODS outperforms the baselines by a wide margin.

heading: 6 Conclusion
text: In this paper, we proposed a simple yet effective training scheme for distilling knowledge from an ensemble of teachers. The key idea of our approach is to perturb training data in a proper way to expose student networks to the diverse outputs produced by ensemble teachers. We employed ODS to find such diversifying perturbation. Our distillation scheme with ODS perturbation can be interpreted as a sample-efficient approximate Jacobian matching procedure. We empirically verified that our method enhances the diversities of students, and that such enhanced diversities help to reduce the performance gap between student and teacher models in KD. An interesting future work would be to rigorously analyze the Jacobian matching aspect of our method and to provide theoretical justification and a better distillation method to harness it. Also, in this paper, we constrained ourselves to BE students, and applying our framework to other student network architectures such as Bayesian neural networks would be interesting.
Limitations Although we have shown promising results, our method still has several limitations. Due to its nature as a KD framework, the need to train M teachers involves a considerable computational cost. Also, our method cannot improve the student beyond its capacity. For instance, in Table 2, WideResNet-28x2 distilled from WideResNet-28x10 does not greatly benefit from our method. Arguably the most critical limitation is that our method is a heuristic and lacks any theoretical guarantee. In that sense, as mentioned above, rigorously analyzing the effect of ODS perturbation will be important future research.

heading: Acknowledgement
text: This work was partly supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2019-0-00075, Artificial Intelligence Graduate School Program(KAIST), No. 2021-0-02068, Artificial Intelligence Innovation Hub), National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF2021R1F1A1061655, NRF-2021M3E5D9025030), and Samsung Electronics Co., Ltd (IO20121408176-01).

Review summaries:
The paper studies the question of ensemble function diversity and how to preserve it on ensemble distillation to a single model. The authors use input perturbations that make the models, especially well trained models that are near 0 loss on the training set, disagree on the input, and then use knowledge distillation from individual members of the ensemble to individual subnetworks of the distilled model. The authors show that without this approach, the distilled models suffer from lack of function diversity, and that their approach enhances it and in accordance with their theory enhances performance as well.

This paper proposes to improve Knowledge Distillation (KD) for ensemble models. The authors apply ODS [Tashiro et.al, 2020] to each teacher model in the ensemble to obtain perturbed data. Thereafter, the perturbed data is used to train student models, each corresponding to one teacher model. The authors argue that ODS can improve the diversity among student models in the ensemble.

This paper focuses on the topic of effectively distilling ensembles of models into a single model. Assuming that the distilled model should inherit the diversity of the ensemble, they attempt to make the distilled performance more comparable to the ensemble by using a (adversarial) perturbation strategy that directly identifies sources of diversity within the ensemble (i.e., where models within the ensemble disagree with each other). The authors demonstrate empirically that utilizing such a diversity-inducing objective within distillation improves the final performance of the distilled model in comparison to the ensemble on image classification experiments. They motivate this objective for distillation with intuitive descriptions of the importance of diversity within an ensemble, as well as by making connections to gradient matching between the student and teacher. 

This paper attempts to reduce the performance gap between an ensemble and a model that was distilled from it. The main idea is that, when distilling, the model should “absorb as much function diversity inside the ensemble as possible”. To do that, the paper proposes using ODS perturbation strategy, that samples diverse predictions for a given input form the ensemble, to use as label during distillation.