Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a functional ensemble distillation (FED), a novel approach for distilling an ensemble of deep neural networks into a single model. The main idea is to create a generator in a student network that takes Gaussian noises as additional inputs and emulates samples from the posteriors from them and learns the diverse prediction of ensemble teachers in a nonparametric fashion using Maximum Mean Discrepancy (MMD). Additionally, to resolve the issue of ensemble teachers almost completely agreeing with each other on the training set, the paper further proposes to use mixup augmentation during training. The proposed method is validated on some benchmark image classification tasks.
strengths_and_weaknesses: Strengths
- The paper is relatively well written and easy to follow.
- The motivation is well set.
- The distillation via MMD seems novel (in the ensemble distillation literature, but not in general).
- The experimental results seem promising.

Weaknesses
- Only evaluated with accuracy and ECE, so the results do not fully reflect the uncertainty quantification aspect of the models.
- Somewhat unfair comparison environment for DM (see below).
- No experiments on larger scale datasets such as TinyImageNet or ImageNet.
 
questions: - The paper seems to mainly discuss the setting where an ensemble constructed from cSGHMC are used as a teacher. Does it work well with the typical deep ensembles with a much less number of ensemble models, $M \in (4~8)$ for instance?
- Similar to the above question, how does FED perform w.r.t. the different number of ensembles $M$? Does it work well with a much less number of ensemble members? This is particularly important because in real-world scenarios it is often infeasible to use 120 samples during training in a restricted resource setting.
- I find the results confusing in some sense; most of the time, FED shows higher accuracies and lower ECEs than the baselines, but the agreement values are quite bad, which potentially indicates that the predictions produced by FED are not as diverse as teachers. This contradicts the previous reportings that the diversities in the predictions are actually key to the robustness and better uncertainty quantification in ensembles. Can you elaborate on this?
- Usually, when it comes to comparing ECEs, it is recommended to first do the temperature scaling and compare the ECEs with the optimal temperatures. Judging from the numbers in the results, the scales of the ECEs are not similar to each other, so I suspect that the models are not temperature-scaled before computing ECEs. Can you further compare the ECEs with calibrated models?
- In my opinion, ECE is not a good metric on its own to capture various aspects of uncertainty quantification of the model, so it is always beneficial to look at multiple uncertainty metrics such as negative log-likelihoods, brier scores, and out-of-distribution detection performances. Especially, related to the question above, since the proposed model exhibits high accuracy but high agreement values, it would be important to look at these metrics.
- The baselines can also benefit from mixup augmentation. Have you try applying it for the baselines?

limitations: - As far as I see from the paper, DM and Hydra use only 8 heads but still distill from 120 samples; this might seriously harm the performance of the baselines, especially because these methods are designed to distill each teacher member to each subnetwork in a one-to-one fashion. If possible, it would be good to see the comparison in the setting where the teachers come with a small number of ensemble members so that one can directly distill DM and Hydra in a one-to-one fashion.
- As mentioned above, no metrics other than ACCs and ECEs. 
- No experiments on large-scale image classification benchmarks, such as ImageNet. I find the distillation method often suffers when it comes to the scale of ImageNet, so would be good to see the scalability of the proposed methods on such datasets.
- The main contribution can be understood as a combination of two ingredients; 1) design of the student networks that introduce Gaussian perturbations for inputs and intermediate layers, and 2) learning with MMD. Without an ablation study, it is hard to see the net effect of each component. For instance, we can try learning the proposed model with typical knowledge distillation loss, or try distilling a Hydra architecture with MMD loss.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a novel approach for distilling a Bayesian ensemble of models:  

$\int_{\theta}p(y | x, \theta) p(\theta | D) d\theta \approx \frac{1}{M}\sum_{\theta_i \sim p(\theta | D) } p(y | x, \theta_i) $

Existing methods tries to distill the ensemble by fitting a Dirichlet distribution: 

$f(x; \phi) = \alpha$, where $\alpha$ is the concentration parameters that fully characterize the Dirichlet distribution 

This paper proposes to learn a generator that can generate predictive distribution like 
$p(y | x, \epsilon_i)$, where $\epsilon_i$ are random samples from a fixed distribution. 

This is in spirit very much like the reparameterization trick for variational auto-encoder [1]. 

The inference can be executed by sampling a batch of $\epsilon$’s and then run the forward function of the learned generator on this batch in parallel. 

To promote the diversity of the ensemble, during training the generator, mixup data augmentation is applied, which is shown to significantly boost the accuracy and diversity. 


[1] Kingma & Welling, Auto-Encoding Variational Bayes, 2013

strengths_and_weaknesses: Strengths 

- The proposed method is novel and general purpose, can distill any ensemble of models   
- The idea of using mixup augmentation for distillation is creative and interesting
- The experimental results seem strong, with well designed ablation studies on mixup.  

Weaknesses: 

- it’s claimed to be a general-purpose distillation method, but only classification tasks are considered in the experiments; it’s not clear how good it performs on regression tasks, and maybe compare with EnDD fitting, e.g., a Gaussian distribution (which seems straightforward to adapt) 
- one advantage of the proposed method is claimed to be able to measure the correlation between two inputs, but seems it’s not discussed how to do that, and not experimentally validated that the correlation is a close approximation of the original ensemble
- there seems to be large computational overhead during inference since each example need to be extended to a batch with different $\epsilon$ samples, so inference cost is multiplied by the number of samples, though they could be run in parallel. 

questions: - how does it compare, in terms of accuracy, to a naive baseline where we use multiple teachers (i.e., the ensemble of models) to distill one student model $P(y|x)$? 
 

- What's the decision making algorithm given an ensemble of predictions or a Dirichlet distribution? Averaging $p(y | x, \theta_i)$ or majority vote? for EnDD, do you take the mean of the learned Dirichlet distribution or take samples from that and then take majority vote? If the former, how do you measure agreement metric? 

- could you elaborate on how to set the noise STD to be a learnable parameter? 

limitations: n/a
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes Functional Ensemble Distillation (FED), a new method for distilling (the predictions of) an ensemble of models into a single model. FED trains a generator network which learns a distribution over functions by mapping the input together with a sample of an external random noise variable to the output. As forward passes with the distilled generator model can be made for multiple samples of the random noise variable in parallel, the method is efficient to run at test time. In contrast to previous work, their approach can also quantify the correlations between predictions. Furthermore, the paper provides a solution to the case where all models in the ensemble overfit to the training data and make very similar predictions to achieve zero training loss, such that distilling their predictions becomes sub-optimal: using an auxiliary dataset generated via mixup data augmentation, the individual model predictions can be made much more diverse, which helps distill a more performant model. The paper empirically validates the efficacy of the proposed distillation method in terms of predictive accuracy and calibration compared to other baselines.
strengths_and_weaknesses: **Summary of Review**

The proposed new method to address the important problem of ensemble distillation is clever and empirically shown to significantly boost performance in some settings. While the method seems promising, I see some issues with the experiments (detailed below) that should be addressed to strengthen the work. Overall, I am leaning towards recommending rejection of the paper in its current form, but I am happy to reconsider my judgement if my concerns can be adequately addressed.

**Strengths**
- The manuscript is overall well written and easy to follow.
- The work tackles an important problem, namely making ensembles more efficient via distillation.
- The proposed FED approach for distilling an ensemble using a generator network with an external noise random variable is novel, sound, and intuitively sensible.
- The proposed method for diversifying the ensemble member predictions in the overfitting regime by generating an auxiliary dataset via mixup augmentation is simple and intuitively appealing.
- The empirical evaluation is thorough and demonstrates that FED can achieve significant performance gains in terms of predictive accuracy and calibration compared to relevant baselines across several image classification benchmarks.

**Weaknesses**
- I am not sure if the empirical comparison is fair, as the different methods seem to have different runtime requirements; in particular, you claim that the inference runtime of FED scales as 1 + (# of epsilon samples) / 35, which for 120 samples (i.e. the number you use in the experiments) is roughly 4.5; in my understanding, this means that the inference time of FED is 4.5x slower than that of a single deterministic model (while in contrast, the vanilla ensemble is 120x slower; I requested clarification for this in the Questions below, so please let me know if my understanding is incorrect); if this assumption is correct, this would mean that FED is significantly slower than some of the baselines considered; e.g., in my understanding, EnDD should have a runtime similar to that of a single model (again, please correct me if I'm mistaken); all in all, my concern is that the performance comparison should be conducted more carefully, taking inference runtime into account, to make it more meaningful; to this end, one could e.g. plot the results in 2D with runtime on the x-axis and performance on the y-axis; different methods would then result in different trade-offs on this plot, yielding Pareto curves that would be much more insightful than just single performance numbers; at the end of the day, a practitioner is probably interested in one of two questions: 1) given a fixed runtime budget X, what’s the best performance Y that I can achieve?, or 2) given a desired performance Y, what’s the fastest runtime X to achieve it? to make the comparison even more interesting, it would be great to consider different settings for each individual method for trading off between runtime and performance; I'm not fully sure how to best do this, but e.g. for FED, one could try varying the number of epsilon samples (for a fixed number of ensemble members) to make the method cheaper; conversely, one could try to make the baselines more expensive in more way; this would allow you to compare performance of the methods for a given fixed inference budget.
- The empirical evaluation could be significantly improved by considering a more diverse set of model architectures and benchmarks; the paper focuses on a single architecture (ResNet18) and three small-to-medium-sized image classification benchmarks (CIFAR-10, CIFAR-100, STL-10); it is not clear if the drawn conclusions would extend to other model families (e.g. transformers), data modalities (e.g. text) and benchmark sizes (e.g. ImageNet); to gain more space for reporting additional major empirical results, one could e.g. move Section 5.4 (on the OOB methods for inducing diversity without an auxiliary dataset) to the appendix -- while the experiment and conclusion in that section are certainly interesting, they do not seem to necessary to have in the main text of the paper due to the comparably poor results.
- The paper emphasises that, in contrast to previous methods (which are e.g. based on fitting a Dirichlet distribution over predictions), FED is able to capture correlations between outputs, and claims that this is an important feature for many real-world applications (e.g. l. 9-11, l. 52-55, l.149-153); as this is mentioned already prominently in the abstract as a main selling point for the method (l. 9-11), I was expecting at least some empirical evidence for the benefit of this feature; unfortunately, no explicit evidence is provided beyond some vague intuitive written arguments (l. 52-55, l. 149-153). It would significantly strengthen the paper if capturing these covariances could be shown to improve performance in some application. (While I understand that FED is shown to improve upon previous methods, it is not clear to me how this gain can be directly attributed to the aforementioned property.)

**Minor Issues**
- Table 3: bolding the best values would improve clarity
- It would be useful to put a brief take-away message into the caption of each table/figure
- The paper misses the Laplace approximation as another fundamental Bayesian inference approach for neural networks (in addition to VI and MCMC); for a recent overview, see e.g. Daxberger et al., *Laplace Redux -- Effortless Bayesian Deep Learning*, NeurIPS 2021
- In addition to the runtime comparison requested above, it would also be useful to have a more clear/complete comparison of the memory requirements of all methods; you mention the memory burden of FED in Appendix C.2, and e.g. that of the Hydra baseline in Appendix A; it would be useful to have a table/plot comparing this for all methods considered

questions: - Do you have some intuition as for why the OOB methods in Section 5.4 perform substantially worse than the mixup approach? Is it because they simply use less data for training? Or is it because mixup introduces additional regularization that boosts performance?
- I'm not sure if I fully understand what you mean when saying that FED scales approximately as 1 + (# of epsilon samples) / 35; e.g., for 120 epsilon samples, this would roughly be 1 + 120 / 35 = 4.43; is this referring to a multiplicative factor relative to the inference time of a single deterministic model (i.e. saying that FED with 120 samples is ~4.5x slower than a single model)?
- Relatedly, in Appendix C.1 you mention that inference with an ensemble of 120 models takes 4.075s wall-clock time. As inference time should grow linearly with the number of ensemble members, I would expect a single model to roughly take 4.075s / 120 = 0.034s for inference. It is then claimed that FED takes only 0.020s with 120 epsilons, which is almost twice as fast as that. How can it be that FED inference time is faster than that of a single deterministic model? Following your claim of the approximate scaling of FED, shouldn't the runtime of FED be much higher than what you report? Or am I misunderstanding something here?
- Why did you not include the most basic distlliation baseline (i.e. simply the original Hinton et al. distillation procedure) in your evaluation? I believe this would be insightful to some readers.

limitations: Yes, the authors have adequately addressed this.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose a method for distilling an ensemble of models. This improves performance (only one model must be stored and used). Rather than making any substantial assumptions about the probabilistic model, the method uses relatively small amounts of added noise and data augmentation.
strengths_and_weaknesses: I liked this paper. It is clear and very well written. I expect it to have significant impact and it is fairly original. Most of my comments are minor and reflect things I feel could have made the arguments even stronger.

The title/name is 'Functional' but the method doesn't feel super functional to me. Part of the loss is arguably functional, but the method is basically parametric. Is there a clearer way to communicate the method?

ln 39 "using" and "use" redundant.

There was some overlap of background/intro/related work. Perhaps a little more editing could let you hit your contribution sooner.

I don't understand the motivation for concating 3 Gaussian RVs to the input and adding noise (is it a vector? scalar?) for each block. You mention you considered other things in teh paragraph starting ln 177. Are you able to provide ablations? It isn't a deal breaker, but this felt pretty arbitrary.

In line 184 you mention non-negligible overhead. Please can you just say what the overhead is? In params and FLOPS/ms?

You mention in ln 205 that you tried a GAN thing and it didn't work as well. Can you just provide the ablation in the appendix? There's a general theme here that you allude to some work investigating alternatives and it feels like you could maybe just show the work? This could go into Table 3.

In line 234, a nitpick. It's not that I think you *should* use batchnorm, but I think you are overstating the point. First, Ismailov et al. only say that batch norm makes it harder to clearly interpret the probabilistic model, not that it is wrong. I think that's the correct view---the probabilistic model becomes a bit more complicated but it isn't all of a sudden non-Bayesian. Second, Bayesian inference doesn't make an IID assumption---in fact one of the most important parts of the Bayesian approach is that the data are not distributed at all! So I think this statement is incorrect.

You use 120 models for your ensemble. That's rather a lot. Typically people doing ensembling might use between 3 and 10. Is there a reason for your choice? Is it perhaps important for your method to have lots of ensemble elements? In that case there might be an unacknowledged limitation of the approach, that during training you have to produce many more ensemble elements in order to get a good distillation (so you are trading off training time for test time). I would like to see some ablations with 5 ensemble members, say, to understand this tradeoff better.

Regarding ensemble agreement, you do show in Table 1 that the ensemble members have high agreement on the training data. But you claim that this leads to problems with the distilled ensemble. I didn't understand you to have provided any evidence for that second claim. Have I missed it?

In Table 2 there are many apparent errors in what has been made bold. For example, FED is shown bold for ECE on STL despite the fact that it is definitely not as good as EnDD_{AUX} (I understand of course that it doesn't see as much data!). Similarly, in some sense AMT has the 'best' agreement on STL-10 but it is DM_G which is bolded (and moreover the three DM methods are statistically indistinguishable). If you are going to bold things, please can you go through this carefully to make sure that the bold is correct (I don't claim to have caught all the corrections).

The 'Bayesian' baseline in Table 4 looks like it has overfit quite badly. Do you think that maybe there is a tuning problem and the baseline is a bit weak? Also, from the body of the text it seems like 'Bayesian' is the same as 'Ensemble' from other tables? Or is there a reason you use a different name?
questions: Mostly discussed above:
- Why noise concat and addition?
- What actually is the overhead?
- Augmentation ablation?
- Why so many ensemble elements? Does it break with fewer?
- Is there evidence for the claim that ensemble agreement hurts the distilled model?
- What is the 'Bayesian' baseline compared to the 'Ensemble' baseline?
limitations: Seems like there might be an ensemble element number limitation that isn't addressed?
review_title: null

Meta-review: metareview: The paper proposes a method for ensemble distillation, motivated by the need for efficient Bayesian machine learning. The method seems novel and can potentially make a significant contribution to the literature of ensemble distillation and Bayesian machine learning.

The reviewers found the paper well-written and the empirical results compelling. Some concerns were raised about the fairness of the empirical evaluation in the first round of reviews, but these were mostly addressed during the discussion with the authors. Seeing as no major concerns remain, I'm happy to recommend acceptance.

confidence: Certain
Decision: Accept