Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: The paper proposes to distill the predictions of an ensemble with a multi-headed network, with as many heads as members in the original ensemble. Distillation proceeds by minimizing the KL divergence between the predictions of each ensemble member with the corresponding head in the student network. Experiments illustrate that the multi-headed architecture approximates the ensemble marginally better than approaches that use a network with a single head.

The paper presents a straightforward idea and fairly unsurprising results —  a multi-headed architecture with each head matching an ensemble member more faithfully represents the original ensemble. This improved fidelity, however, comes at the cost of increased computation and storage requirements (which scale linearly with the size of the ensemble). It is unclear that the marginal improvements demonstrated justify the increased cost and how this approach would scale to larger ensembles. The paper would have been more interesting if the authors had managed to demonstrated significant improvements over competitors on not toy (MNIST / CIFAR) problems. Unfortunately, this is not the case and the fact that similar ideas (Lan et al.) have been proposed in the past (which the authors, to their credit, cite) leads me to recommend a rejection.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: Overview:
This work introduces a new method for ensemble distillation. The problem of making better ensemble distillation methods seems relevant as ensembles are still one of the best ways to estimate uncertainty in practice (although see concerns below). The method itself is a simple extension of earlier “prior networks”: the original method suggested to fit a single network to mimick a distribution produce by given ensemble, and here authors suggest to use multi-head (one head per individual ensemble member) in order to better capture the ensemble diversity. 
Authors report results on multiple relatively standard benchmarks (MNIST, CIFAR, etc), and seem to outperform the baseline by a small margin. The choice of baselines is reasonable.

Writing:
The paper is well-written, illustrations are good.

Decision:
The method itself is very easy to implement, and does seem to outperform the baseline (prior networks). However, I am a bit concerned that the method itself seems like a trivial extension of the prior work, and does not really provide much addition insight. In addition, the results are reported on a set of small-scale benchmarks and seem incremental: it can be OK, but it would be really great to see a somewhat more realistic application.
Thus, I am on the fence with this one, but generally positive about this work, thus “weak accept” rating.

Questions / concerns:
* I honestly do not see the point on having an additional column in tables if all the values are N/A. 
* The names in Table 4 are mixed up.
* Arguably, a lot of applications that would actually rely on uncertainty estimation might require online training of some sort. This means that in those scenarios one does not actually have access to a pre-trained ensemble. I understand that this might not be the main focus of this work, but it seems like a major limitation of “distillation” approaches in general, which should / could be addressed in some way?

<update>
Thanks for a detailed answer. I am not very convinced by the argument about online-vs-offline training, but I do not see a reason to decrease my rating. I can see this work useful in practice.
</update>

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Summary & Pros
- This paper proposes a simple yet effective distillation scheme from an ensemble of independent models to a multi-head architecture for preserving the diversity of the ensemble.
- The proposed scheme provides the same advantages of the ensemble in terms of uncertainty estimation and predictive performance, but it is computationally efficient compared to the ensemble.
- This paper experimentally shows that multi-head architecture performs well on MNIST and CIFAR-10 in terms of accuracy and uncertainty.

Concerns #1: Novelty of the proposed method
- The multi-head architectures have been widely used in various settings, especially multi-task learning. As the authors mentioned, it also used for online distillation [1]. Although its goal is different from this paper, just applying such multi-head architectures seems to be incremental.

Concerns #2: Insufficient experiments
- To evaluate OOD detection quality, ID/OOD datasets should be stated and various metrics (e.g., AUROC) should be measured like other literature, e.g., Table 2 in [2]. Such OOD detection quality is important to evaluate the quality of uncertainty estimation.
- This paper provides experiments on only small-sized 10-class datasets, MNIST and CIFAR10. To verify the effectiveness of the proposed distillation method, other large-sized datasets should be tested, e.g., CIFAR-100, ImageNet.
- There is no ablation study on the effect of the number of size of heads in Hydra. To achieve similar performance to the ensemble, how many heads are required?

Concerns #3: Week efficiency
- As reported in Table 5, in the case of CIFAR10, Hydra has 14x more parameters and 6x more FLOPs. Despite such a large number of parameters, the performance gain seems to be incremental.
- A comparison with an ensemble with M=14 models should be tested because this ensemble has the same number of parameters compared to Hydra with M=50 heads. I think it might achieve good performance on the evaluation metrics.

Concerns #4: Incremental improvements
- Accuracy gain is too marginal even Hydra uses 14x more parameters.
- ONE [1] might be a stronger baseline because ONE achieves 94% accuracy on CIFAR-10 using ResNet32 with only 2~3 heads while Hydra achieves only 90% even it uses 50 heads. Moreover, since ONE has multiple heads, uncertainty estimation is also available. So it should be compared with the proposed method.
- In OOD detection tasks, Hydra underperforms Prior Networks on 5 of 8 datasets (note that PN (2.60) is better than Hydra (3.11) in the case of MNIST (test)). To overcome this gap, the proposed method requires more parameters.

[1] Zhu, Xiatian, and Shaogang Gong. "Knowledge Distillation by On-the-Fly Native Ensemble." Advances in Neural Information Processing Systems. 2018.
[2] Andrey Malinin and Mark Gales. "Predictive Uncertainty Estimation via Prior Networks." Advances in Neural Information Processing Systems. 2018.

review_title: Official Blind Review #3

Meta-review: This work introduces a simple and effective method for ensemble distillation. The method is a simple extension of earlier “prior networks”: it differs in which, instead of fitting a single network to mimic a distribution produced by the ensemble, this work suggests to use multi-head (one head per individual ensemble member) in order to better capture the ensemble diversity. This paper experimentally shows that multi-head architecture performs well on MNIST and CIFAR-10 (they added CIFAR-100 in the revised version) in terms of accuracy and uncertainty.

While the method is effective and the experiments on CIFAR-100 (a harder task) improved the paper, the reviewers (myself included) pointed out in the discussion phase that the limited novelty remains a major weakness. The proposed method seems like a trivial extension of the prior work, and does not provide much additional insight. To remedy this shortcoming, I suggest the authors provide extensive experimental supports including various datasets and ablation studies.  

Another concern mentioned in the discussion is the fact that these small improvements are in spite of the fact that the proposed method ends up using many more parameters than the baselines. Including and comparing different model sizes in a full fledged experimental evaluation would better convey the trade-offs of the proposed approach.

Decision: Reject