Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the problem of progressive distillation, which is defined as approximating a large model with an ensemble of multiple small models. The paper poses this problem as a two-player zero-sum game where a “distribution player” poses a distribution over example assignments to learners to maximize loss, and the “hypothesis player” chooses hypotheses for each learner to minimize loss.
strength_and_weaknesses: The problem setting is interesting and well-motivated. Standard distillation (one big model to one small model) is likely to result in underfitting, whereas considering an ensemble of small models increases the capacity while ideal latency can be kept low through parallelization.

The intermediate layer connection idea (section 3.1.2) is interesting - instead of just considering a bigger model class, you can start from embeddings from previous generation nets; this is probably better for when you want to do early-exit or anytime inference with limited compute.

In experiments, the paper reports benefits in six different datasets, each with different network architectures. However, I think the evaluation would be more convincing if it considered more baselines such as a basic boosting algorithm.

Minor comment: the paper says “SVNH dataset” repeatedly; is this a typo for SVHN (Street View House Numbers)?
clarity,_quality,_novelty_and_reproducibility: The writing wasn't perfect, but clear enough to understand the main points. I think section 3 in particular was structured in a way that was hard to parse at first. To my knowledge, the problem setting and approach is novel. The paper and appendix do not provide enough information to reproduce the experiments, such as chosen hyperparameters or exactly how the smaller capacity models were constructed.
summary_of_the_review: I think the problem setting and method is interesting, but the experimental evaluation is somewhat weak.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a method (B-DISTILL) to distill an (expensive) trained model into an ensemble of weak learners. These learners are meant to provide efficient inference, with progressively better results as the ensemble size increases: this allows for a straightforward way to trade off accuracy and compute at inference time, in cases where inference may be interrupted abruptly, or when variable levels of accuracy are expected.

The distilled ensemble is obtained by solving a two player zero-sum game, with convergence and generalization bounds provided under weak assumptions. In order to the $n+1$-th model to refine the prediction of the previous $n$ models, the authors also allow each successive model to _reuse_ the feature transformation of the previous ensemble member.

Experimentally, authors verify their method on simulated, vision, speech, and sensor-data classification tasks, and across a variety of architectures.
strength_and_weaknesses: # Strengths
- The paper is very well written, and the exposition is overall clear (aside from issues with baselines and $K$ matrices mentioned below)
- The method has a clear application and is of similarly clear value to the community (anytime and efficient inference).
-  The authors provide convergence guarantees ($\mathcal O(\sqrt{\ln 2N / T})$ for $T$ the ensemble size and $N$ the dataset size)

# Weaknesses
- The description of the experimental baselines should be much clearer. As it stands, I found it difficult to be certain whether the authors consider Fixed-model to be an ensemble (which appears to be the case), and how successive ensemble members differ from each other.
- It does not appear that the authors included early exit baselines; it would strengthen the paper significantly to compare to work such as (Dennis et al., 2018), despite the fact that such work is not distillation based.
- The Figures (and Figure 3 in particular) are difficult to view (due to size, even on a screen and zooming in), and to understand in general. Am I correct in understanding that round $T$ reports the FLOP percentage for an ensemble of size $T$ (assuming each round of FIND-WL succeeds)? For the FLOP analysis, can you confirm that distinction between fixed-model and B-Distill is inconsequential?

# Questions to the authors
- Could you clarify why you need to split the matrices $K$ into $K^+$ and $K^-$?
- I'm confused about why $p$ in (3) has is of shape $N \times L$. I understand that $N$ is for the distribution over samples of the dataset; is the $L$ dimension to reweight the loss for different logit dimensions on each sample?
- In Theorem 2, is $\mu$ a distribution (and the inequality stochastic dominance), or a scalar?
- Experimentally, how often did FIND-WL fail?

# Nitpicks
- Please add a $y$-axis to Figure 5. Are you reporting accuracy?
- Consider moving the legend above the sub-figures when possible, to improve readability. 
- Is there an extraneous index $j$ in (3)?
clarity,_quality,_novelty_and_reproducibility: **Clarity**: Overall, the paper is clear, although the current discussion surrounding the probability matrices $K$ and the exact setup for the baselines is insufficient. Should these issues be addressed, there would be no issues with clarity.

**Quality**: The results seems of good quality (assuming my understanding of the baselines is correct).

**Novelty**: The work is fairly novel, although its experimental results should be placed more clearly within the context of early exiting/anytime inference.

**Reproducibility**: The paper appears mostly reproducible; however, the authors should include the detailed architectures used as base learners in all experiments.


summary_of_the_review: This paper addresses an interesting problem (distilling a model into an ensemble of weak learners to control inference cost and accuracy trade-offs). Taking a game theory approach, authors introduce an algorithm with convergence guarantees, and show experimentally that their approach successfully trades off computational costs with model accuracy. 

However, I found it quite difficult to understand the detail of the experimental results, and of the baselines in particular. This work would be very much strengthened by a clearer experimental section, and the inclusion of baselines from the multiple-instance learning / anytime inference literature.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose the problem of progressive distillation: approximating a large model with an ensemble of smaller models. They seek a solution that \emph{decomposes} the large model, such that the quality of the approximation improves monotonically with the size of the ensemble. They formulate progressive distillation as a two player zero-sum game and present B-DISTIL, an algorithm for solving this game. One notable feature of their algorithm is that it allows models to reuse intermediate representations from other models in the ensemble. They evaluate their algorithm on image classification (ResNets on CIFAR10 and DenseNets on SVHN), speech classification (LSTM on Google-13), and sensor data (GRU on DSA-19). They find that their B-DISTIL-generated ensembles allow favorable tradeoffs between computation time and accuracy.
strength_and_weaknesses: **Strengths**

Good motivation

Clever method—reusing intermediate representations within an ensemble is an elegant idea.

**Weaknesses**

Experiments are not very convincing; baselines and benchmarks are insufficient.

Minor clarity issues.

clarity,_quality,_novelty_and_reproducibility: **Empirical Rigor**

In general, the experiments feel like a bit of an afterthought.

The experiments are lacking baselines. What are the current best methods to reduce latency at the cost of reduced accuracy? One simple baseline that comes to mind is training a smaller model from scratch. For a given efficiency ensemble latency, find a model architecture that has similar latency and compare accuracy. It’s possible you do this in Figure 5, but I did not find that figure particularly easy to understand.

CIFAR10 and SVHN are not particularly convincing benchmarks. I would encourage the authors to attempt to scale their experiments to at least CIFAR100 or Tiny-ImageNet.

**Clarity**

You mention that student models have “significantly fewer parameters and resource requirements”. This is very vague and needs to be quantified.

X- and Y-axes in Figure 5 need labels.

summary_of_the_review: Clever idea, needs more rigorous empirical evaluation. Could be publishable with more thorough experimentation.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The reviewers and meta reviewer all carefully checked and discussed the rebuttal. They thank the authors for their response and their efforts during the rebuttal phase.
The response helped resolve some important concerns (e.g., strengthening of the experiments and their reproducibility).
The reviewers and meta reviewer all acknowledge that the submission proposes a principled construction of an ensemble, with appealing properties (e.g., anytime inference), within the context of distillation.

At this stage, while the submission does not suffer anymore from major flaws, there are still some arguably important aspects that warrant further consolidations/investigations, for instance:
* Improve the clarity of the exposition of the method (especially in Section 3.1.2)
* Stress-test the scalability of the approach (e.g., the matrices K’s grow with both the number of samples and classes — what happens for standard datasets like ImageNet-21k with ~12M points and 21k classes?)
* Further demonstrate the practical significance of the approach thanks to an actual, realizable gain in efficiency as directly measured by inference time (without using a conversion from FLOPs). Having a concrete implementation, compatible with modern hardware constraints (e.g., TPU with fixed-size batch sizes), is challenging for methods based on sequential-prediction and/or early-exit logic.   

Because of the extremely competitive landscape of the submissions this year, the paper remains under the cut and has ultimately not been selected for acceptance.
We are convinced that the suggestions above will help strengthen the paper for a future resubmission, which the reviewers and meta reviewer all encourage.

justification_for_why_not_higher_score:  * The practical significance of the approach is not sufficiently vetted and demonstrated:
   * Scalability
   * More concrete evidence of the resulting gain in inference time.
* Exposition of the method lacks clarity.
Decision: Reject