Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a two stage distillation framework. By falling back hard samples to the large models and use small models for easy ones, such an approach could achieve both efficiency and performance. 
main_review: Traditionally model distillation usually sacrifice performance for improved efficiency. In this paper, the authors proposed a two stage framework based on distillation which can achieve both the modeling benefits of the large models and preserve the efficiency with lightweight models. The key component is a well designed loss function for the student model which can help in the delegation process. Overall, I believe the paper is well motivated, the method is described clearly, and thorough experiments are conducted. 
summary_of_the_review: I only have one major concerns towards the experiment. The authors propose several different approaches in the paper, including class-specific distillation and margin based distillation, and corresponding delegation methods. However in the experiment section it looks like they only focus on class-specific distillation and some related methods which are introduced in the appendix. 

I would actually suggest the authors to make the paper more self-contained, if the methods are introduced in the main text they should be evaluated as well. And some alternative methods can be included in the appendix. Also, I feel like it would be better to have a high level summary in terms of the pros and cons for each method, along with the recommendation for users in practice. 

Minor issues: line 4 of Introduction, upto -> up tp
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a new method to do efficient inference with a teacher-student setup for model distillation. It contrasts itself to traditional model distillation because it is optimized for the situation where rare and difficult cases can still be sent to the original teacher network.
main_review: The paper tackles an important and practical problem: So-called "edge" devices such as mobile phones cannot process heavy duty models. However, requirements on bandwidth and latency may prevent one from sending every example to the cloud for processing on a big GPU capable machine. So, this paper presents a hybrid approach where most of the cases can be processed on the "edge" but those edge devices can elect to send rare cases to the cloud for a heavy duty model.

Despite being an important problem, the solutions in the paper "label delegation" and "margin delegation" are very heuristic and arbitrary. Solutions like early-exit networks seem much more principled in comparison. They are also crucially dependent on hyper-parameters, which the paper does not address tuning.

Heuristic solutions are still noteworthy for publication if they beat all other known approaches in a rigorous benchmark. However, in this paper, most of the solutions benchmark the method against variations on itself.

The paper would be stronger if it had more of these comparisons.
summary_of_the_review: The paper attempts to solve and important problem, but is very heuristic and does not validate that it is better than the prior art.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a two-stage distillation framework to improve inference efficiency and reduce the dependency on large teacher models. The goal of this framework is to only use the large/teacher model for difficult and rare examples and to use the student, smaller model for the more frequent easy examples. The procedure is composed of a training phase and an inference phase. During the training phase, the dataset is separated into two subsets: one containing the hard/difficult examples and one containing the easy examples. Using a loss incorporating a combination of label-smoothing and distillation, the student model is taught to be certain on the easy examples (by using distillation of the large model) and to be less certain on the hard/difficult examples. During the inference phase, in order to route the hard examples to the larger teacher model, several possible methods are proposed relying in particular on whether the margin of the student’s softmax is higher or lower than a threshold. 

The authors validate their two-stage framework empirically on three benchmark image datasets (CIFAR-100, ImageNet-1k and ImageNet-21k) and two benchmark NLP datasets (SQuAD and MNLI). 
main_review: Strengths
- It is a well written and clear paper providing a two-stage inference framework that can be used to reduce the dependency on large models and thus improve inference efficiency. 
- The relevance of the framework is validated empirically on benchmark datasets in Computer Vision and NLP. 

Weaknesses
- Even though the framework is clear and easy to use. The results are not very convincing as the performance of the two stage framework is very close to distillation which is also efficient and does not rely on the teacher model during inference. In particular in Figure 2, 3 and 4 curves of the proposed approach seem to be very close to the baseline standard distillation. The gains are clearer in Table 1 and Table 4. Could the authors add a similar table for Figure 4? 
- It is not clear which approach (CD-I, CD-II, CD-III) and which parameters (alpha) to use in practice. Could the authors provide a summary of the findings regarding recommendations on which approach to use in which cases based on the experiences? 

Questions
- In 4.1, the authors mention that « As a result of standard distillation, the lite student behaves as a well-calibrated model ». Could the authors justify this statement? I am not sure to understand why it is necessarily the case. 
- In Table 1, the CD-II approach obtain a much lower Accuracy score than the other approaches (0.24), Could the authors explain that?

Typos
- Introduction, third paragraph : « real-world data »
- 3.2, 2nd paragraph : « the teacher scores »

summary_of_the_review: Even though the performance of the two-stage framework seems to be higher than the standard distillation, I am not sure yet if the gains of the proposed approach compared to standard distillation justify the acceptance of this paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies efficient inference problem for large models. It proposes to train a small student model, and performs inference for easy data on the student model, and for hard data on the original large model. Experiments show the proposed method performs better than the simple baseline of standard distillation model.
main_review: Strength: The paper is fairy well written and easy to follow.

Weaknesses:
There are multiple weaknesses. The idea of combining a small student model and the original large model for inference is pretty intuitive (I am not saying intuitive is a bad thing) but does not demonstrate well. It is not clear why such a method can be interesting and be better than existing methods. For example, Section 2 discusses some related work on efficient inference. However, these are never compare with the proposal method, neither theoretically nor empirically. Especially, in the experiments, only the standard distillation baseline is compared with the proposed method, which is quite limited and fail to demonstrate the advantages of the proposed method. Thus, the motivation of the proposed method is quite unclear. In addition, since the focus of this paper is on inference efficiency, it is necessary to compare the inference time, in addition to the current accuracy.
summary_of_the_review: This paper fails to demonstrate the motivation and advantage of the proposed method for efficient inference of large model. It does not meet the acceptance bar of the conference.
review_title: null

Meta-review: This paper a distillation framework where a light-weight student model is trained to handle easy (frequent) instances, while the large teacher model is still used to handle the more difficult (rare) inputs. The models are trained to perform well in this two-stage inference setting. Experiments are conducted on computer vision and NLP tasks. While the idea is potentially interesting, the experimental results are fairly weak and not very convincing.
Decision: Reject