Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors presented a model distillation method that can actively select the training samples for distillation. The method is efficient and robust - in the sense that the method suffers less from inaccurate soft labels. The proposed method is parameter-free. The authors further verified the efficacy of the proposed method empirically.
strength_and_weaknesses: Strength:

1. The author presented a mathematical formulation that captures the objective of training on informative soft labels that are accurately labeled by the teacher.

2. The solution takes a simple and intuitive game-theoretical viewpoint, which can be obtained in near-linear time without tuning the hyperparameter.

3. The author also presents extensive empirical evaluations that demonstrate the effectiveness of the proposed framework.

Questions and Weaknesses:

1. At the beginning of page 3, the author state that "a very limited number of ground truth label queries can be made for the points in $X_i$ throughout the distillation process". From my understanding, the ground truth label queries are different from querying the teacher model for soft labels. How exactly are those ground truth label queries used?

2. The game-theoretic approach can deal with very bad quality soft-label, which seems somewhat pessimistic. Is there any way we can estimate the quality of the soft-label without using oracle? For instance, what will happen if we use the margin of the teacher-generated soft-label as a proxy for $c_i$ in equation (2) ?

clarity,_quality,_novelty_and_reproducibility: The paper is well-written and the proposed approach is simple and interesting.
summary_of_the_review: This paper proposed an intuitive, interesting and effective approach for active distillation. With the theoretical understanding and the extensive empirical results, I think this paper is above the acceptance threshold.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes an active learning-based approach for knowledge distillation. It assumes that the teacher model does not have access to a fully labelled dataset either, and there is a cost associated to every call of the teacher model to produce soft labels for the unlabelled data. The aim is to minimize the total cost of making teacher model inference calls while maximizing the student accuracy learned from the soft labels produced by the teacher model. Its core idea is to select the samples where the teacher model is more confident instead of those where the student model is most unconfident. Experimental results on multiple real datasets confirm the effectiveness of the proposed technique. 
strength_and_weaknesses: Strengths:

1. The proposal that considers the problem of importance sampling for simultaneous efficiency and robustness in knowledge distillation is novel and interesting.

2. The proposal is supported by non-trivial theoretical analysis. 

3. The proposed techniques are shown to be effective on multiple real datasets. 

Weakness:
The paper motivates with KD using large NLP models such GPT-3. It would be good to incorporate experiments on NLP datasets and tasks. 


clarity,_quality,_novelty_and_reproducibility: The paper is well written overall and presents an interesting idea supported by solid experimental results. 

The paper can use a proofread to fix minor presentation issues such as: What is X_B in line 2 of Section 2.1?
summary_of_the_review: The paper considers the problem of importance sampling for simultaneous efficiency and robustness in knowledge distillation which is an interesting idea. The proposed solution is supported by non-trivial theoretical analysis and strong results on real datasets. Overall, the paper presents a solid contribution that has the potential to intrigue follow-up studies. 

=== Update after rebuttal ==

Thanks for the response. I have no further questions.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies robust active distillation that can simultaneously achieve efficiency and robustness. The query rule of the algorithm follows from the solution of a minimax problem that tries to maximize the "gain" given a limited number of mislabels. The minimax optimization problem can be solved in nearly linear time (with closed-form solutions provided in Thm 1 and Cor 1).  The authors conduct extensive experiments which empirically confirm the efficacy of their proposed method.
strength_and_weaknesses: Strength: 
1. The authors proposed an interesting minimax optimization problem to select data points, which ensures both efficiency and robustness. The idea behind this optimization problem seems novel to me.
2. The authors identify a theoretically-motivated weight parameter such that the expected gain is never negative. As a result, the proposed algorithm can be operated in a parameter-free manner. 


Weaknesses:
1. The minimax problem seems to be only solved in the case when b=1 (in Thm 1). When the budget is larger than 1, it seems that the authors simply multiply the solution (obtained when b=1) by the budget. Can we directly solve the minimax optimization problem when the budget is greater than 1?
2. The choice of m is estimated based on sample inaccuracy, which seems to require hard-labeled data points. Are there ways to get around with this?

Questions: 
1. In algorithm 1, why both the teacher and the student are trained on the same hard-labeled data points? I thought the teacher should be trained on a much larger dataset in order to provide soft-labels that are better than the ones generated by the student.
2. Following the above question, can you provide more explanations on why the student can eventually outperform the teacher in your experiments?
clarity,_quality,_novelty_and_reproducibility: This paper is generally well-written and easy to follow. The minimax formulation to simultaneously achieve efficiency and robustness seems to be quite novel to me. The authors also provide details (e.g., hyperparameters) to reproduce their results.
summary_of_the_review: Based on the detailed reviews above, I think this paper has proposed some interesting and novel ideas. I also would like to hear the author's rebuttal regarding the weaknesses/questions. 

====after rebuttal====

Thank you for your response. I would like to keep my current scores. I believe the paper can be made stronger if the authors can provide some analyses for the case with b>1.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studied large-scale knowledge distillation about the trade-off between its efficiency (i.e., the number of soft-labels to be queried to the teacher model) and robustness (the accuracy of soft-labels provided by the teacher model). A theoretically guaranteed method was proposed to query the soft-labels of data that are simultaneously informative and correctly labeled by the teacher model. The authors did a good job in the rebuttal to successfully address the concerns from the reviewers. All reviewers agreed that the novelty and significance are above the bar and thus we should accept it for publication.
justification_for_why_not_higher_score: The submission is not more competitive since its problem setting (i.e., large-scale knowledge distillation) is very specific.
justification_for_why_not_lower_score: All reviewers agreed that the novelty and significance are above the bar.
Decision: Accept: poster