Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a knowledge distillation approach with multiple teachers by introduce a weighting on teachers, and the weights are optimized in a bi-level manner. It achieves state-of-the-art results on several GLUE tasks compared with other model compression methods.
strength_and_weaknesses: Strength:
* The authors propose a reasonable bi-level optimization for the weighting of teachers and achieve strong results.

Weakness:
* The paper writing is comparatively weak, e.g., The figure 1 is even harder to understand than the main text part.
* Lack of meaningful analysis and discussion towards the result numbers. For example, overall the proposed approach performs on par with MoBERT. Why in some tasks it performs worse? Is the proposed approach orthogonal to the previous methods? Will the performance be better if multiple techniques are applied together?
clarity,_quality,_novelty_and_reproducibility: Some places of the paper writing is unclear. The proposed method is novel, and the experimental setting looks clear.
summary_of_the_review: This paper proposes a reasonable approach to weight multiple teacher in knowledge distillation, and gets good results on the GLUE benchmark, but the results are not discussed and analyzed thoroughly, and the paper writing is comparatively weak.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposed a new knowledge distillation paradigm for BERT, AutoSKDBERT, which stochastically sampled a teacher from the predefined teacher team by a categorical distribution. They proposed a two-phase optimization framework to better incorporate the categorical distribution and student model. To alleviate the gap between categorical distribution optimization and evaluation, they proposed an SSWO strategy for optimization. The experimental results show superiority compared with strong baselines.
strength_and_weaknesses: ### Pros:
1. The proposed method is simple yet effective, and it provided a more flexible way to give full play to the teacher team.
2. The experimental results show obvious superiority compared to the SOTA baselines on the GLUE benchmark.

### Cons:
1. One critical problem is how to choose the ineffective teacher number, i.e., $m$. It is a very important parameter in the whole paradigm. However, as claimed in Sec 3.2.2, $m$ is a hyperparameter and it may differ a lot for different tasks.
2. As claimed in Sec 3.3.2, AutoSKDBERT delivers 25 categorical distribution candidates and selects the best one. How to produce 25 candidates in detail?
3. As stated in Sec 3.3.1 and Sec 4.1, are there 25 categorical distribution candidates in both phase-1 and phase-2? Or phase-1 will only produce 25 candidates in the ablation study experiments?
4. In Sec 3.3.1, what does *trains them from scratch* mean? What is *them*? the student model?
3. In figure 2, we can see $T_{14}$ plays a significant role, but $T_{13}$, which has the same number of parameters as $T_{14}$, is much weaker. The only difference is that $T_{14}$ adopts *whole word masking*. Since *whole word masking* is signally effective, why not adopt it on other teacher models? 
4. Because $T_{14}$ is a new type of BERT model with better performance, do the baselines use such a model as one of the teacher models? If not, what are the results without $T_{14}$? What if the baselines use the same teacher models?
6. Since some teacher models, e.g., $T_{14}$, perform distinctly better, it will be good to compare with the best single-teacher KD.
5. What's the number of teacher models of the baselines? Is it a fair comparison if the baselines use fewer teacher models than 14?
8. In Sec 4.2, for the CR setting, the training involves all the teachers, why not use the same way in evaluation? Then it will be consistent between training and evaluation and can compare with SSWO more fairly.
1. It seems unclear about the categorical distribution initialization module. Why is the initialized $\theta_i$ like shown in Figure 1, i.e., why are 0.199, 0.201, 0.199, 0.200, and 0.201?


clarity,_quality,_novelty_and_reproducibility: - This paper provides a new paradigm to distill a student model from multiple teacher models. It's mainly about application and techniques. Some critical parameters need manual tuning.
- Some clarification of details is not very clear. But the main idea and motivation are clear.
- This work should be original.

summary_of_the_review: Overall, the proposed method is simple and easy to implement. And it could raise better performance than SOTA. But there are lots of tricks to boost the performance, like 1. generating 25 candidates, training 25 student models, and choosing the best one; 2. mutual-tuning $m$; 3. using a stronger teacher model (*whole word masking*). However, there lack of details and analysis of the tricks. Further, the clarification should be clearer.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Given a predefined teacher team, the paper aims at finding an optimal categorical distribution of these teacher models for student network distillation. Rather than simple ensemble learning (uniform weight), the paper proposes a two-stage selection and optimization strategy. In the first stage, ineffective teachers are identified and discarded by optimizing the categorical distribution. In the second stage, the categorical distribution is further tuned for effective teacher teams. The paper conducts experiments on the GLUE benchmark and make comparison with the most relevant works such as DistilBERT and TinyBERT.
strength_and_weaknesses: **Strength**:

1. The formulation of the problem is clear and the paper is well-written.
2. Most relevant works are discussed. The experiments show that AutoSKDBERT achieves good results.

**Weaknesses**:
1. To show that optimizing the categorical distribution is necessary, I wonder whether a single-teacher KD is enough. Table 1 shows some results. However, I would like to see the complete results from T1 to T14. Besides, does "Best Single Teacher" refers to the teacher that can achieve the best performance for student distillation or the teacher itself that achieves the best performance?
2. The novelty of the proposed method is limited. 
clarity,_quality,_novelty_and_reproducibility: The formulation of the problem is clear and the paper is well-written. However, the novelty is quite limited. The paper presents experimental details for reproducibility.
summary_of_the_review: Although the proposed method achieves decent results, the technical novelty is quite limited. Optimizing the categorical choice while optimizing the network parameters is a widely studied bilevel optimization problem in neural architecture search and other areas. Another important concern is that a more simple baseline should be considered such as single best teacher and Gumbel softmax optimization to systematically study whether it is necessary to optimize the categorical distribution.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, authors proposed AutoSKDBERT, a new knowledge distillation paradigm for BERT compression, that stochastically samples a teacher from a predefined teacher team following a categorical distribution in each step, to transfer knowledge into student.

With extensive experiments on GLUE benchmark show that the proposed AutoSKDBERT achieves state-of-the-art score compared to previous compression approaches on several downstream tasks.
strength_and_weaknesses: Strength 
Authors proposed a novel learning algorithm to learn from multiple teachers.

Weaknesses
This paper only focused on bert. My understanding is the proposed method can be adapted to other network architecture. The impact would be larger if such method is independent of the network architecture of teacher/student. 
clarity,_quality,_novelty_and_reproducibility: Clarity:

Overall the clarity is good.

Some questions:
In the second paragraph in Introduction, "the ensemble of multiple teachers are not always more effective than the single teacher for student distillation........ On the other hand, between the large-capability teacher and small-capability student, there is a capability gap which can be prone to unsatisfactory distillation performance" I wonder why the capability gap of student and teacher can be used to explain why the *ensemble* of multiple teacher? Is it because the capacity is low in student, so adding more capacity (by ensemble) does not help?

In table 1, I am a bit confused why different data set uses so many different metric. It would be helpful to report 2-3 metrics for all dataset, to avoid potential cherry picking.


Quality, Novelty


Here are some questions:
It would be also interesting to see the training time for distilling of the proposed method compared to other existed method.

Giveh there are multiple metrics, it is hard for readers to understand the magnitude of the improvement. It would be helpful to add information like standard error/confidence interval for each metric.

For the comparison benchmark, which of them are using multiple teachers? It is important to have a fair comparison given the improvement can come both from the proposed distilling as well as more teachers. I would recommend add some simple heuristic based distillation method with multi-teacher as benchmark.


Reproducibility

Overall the experiment description is clear, and data used in experiments are open-sourced public data. However, it may still hard to reproduce without the experiment script.

For "The hyper-parameters for student distillation.", it would be helpful to determine why different dataset has different parameter. Are those hyper-parameters recommended by other research? 

*More important*, "The hyper-parameters for categorical distribution optimization." how such parameter are selected? Is the algorithm sensitive to such parameters?

Also it may be helpful to add how multi-teacher is used? Is it just use all teachers in all examples? As there are so many distill methods, it would be helpful to add details for reproducibility.
summary_of_the_review: I choose the final decision 'marginally below' given my questions above as well as the proposed method for multi-teacher is only studied on BERT. If authors are able to clarify and extend the work, I would be happy to reconsider the decision.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a new knowledge distillation approach that uses multiple teacher models with weights. The teacher weights are optimized in a bi-level manner. The approach shows improved performance on GLUE benchmark compared to certain baseline methods. The reviewers have found that more experimental comparisons and analyses are needed to validate the effectiveness of the approach, such as whether the improvement is due to the new knowledge distillation approach per se or the use of multiple teachers, and others. The paper presentation also needs to improve. 
justification_for_why_not_higher_score: * Lack of sufficient experimental comparisons and analyses
* Writing needs to be improved
Decision: Reject