Kati8/ICLR_2023_2025 huggingface dataset:
authors: Bo Peng
authors: Jie Lu
authors: Guangquan Zhang
authors: Zhen Fang
reviews: 
Comments: This research investigates the process of knowledge distillation from a mixture-of-experts perspective. The authors redefine the distillation objective by incorporating teacher predictions as latent variables and introduce MoE-KD to deconstruct the vanilla classification task into a blend of simpler subtasks with the teacher classifier acting as a gating mechanism to assess the significance of the subtasks. Experimental results indicate that the proposed MoE-KD, functioning as an Expectation-Maximization algorithm, surpasses sophisticated knowledge distillation techniques across various benchmarks.
Strengths: 1. The concept of integrating teacher predictions as latent variables and redefining the distillation objective using the MoE framework is intriguing and commendable.  

2. The experiments consistently show that the proposed MoE-KD outperforms existing distillation methods.
Weaknesses: 1. As the proposed method is formulated as an Expectation-Maximization algorithm, the computation cost of the proposed method seems much more expensive than existing methods. It would be better if the authors could provide the computation comparisons in the experiments. 

2. It seems the term P^S_{theta}(Y^T=k|x_i) in Eqn. (12) intractable as the predictions of the teacher could not be derived by the student. How the authors address the problem? (If some misunderstandings exist, please correct me).

3. It would be better if the authors could provide one diagram or pseudo code of the whole algorithm to illustrate the framework of proposed method. In the current status, it seems a bit confusing how the student model in the experiments is adapted to a MoE version, and, after training, how the student model makes inference without the teacher predictions. 

4. The experiments are conducted on only CNN networks (both the teachers and the students). I wonder how such a method performs with vision transformers. 

5. Some highly related works are missing in the related work. For example, Xue et al., [1] also propose use a Mixture-of-Experts model to explain knowledge distillation. What are the differences between their work and the proposed method? The authors should include discussions and comparison with such highly related work.

[1] KDExplainer: A Task-oriented Attention Model for Explaining Knowledge Distillation. IJCAI 2021
Recommendation: 5
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: This paper proposes a new knowledge distillation method named MoE-KD, which partitions the predicted logits into multiple groups from a mixture-of-experts perspective, and then conduct distillation separately in each group. Then, by deriving the evidence lower bound of classification objective, the authors further formulate the distillation objective as an expectation-maximization algorithm to adress the nontrival learning problems in the vanilla log-likelihood objective. Experiments on CIFAR-100, ImageNet, and various transfer learning datasets are conducted.
Strengths: 1. This paper proposes a novel perspective of knowledge distillation by splitting the classification into multiple subtasks, which sounds interesting and effective to solve the distribution imbalance problem occured in multi-label classification distillation.

2. The authors conduct extensive experiments to validate the effectiveness. The proposed method outperforms previous KD methods in almost all the experimented settings. Notably, when with the latest strong training strategy and large teacher-student gap, the method can still achieve the optimal performance.

3. The paper gives sufficient comparisons to existing methods, and clearly elaborates the novelty of the method.
Weaknesses: 1. The methodology section may be difficult to follow and the readibility can be improved. The transition from Section 4.2 to 4.3 feels abrupt, and the shifts from EBLO maximization to EM algorithm lacks a clear, guiding explanation that ties the two subsections.

2. The ablation study in Table 4 is insufficient. It evaluates the impact of removing specific components from the complete method to demonstrate the effectivenesses. However, it does not provide insight into the standalone effectiveness when applied each to the KD baseline alone, if feasible. For example, one would like to see the effect of using gating function to partition the task into subtasks in KD baseline.

3. It would be better to design some toy experiments or show some visualizations to demonstrate the influence of the method, rather than simply comparing the final accuracies. For example, visualize the distribution of gates; how is the sharpness of the distribution? Any insights from the learned distributions?
Recommendation: 5
Reviewer confidence: 3
Soundness/correctness: 3
Clarity: 2
reviews: 
Comments: The authors raise an issue in knowledge distillation regarding cases where the ground truth and the teacher’s prediction do not align. To address this, they propose viewing the existing classification objective as a latent MoE (Mixture of Experts) from the teacher's perspective and suggest an optimization method based on the EM (Expectation-Maximization) algorithm.
Strengths: - It is interesting to interpret the existing objective from the perspective of the teacher as a latent variable.
- They provide mathematical explanation of MoE-KD.
Weaknesses: - The consistency between the problem statement, the proposed algorithm, and the problem-solving approach is lacking.
  - How frequently do cases occur in the training data samples where there is a significant discrepancy between the ground truth and the teacher’s prediction?
  - From what perspective is the proposed method better than the traditional direct KL divergence approach? Does it help facilitate learning toward the ground truth more effectively when there is a significant difference between the ground truth and the teacher’s prediction?
  - It would be beneficial to show the training results using only data samples with a large gap between ground truth and the teacher's prediction.
Recommendation: 5
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 2