The Privileged Students: On the Value of Initialization in Multilingual Knowledge Distillation
Authors: Haryo Akbarianto Wibowo, Thamar Solorio, Alham Fikri Aji


Reviewer paper summaries: The paper investigates the impact of knowledge distillation (KD) and model initialization on the performance of multilingual models. The authors focus on the efficiency and effectiveness of copying the teacher model’s weights directly to the student model to enhance initialization. They discover that this weight-copying approach significantly improves downstream task performance more than the distillation process itself across various multilingual settings. Additionally, the paper demonstrates that efficient weight initialization preserves multilingual capabilities even in low-resource scenarios.

The paper explores the impact of knowledge distillation (KD) and model initialization in multilingual natural language processing (NLP) tasks. The authors propose to initialize the student model’s by selectively copying the teacher model’s weight inspired by TinyBERT and DistillBERT. The findings suggest that this initialization strategy significantly enhances the student model's performance, surpassing the benefits of the distillation process itself. The paper also showed that fine-tuning on just English during distillation can still improve performance when copying the weight from the teacher model.
Summary of strengths: - The paper offers a detailed analysis of the different components of knowledge distillation, specifically model initialization and the distillation process itself. It addresses a critical research gap by identifying which factors contribute most to the overall performance of multilingual models.
- This paper is clear and easy to follow.

1. The paper introduces a fresh perspective on the role of initialization in KD, particularly in multilingual contexts. This provides valuable insights into distillation in general.

2. The paper includes a thorough experimental analysis, comparing different initialization methods and their impact on model performance. The proposed weight-copy initialization method significantly improves performance on multilingual tasks, even in low-resource settings, which is practical for building efficient multilingual models with limited data.

3. The paper provides a detailed analysis section to study the impact of initialization on convergence speed.
Summary of weaknesses: - My main concern is that the paper provides limited novelty beyond existing methods. Although the weight-copying approach is applied in a multilingual context, the method itself is not entirely new, as it draws heavily from DistilBERT. The paper might benefit from a more innovative methodology or additional unique contributions.
- The proposed method is only validated on one base model.
- Some sections of the paper, particularly those detailing the technical aspects of the methodology, could be clearer. For example, In equation (6), why does teacher layer j map to student layer i*2?

1. The paper showed that copying the weights of the teacher model shows significant performance improvements, especially in the zero shot setting in section 4.1, but the results are limited to sentiment classification tasks. It is not clear whether the same conclusions can be made for other tasks such as question answering or generative tasks.

2. It would be interesting to see whether the shown pattern is generalizable across different teacher-student model ratio and different teacher model sizes.

3. In terms of cross lingual generalizability investigated in the model, the discussion seems a bit limited. It would be interesting to investigate which unseen languages benefit the most and the least from the different KD and non-KD settings. This could provide more insights into the nature of multilingual LMs and strengthen the paper, rather than just describing the benchmark performance.

4. Copying the weights from the teacher model essentially gives the model more information compared to the from scratch initialization, because the teacher weights are either pre-trained on other data or fine-tuned on the training set. Roughly speaking, the from-teacher setting has seen 2X times the training data (ignoring the fact that only parts of the parameters are copied). Then, a more fair comparison would be to increase the fine-tuning epochs of the from-scratch initialization setting. If this still significantly underperforms the from-teacher setting, it is a stronger evidence that the copy-weight approach works.
Comments, suggestions and typos: Comments:
- The paper lacks novel methods, but it could benefit from stronger theoretical justification. Specifically, it should explain more on why knowledge from seen languages can transfer to unseen languages through teacher weight copying. Providing deeper theoretical insights would enhance the overall argumentation. 
- Unfortunately, I find Section 4, especially Sections 4.2 and 4.3, unclear. The concept of copying weights from the teacher to help student models converge is intuitive and doesn’t require extensive explanation. Section 4.1 is more useful, but as I mentioned before theoretical justification could be conducted. 

Typos:
- Line 028: these models --> These models

“Across multilingual settings” appeared many times in the paper but I find this vague and unnecessary.

Equation 6: is S and T switched here? It should be SA_S^j = SA_T^{j*2} according to the text descriptions.

Line 436: use “`” for left single quotation mark
Overall assessment scores: 2.5

4