The performance differences between No-KD and KD strategies observed in the study could be attributed to various factors beyond just the computation budget. While the computation budget plays a significant role in determining the amount of data processed and the model's learning capacity, other factors may also contribute to the performance disparities. One key factor is the ability of KD strategies to leverage the teacher's knowledge effectively during the distillation process. Advanced KD strategies like TinyBERT and MiniLM employ sophisticated techniques to distill knowledge from the teacher model more efficiently, leading to enhanced performance in the student model. Additionally, the inherent advantages of distillation techniques, such as transferring knowledge from a larger, pretrained model to a smaller model, can contribute to improved performance by leveraging the teacher's expertise and generalization capabilities. Furthermore, the specific distillation objectives and methodologies used in KD strategies can also impact the performance differences, as they determine how effectively the student model can capture and utilize the distilled knowledge from the teacher model.