## 2 Related Work  
[PERSON] et al. [1] successfully employed KD for the image classification task to compress large networks or ensembles of networks into smaller models that achieved similar performance to the large networks. [PERSON] and [PERSON] [2] were the first to apply KD techniques for neural MT (NMT). Subsequently, many researchers have explored various applications of sequence-level KD for NMT [7; 8; 9; 10; 11; 12]. More specifically, sequence-level KD [2] is an effective technique which involves training small student models based on pseudo-target sequences generated by large teacher models. This technique significantly reduces both model size and inference time, typically with minimum loss in performance [9; 10; 11; 13; 14; 15]. In other words, this approach is straightforward, as it only requires generating forward-translated synthetic data (i.e., soft target labels) using the teacher model, and has proven to be effective in training smaller models with minimum loss in performance [16; 17; 18; 19]. In sum, there are two stages for the KD pipeline: the first stage involves generating distilled data using large teacher models, and the second involves training and fine-tuning of the student models using the distilled data created in the first stage. The literature on KD for NMT is rich and extensive. For example, researchers have used student models to fine-tune teacher-generated chains of thought (CoT) (CoT is a series of intermediate reasoning steps which is incorporated in LLMs to improve their ability to perform complex reasoning) [20; 21; 22; 23; 24; 25] and other prompting trajectories (rationales) [20; 21]. Researchers have also explored alternative prompting methods [20; 21] to determine how sequence-level distillation generated by the teacher can be more effective in incorporating reasoning or decision-making capabilities in students. In fact, most sequence-level KD uses known correct or incorrect trajectories to fine-tune student models [26; 27; 28]. Multi-domain KD, proposed in [29], distills multiple expert models into a single student. The authors kept the model architecture and model capacity fixed, showing improvement in BLEU [30] scores across all domains over multi-domain models without any increase in translation time or memory usage. They used a fixed depth and architecture of the teachers, making their approach architecture-independent and easy to combine with other multi-domain NMT models. [PERSON] et al. [31] enriched fine-tuning data, with noisy rationales being rephrased; however, they did not select annotated samples in order to save on teacher budget. [PERSON] et al. [32] investigated the impact of KD training on CO\({}_{2}\) emissions and conducted empirical evaluations of various KD strategies. Their study included key parameters (i.e., CO\({}_{2}\) emissions, translation time, translation quality (BLEU)) with explicit measurement of GPU power consumption during translation. By experimenting with different teacher model variants and continuously monitoring energy usage, they estimated the computational cost of translation and assessed student model performance. Their findings suggest that the proposed methods offer a computationally efficient alternative to standard KD approaches, achieving similar accuracy levels while lowering energy demands. Rather than just aiming to reduce model size or improve inference speed, their primary focus was on improving the overall efficiency of the KD process. More recently, [6] tested several pseudo-label filtering methods in a speech translation task, including proxy models, uncertainty quantification (entropy and geometric mean of confidence scores), negative log-likelihood, multimodal embeddings, and perceptual evaluation of speech quality. Their experiments demonstrate that these unsupervised techniques enable distilled models to outperform or match the performance of supervised distillation setups while also being more computationally and memory efficient than the larger teacher models, particularly on dialectal speech.