On Knowledge Distillation for Direct Speech Translation
Authors: Marco Gaido, Mattia A. Di Gangi, Matteo Negri, Marco Turchi (Fondazione Bruno Kessler, Università di Trento, AppTek Aachen Germany, 2020)
Topic: Machine Translation

Abstract
Direct speech translation (ST) has shown to be a complex task requiring knowledge transfer from its sub-tasks: automatic speech recognition (ASR) and machine translation (MT). For MT, one of the most promising techniques to transfer knowledge is knowledge distillation. In this paper, we compare the different solutions to distill knowledge in a sequence-to-sequence task like ST. Moreover, we analyze eventual drawbacks of this approach and how to alleviate them maintaining the benefits in terms of translation quality.

Introduction
Speech translation (ST) refers to the process of translating utterances in one language into text in a different language. Direct ST is an emerging paradigm that consists in translating without intermediate representations AUTHOR. It is a newer and alternative approach to cascade solutions AUTHOR, in which the input audio is first transcribed with an automatic speech recognition (ASR) model and then the transcript is translated into the target language with a machine translation (MT) model. The rise of the direct ST paradigm is motivated by its theoretical and practical advantages, namely: i) during the translation phase it has access to information present in the audio that is lost in its transcripts (eg. prosody, characteristic of the speaker. For instance, the pitch of the voice is a cue for the sex of the speaker. Although the gender is a social aspect and does not depend on physical attributes, in many cases sex and gender coincide, so systems relying on this are likely to have a better accuracy than those that do not have access to any information regarding the speaker AUTHOR.), ii) there is no error propagation (in cascade systems the errors introduced by the ASR are propagated to the MT, which has no cues to recover them), iii) the latency is lower (as data flows through a single system instead of two), and iv) the management is easier (as there is a single model to maintain and no integration between separate modules is needed). On the downside, direct ST suffers from the lack of large ST training corpora. This problem has been addressed by researchers through transfer learning from the high-resource sub-tasks AUTHOR, multi-task trainings AUTHOR, and the proposal of data augmentation techniques AUTHOR. In this work, we focus on the transfer learning from MT. The classic approach consists in pre-training the decoder with that of an MT model. Its benefit, however, is controversial: indeed, AUTHOR showed that it is effective only with the addition of an adapter layer, but this has not been confirmed in AUTHOR, while in AUTHOR it always brought improvements. Another, more promising possibility consists in distilling knowledge from an MT model. Knowledge distillation (KD) is a knowledge transfer technique introduced for model compression AUTHOR. A small student model is trained computing the KL-divergence AUTHOR with the output probability distribution of a big teacher model. Although KD was introduced in the context of image processing, its effectiveness suggested its adoption in other fields. Specifically, AUTHOR showed that using an MT system as teacher brings significant improvements to direct ST models. However, they did not compare the different methods to distill knowledge in sequence-to-sequence models AUTHOR and they did not analyze possible negative effects of adopting this technique. In this paper, we analyze different sequence-to-sequence KD techniques (word level, sequence-level, sequence interpolation) and their combination in the context of direct ST. Then, we study the effect of the best technique on a strong system trained on a large amount of data to reach state-of-the-art results. We show that word-level KD is the best approach and that fine-tuning the resulting model without KD brings further improvements. Finally, we analyze the limitations and the problems present in models trained with KD, which are partly solved by the final finetuning.

Conclusion
We presented and analyzed the benefits and issues brought by distilling knowledge from an MT system for direct ST models. We compared the different KD techniques and our experiments indicated that the best training procedure consists in a pre-training with word-level KD and a fine-tuning without KD. Then, we showed that KD from MT models causes an increased gender bias, omission of sentences in multi-sentential utterances and more generic word/verbal-tense choices. Finally, we demonstrated that a fine-tuning helps resolving these issues, although the exacerbation of gender bias is not solved, but only alleviated.