Before revision: We propose an effective approach to utilize pretrained speech and text models to perform speech-to-text translation (ST) . Our recipe to achieve cross-modal and cross-lingual transfer learning (XMTL) is simple and generalizable: using an adaptor module to bridge the modules pretrained in different modalities, and an efficient finetuning step which leverages the knowledge from pretrained modules yet making it work on a drastically different downstream task. With this approach, we built a multilingual speech-to-text translation model with pretrained audio encoder ( wav2vec ) and multilingual text decoder (mBART), which achieves new state-of-the-art on CoVoST 2 ST benchmark 1 for English into 15 languages as well as 6 Romance languages into English with on average + 2.8 BLEU and + 3.9 BLEU , respectively.
After revision: We propose an effective approach to utilize pretrained speech and text models to perform speech-to-text translation (ST) translation by efficient transfer learning from pretrained speech encoder and text decoder. Our key finding is that a minimalistic LNA (LayerNorm and Attention) finetuning can achieve crosslingual and cross-modality transfer ability with only finetuning less than 10\% of the pretrained parameters. This enables effectively leveraging large pretrained models with low training cost. Using wav2vec ) and multilingual text decoder (mBART), which achieves new state-of-the-art on CoVoST 2 ST benchmark 1 for English into 15 languages as well as 6 Romance languages into English with on average + 2.8 BLEU and + 3.9 BLEU , respectively.