NICT’s Cascaded and End-To-End Speech Translation Systems using Whisper and IndicTrans2 for the Indic Task (2024)

Abstract: This paper presents the NICT’s submission for the IWSLT 2024 Indic track, focusing on three speech-to-text (ST) translation directions: English to Hindi, Bengali, and Tamil. We aim to enhance translation quality in this low-resource scenario by integrating state-ofthe-art pre-trained automated speech recognition (ASR) and text-to-text machine translation (MT) models. Our cascade system incorporates a Whisper model fine-tuned for ASR and an IndicTrans2 model fine-tuned for MT. Additionally, we propose an end-to-end system that combines a Whisper model for speech-totext conversion with knowledge distilled from an IndicTrans2 MT model. We first fine-tune the IndicTrans2 model to generate pseudo data in Indic languages. This pseudo data, along with the original English speech data, is then used to fine-tune the Whisper model. Experimental results show that the cascaded system achieved a BLEU score of 51.0, outperforming the end-to-end model, which scored 19.1 BLEU. Moreover, the analysis indicates that applying knowledge distillation from the IndicTrans2 model to the end-to-end ST model improves the translation quality by about 0.7 BLEU.

heading: 1 Introduction and Related Work
text: Speech-to-text translation is crucial for breaking the language barriers during international activities, such as translating diverse languages in online meetings. Although high-resource language pairs often achieve excellent results, the performance for low-resource language pairs remains unsatisfactory (Radford et al., 2023; Joshi et al., 2020), such as English to Indic languages. This paper presents NICT’s submission to the Indic Track of IWSLT 2024, which includes translation directions from English to Hindi, Bengali, and Tamil. An overview of the cascade and end-to-end systems is illustrated in Figure 1.
Data scarcity is a significant challenge for the English to Indic languages ST task due to its lowresource scenario and we are using data-driven neural models. Data augmentation on speech and text data is an efficient way to address this challenge (Shanbhogue et al., 2023; Mi et al., 2022). Assisting information such as phonetic information (Cheng et al., 2021) and spectral features (Berrebbi et al., 2022), or knowledge transferred from related languages (Anastasopoulos et al., 2022; Gow-Smith et al., 2023; Song et al., 2020) can also enhance the performance. To this end, we use data combined data from three directions rather than using them separately.
Cascade and end-to-end (E2E) systems are two popular paradigms in ST with their advantages. In general, cascaded systems show higher translation quality (Agarwal et al., 2023) and end-to-end systems usually show lower latency and less modeling burden (Xu et al., 2023). To maximize the translation quality, we adopt the cascaded way and attempt to make full use of the recent advancements in ASR and MT fields (Sperber and Paulik, 2020). Following preliminary experiments, we decided to participate in the unconstrained setting, where we leverage pre-trained models such as Whisper and IndicTrans2 to develop our cascaded and E2E systems. Although additional datasets like IndicVoices are available for Indic languages (Javed et al., 2024), we refrain from using them due to concerns about test set overlap.
We use Whisper (Radford et al., 2023) as our ASR system. Unlike previous work (Wang et al., 2023a) who prompt Whisper without fine-tuning, we fine-tune Whisper-medium on the training data. Our results demonstrate significant improvements through fine-tuning. Although other ASR systems such as HuBERT (Hsu et al., 2021), wav2vec 2.0 (Baevski et al., 2020) and others (Communication et al., 2023; Wang et al., 2023b) exist, we adopt Whisper for its ease of use and its ability to deliver
65
high-quality transcriptions of English speech. We then cascade Whisper with IndicTrans2 (Gala et al., 2023) as our MT system. It supports high-quality translations across 22 popular Indic languages and outperforms the mBART50 model (Liu et al., 2020) and the M2M-100 model (Fan et al., 2020) in directions involving Indic language. Additionally, we explore the potential of the E2E system by employing knowledge distillation from the IndicTrans2 model into the Whisper model. Experimental results show that our cascaded systems are about 32 BLEU better than the E2E systems. Furthermore, E2E systems trained with distilled translations, which are obtained by translating English transcripts to Indic languages via IndicTrans2, tend to be about 0.7 BLEU points better than those using the originally provided gold standard translations.
The remainder of this paper is structured as follows: Section 2 describes the datasets and data preprocessing. Section 3 introduces our cascade and end-to-end models. Section 4 presents the ex-
perimental settings, results, and analysis. Lastly, Section 5 concludes the paper.

heading: 2 Data
text: We show the statistics of the original corpora and how we pre-process the raw data in this section.

heading: 2.1 Dataset
text: We use only the corpus provided on the official site, with statistics shown in Table 1. We do not leverage any extra data, although our systems are
built under the unconstrained condition. In the table, en, hi, bn, and ta represent English, Hindi, Bengali, and Tamil, respectively. We obtain the audio segments of textual sentences from the original files according to the offset and duration information. After segmentation, each data sample contains an audio segment in English, its transcription, and a translation in one of the three Indic languages. We observed that the data belongs to the spoken language domain, where the sentences are shorter compared to sentences in written texts. Moreover, there is almost no punctuation in English transcriptions and Indic language translations.

heading: 2.2 Pre-processing
text: To fine-tune a more robust Whisper model, we combine data from three English datasets as they belong to the same distribution.

heading: 3 Method
text: We describe the training and inference processes of the cascaded and E2E systems.

heading: 3.1 Cascaded System
text: The training and inference phases of the cascaded system are shown in Figure 1a. During training, we fine-tune the Whisper model using English audio paired with its English transcription. We then fine-tune the IndicTrans2 model using the English transcriptions and their corresponding translations in the Indic language. Although the Whisper model without fine-tuning can achieve reasonable performance, we found the format mismatch problem as presented in Figure 2. It is a type of domain mismatch between spoken language and written language, where there is less punctuation in the spoken language. However, this is prevalent in the training and development datasets, so we do not bother processing this further.
During inference, the English transcription generated by the fine-tuned Whisper model is input
into the fine-tuned IndicTrans2 model, which then produces the final output in the Indic language.

heading: 3.2 End-to-end System
text: The training and inference phases of the E2E Whisper model are shown in Figure 1b. During training, we first generate pseudo translation data using the IndicTrans2 model. We then use this pseudo data, instead of the gold transcription, to fine-tune the Whisper model. The motivation is to distill knowledge from a stronger translation model. The outputs of IndicTrans2, which are in a more consistent format, are easier for the Whisper model to learn than the human-annotated transcriptions. As shown in stage 1, we fine-tune the IndicTrans2 using English transcriptions and their translations in the Indic language. In stage 2, we generate pseudo translations for all English transcriptions in the dataset. Finally, we fine-tune the Whisper model using English audio data and these pseudo translations. During inference, we solely rely on the fine-tuned Whisper model to perform E2E ST.

heading: 4 Experiments

heading: 4.1 Settings
text: All our models are multilingual, achieved by combining all data into a single collection and using language indicator tokens to indicate the target language, as is the common practice. For the ASR module, we used the medium architecture of Whisper (Radford et al., 2023), which showed higher performance compared to the tiny, base, and small architectures. During fine-tuning, we set the learning rate to 1e−5, batch size to 16, and epoch size to 50. We allocated 10% of the total training steps for warmup and implemented early stopping if there was no improvement in loss after 1, 000 steps, with evaluations every 100 steps on the development set. For the MT part of our experiments, we used the IndicTrans2 (Gala et al., 2023) model. We fine-tuned using the scripts provided in the IndicTrans2 library1 including data preparation2 and fine-tuning3. Using our fine-tuned IndicTrans2 model, we performed standard beam search decoding with a beam of size 5.
1https://github.com/AI4Bharat/IndicTrans2 2https://github.com/AI4Bharat/IndicTrans2/ blob/main/prepare_data_joint_finetuning.sh 3https://github.com/AI4Bharat/IndicTrans2/ blob/main/finetune.sh

heading: 4.2 Main Results: Submitted Systems
text: Table 2 presents the results of our cascaded and E2E systems on the test set.
The scores are provided by the organizers, who do not provide a comparison with other participants at the time of writing this paper. Nevertheless, it is evident that cascaded systems outperform E2E systems by a wide margin. This indicates that data scarcity is a major problem limiting E2E system development for English to Indic language speech translation. We also provide scores for the same languages on 500 development set samples in Table 3, where we can see that there are similar trends as observed for the test set.

heading: 4.3 Impact of Distillation on E2E Systems
text: In Table 4, we present the differences between an E2E system trained on original translations and those trained on distilled translations. It is clear that distillation, performed by translating English transcriptions into Indic language sentences used as references for E2E systems, leads to a reasonable improvement of 0.7 BLEU.

heading: 5 Conclusion
text: This paper presented NICT’s submission to the IWSLT 2024 English to Indic speech-to-text translation task. We took advantage of the advancements in ASR and MT where we combined the Whisper model and IndicTrans2 model in our cascaded system. In our end-to-end system, we further utilize the pseudo translation data technique, also known
as knowledge distillation, to empower the Whisper model. Future work will focus on combining Whisper with IndicTrans2 jointly to train an even stronger speech translation system.

heading: Future Work
text: Future work will focus on combining Whisper with IndicTrans2 jointly to train an even stronger speech translation system