JHU IWSLT 2024 Dialectal and Low-resource System Description (2024)

Abstract: Johns Hopkins University (JHU) submitted systems for all eight language pairs in the 2024 Low-Resource Language Track. The main effort of this work revolves around fine-tuning large and publicly available models in three proposed systems: i) end-to-end speech translation (ST) fine-tuning of SEAMLESSM4T v2; ii) ST fine-tuning of Whisper; iii) a cascaded system involving automatic speech recognition with fine-tuned Whisper and machine translation with NLLB. On top of systems above, we conduct a comparative analysis of different training paradigms, such as intra-distillation of NLLB, joint training and curriculum learning of SEAMLESSM4T v2, and multi-task learning and pseudo-translation with Whisper. Our results show that the best-performing approach differs by language pairs, but that i) fine-tuned SEAMLESSM4T v2 tends to perform best for source languages on which it was pre-trained, ii) multitask training helps Whisper fine-tuning, iii) cascaded systems with Whisper and NLLB tend to outperform Whisper alone, and iv) intradistillation helps NLLB fine-tuning.

text: Proceedings of the 21st International Conference on Spoken Language Translation (IWSLT 2024), pages 188–201 August 15-16, 2024 c©2024 Association for Computational Linguistics
Johns Hopkins University (JHU) submitted systems for all eight language pairs in the 2024 Low-Resource Language Track. The main effort of this work revolves around fine-tuning large and publicly available models in three proposed systems: i) end-to-end speech translation (ST) fine-tuning of SEAMLESSM4T v2; ii) ST fine-tuning of Whisper; iii) a cascaded system involving automatic speech recognition with fine-tuned Whisper and machine translation with NLLB. On top of systems above, we conduct a comparative analysis of different training paradigms, such as intra-distillation of NLLB, joint training and curriculum learning of SEAMLESSM4T v2, and multi-task learning and pseudo-translation with Whisper. Our results show that the best-performing approach differs by language pairs, but that i) fine-tuned SEAMLESSM4T v2 tends to perform best for source languages on which it was pre-trained, ii) multitask training helps Whisper fine-tuning, iii) cascaded systems with Whisper and NLLB tend to outperform Whisper alone, and iv) intradistillation helps NLLB fine-tuning.

heading: 1 Introduction
text: With recent developments in data-driven machine learning and Transformer-based models (Vaswani et al., 2017), speech translation (ST) systems (which accept spoken input in one language and automatically output corresponding text in another) have undergone major strides in performance (Radford et al., 2023; Barrault et al., 2023; Sperber and Paulik, 2020). While these works demonstrate the effectiveness of using large pretrained models for speech translation between high-resource language pairs and establish new state-of-the-art (SOTA) performance in these setups, less attention has been devoted to whether these advances also benefit lowresource language pairs, and how they compare with SOTA systems for these languages.
Some of the populations with the greatest need for ST tools are those speaking low-resource languages, which typically have less institutional support and funding for the development for NLP and speech tools (He et al., 2024; Kesiraju et al., 2023b; Karakasidis et al., 2023): some speak minority languages in the areas where they live and need translation tools to communicate across a language barrier, or to consume or search for information more effectively online (Neto et al., 2020). Certain populations speaking low-resource languages may also have low literacy rates or limited writing traditions in their native languages, increasing the imperative for speech-based, rather than text-based, translation systems (Besacier et al., 2006).
In this work, we developed ST systems for eight language pairs, as organized in the IWSLT 2024 Dialectal and Low-resource Speech Translation Shared Task. We approached this problem by leveraging systems pre-trained on a large amount of multilingual data and subsequently fine-tuning them for specific tasks: both end-to-end speech ST and cascaded ST (i.e. transcription followed by text-based translation). We compared different approaches and pre-trained models for each language pair, and we experimented with combining data from multi-
188
ple related languages into the same train set. Among the systems introduced, the approaches based on SEAMLESSM4T v2 (Barrault et al., 2023) outperform others for language pairs that it has seen during pretraining and for which supervised ST data are available (e.g. mar-hin, gle-eng, bho-hin, and mlt-eng). In other cases, a cascaded system is the most successful of the proposed approaches, namely, for apc-eng, bem-eng, que-spa, and tmh-fra.

heading: 2 Prior Work
text: A number of prior studies introduce methods aiming to address low-resource ST. In IWSLT’s evaluation for low-resource and dialectal ST 2023, Agarwal et al. (2023) note three practices that consistently help performance: (1) use of pre-trained models, (2) systems combining both end-to-end and cascaded models, and (3) synthetic data augmentation. These recommendations inform our decisions to fine-tune pre-trained models and experiment with both cascaded and end-to-end approaches.
Williams et al. (2023) used cascaded ST systems for Quechua-to-Spanish ST in IWSLT challenge 2023. Shanbhogue et al. (2023) fine-tuned pretrained speech models, and E. Ortega et al. (2023); Laurent et al. (2023) leveraged both pre-trained speech and text models in cascaded systems. Deng et al. (2023); Hussein et al. (2023) explored both end-to-end and cascaded ST. The most comparable submission to ours from the 2023 challenge was that of Mbuya and Anastasopoulos (2023), who used pre-trained models and applied them to several language pairs. With the findings and recommendations from prior work, we adapt a similar approach, but fine-tuning SEAMLESSM4T v2 (Barrault et al., 2023), Whisper (Radford et al., 2023), and NLLB (NLLB Team et al., 2022) instead of self-supervised learning representations (SSLR). Our approach differs from works described above, primarily in that we fine-tune models trained for automatic speech recognition (ASR), machine translation (MT), and ST, rather than fine-tuning representations obtained from language modeling objectives, such as wav2vec2 (Baevski et al., 2020), HuBERT (Hsu et al., 2021), XLS-R (Babu et al., 2022), or mBART (Liu et al., 2020a), for the tasks of ASR, MT, and ST. The findings from our systems shed light on the potential benefits provided by the pretrained multilingual models.

heading: 3 Task Description
text: On the challenge website this year,1 the organizers stated, "The goal of this shared task is to benchmark and promote speech translation technology for a diverse range of dialects and low-resource languages." To forward this aim, this year’s task focuses on ST for eight language pairs: Levantine Arabic to English (apc-eng), Bemba to English (bem-eng), Bhojpuri to Hindi (bho-hin), Irish to English (gle-eng), Maltese to English (mlt-eng), Marathi to Hindi (mar-eng), Quechua to Spanish (que-spa), and Tamasheq to French (tmh-fra). Levantine is one of the most spoken Arabic dialects, with the majority native-speaking populations in Syria, Lebanon, Palestine, and Jordan. Both Levantine Arabic and Maltese are Semitic languages of the Afroasiatic family. Bemba is a Bantu language of the Niger-Congo family, spoken by over 30% of Zambia’s population (Sikasote and Anastasopoulos, 2022). Bhojpuri, Hindi, and Marathi are Indo-Aryan languages; Hindi and Marathi are Scheduled languages in India and have government backing for their support, whereas Bhojpuri, like many other languages on the so-called Hindi Belt, lacks official status, has a much smaller writing tradition, and is only recently gaining attention in NLP (Kumar et al., 2022; Mundotiya et al., 2021; Bafna et al., 2023). Each of the source languages is low-resource, with Tamasheq, Bemba, and Levantine Arabic having the fewest Wikipedia articles overall (Robinson et al., 2023). Despite their low digital support, these languages have a large native speaker base, including Marathi’s 83 million, according to Ethnologue.2
The organizers provide different varieties of data for each of these language pairs. We used predominantly provided datasets, along with some external data, all of which are outlined in Table 1. We differentiate datasets of four types: ASR, indicating source language speech with corresponding transcriptions; E2E, indicating source language speech with corresponding target language translations that could supervise end-to-end ST; MT, indicating source language text with corresponding target language translations; and ST, indicating source language speech with both corresponding transcriptions and target language translations.
Though this year’s task accepts both unconstrained submissions, allowing the use of external
1https://iwslt.org/2024/low-resource 2https://www.ethnologue.com/
datasets and pre-trained models, and constrained submissions, our submission is limited to the unconstrained track, since all of our methods involved fine-tuning pre-trained models.

heading: 4 Proposed Methods
text: We introduce three primary frameworks, which are applied to different language pairs according to the availability of the data: (1) we fine-tune SEAMLESSM4T v2 for end-to-end ST using E2E data; (2) we fine-tune Whisper (Radford et al., 2023) for endto-end ST using E2E (and optionally ASR) data; (3) to form a cascaded ST system, we fine-tune Whisper for ASR using ASR data, then fine-tune NLLB for machine translation (MT) using MT data. The fine-tuning approaches are illustrated in Figure 1. Note that each ST dataset contains exactly one E2E, ASR, and MT dataset implicitly.
We explore various methodological additions to these methods. We look at joint fine-tuning and curriculum learning with the SEAMLESSM4T v2-based approaches. We investigate several finetuning setups for the Whisper-based systems, including pseudo-translation fine-tuning, multitask training with ASR and MT as well as ASR-only and ST-only fine-tuning. We also looked at intradistillation as a method of enhancing NLLB in MT. These ideas are further detailed below.

heading: 4.1 SEAMLESSM4T v2-based systems
text: Barrault et al. (2023) introduce SEAMLESSM4T v2, a model capable of end-to-end expressive and multilingual translations in a streaming fashion. SEAMLESSM4T v2 supports multilingual input and output in both speech and text modalities, with a dedicated sub-model handling each modality combination. It has 2.3B parameters and is pretrained on 1M hours of unlabeled audio in 143 languages, using the w2v-BERT XL architecture (Chung et al., 2021). It is then fine-tuned on text MT into English (x-eng) for 95 languages, ASR for 96 languages, ST into English for 89 languages, and speech-tospeech translation into English for 95 languages, and out of English eng-x for 35 languages. The pretraining languages of SEAMLESSM4T v2 include English, Irish, Maltese, Hindi, Marathi, and Arabic,3 but not Quechua, Tamasheq, or Bemba.
3We assume that the pretraining corpus also contains some Levantine and Tunisian Arabic, but these languages are not labeled distinctly from each other.
Our Systems We fine-tune SEAMLESSM4T v2 on E2E ST data, aiming to leverage the vast pretraining and ASR and ST capabilities of SEAMLESSM4T v2, which we expect to be beneficial in data-scarce scenarios. Although the SEAMLESSM4T v2 models are evaluated mostly on X-Eng/Eng-X directions in Barrault et al., 2023, we hypothesize that they will succeed in X-X directions post-finetuning, due to ASR pretraining in source and target languages. Note that this approach is only applicable to language pairs where E2E data are available (gle-eng, mlt-eng, aeb-eng, bem-eng, que-spa, tmh-fre, mar-hin, bho-hin). We also evaluate the zeroshot performance of SEAMLESSM4T v2 on these language pairs.
Experimental Setup For each language pair, we fine-tune SEAMLESSM4T v2-large for four epochs, with a learning rate of 1× 10−6 and batch size of 32. For que-spa translation, we use learning rate 1 × 10−8 for 15 epochs due to its small dataset size. For bem-eng and tmh-fra, a learning rate of 1×10−7 is used for training. The full hyperparameter list and details of hyperparameter tuning are included in Appendix A.1.

heading: 4.1.1 Multilingual training
text: Mixed Data Training For pairs with the same target language (gle-eng+mlt-eng, bho-hin+mar-hin), we fine-tune SEAMLESSM4T v2 on the combined dataset created by concatenating and shuffling the data, using the same hyperparameter settings as in Section A.1.
Curriculum Training Tunisian Arabic (aeb) and Maltese are both Semitic languages and share close linguistic relationships. We use a 12.6-hour subset of the Tunisian Arabic-to-English (aeb-eng) ST data used by Hussein et al. (2023) to conduct a curriculum training attempt using Tunisian as an augmentation for Maltese. The model undergoes initial fine-tuning on aeb-eng ST for two epochs with a learning rate of 1 × 10−6, followed by a 5-epoch-fine-tuning on mlt-eng at a learning rate of 1× 10−7.

heading: 4.2 Whisper-based systems
text: Whisper (Radford et al., 2023) is an end-to-end multi-task speech model based on a transformerlike encoder-decoder architecture. For this study, we focus primarily on its LARGE-V2 variant, which is pre-trained on 680k hours of multilingual ASR
and X-to-Eng speech translation data. During pretraining, the model is exposed to over 90 languages, including English, Marathi, Hindi, Maltese, and modern standard Arabic. However, Bemba, Bhojpuri, Quechua, Levantine Arabic, and Tamasheq, are absent from the pre-training data.
To address the gaps in language coverage and enhance model performance across diverse linguistic settings, we fine-tune the model in various ways tailored to specific scenarios. As the original model’s pre-training setup, we manipulate the prompt and supervision of the utterances at fine-tuning time to guide the model to perform different tasks, as detailed in the subsequent sections. In addition, for languages previously unseen by the model, we expand its vocabulary and embedding layer to create new language tags for the model to take condition on.

heading: 4.2.1 Fine-tuning paradigms
text: ASR-only Fine-tuning For language pairs with only ASR data or a limited amount of E2E or ST data, such as apc-eng and que-spa, Whisper is trained with only the ASR objective to serve as an ASR module in a cascaded system. The training and decoding prompt used is the conventional <|src-lang|>
. The resulting cascaded system’s MT module is an NLLB model described in § 4.3.
E2E-only Fine-tuning We train with Whisper’s ST-only objective for the tmh-fra pair. However, because Whisper is pre-trained for X-Eng ST only, instead of directly translating into French, we finetune the system to translate Tamasheq speech into
English text. Specifically, we translate the French labels of the E2E data into English using NLLB out of the box to formulate a tmh-eng E2E dataset. We then fine-tune Whisper with this dataset and utilize the trained model as the ASR module for a cascaded system, whose MT module is also NLLB. Similarly, English-to-French translation is conducted out-of-the-box.
Pseudo-translation For bho-hin and mar-hin language pairs, due to the absence of 3-way parallel ST data, the phylogenetic proximity between the languages, and the non-English-centric translation directions, we explore a novel adaptation of the model which we call pseudo-translation. Specifically, to enable Whisper to translate into non-English languages, we prompt the model to "transcribe" the source language speech signals with the target language transcription prompt, i.e. <|tgt-lang|>
. Conceptually, this is equivalent to treating Bhojpuri and Marathi as pseudo-Hindi speech and conducting ASR (an approach that is especially linguistically motivated in the case of Bhojpuri, as it is closely related to Hindi). Such design is motivated by the fact that Whisper is pre-trained with weakly supervised data, which implicitly empowers the model’s audio-conditioned language model to perform some extent of de-noising. Consequently, we may model the non-English translation process as a noisy transcription task with the proposed prompts.
Multi-task Learning Previous yet unpublished experiments suggest that multi-task learning (MTL) tends to improve the model’s performance across
downstream metrics. Hence, for bem-eng and mlt-eng, as the 3-way parallel ST data is sufficient, we fine-tune Whisper on both the ASR and E2E ST tasks with E2E X-Eng ST being the end goal. In particular, we create the ASR and E2E ST dataset objectives respectively with their corresponding prompts, i.e. <|src-lang|>
 and <|src-lang|>
, and concatenate them to form a multi-task dataset for fine-tuning, allowing the sampler to draw samples with different supervisions stochastically. Kesiraju et al.’s (2023a) use a large amount of Marathi ASR data (He et al., 2020; Bhogale et al., 2022) for Marathito-Hindi ST. Therefore, we further extend the idea of constructing data to mar-hin, which has abundant non-parallel ASR and E2E ST data yet no 3-way parallel data. We combine the pseudotranslation technique to perform non-parallel ASR and E2E pseudo-ST multi-task training.4

heading: 4.2.2 Whisper training details
text: We employ a range of techniques to expedite the training of Whisper and optimize the utilization of our hardware resources. Specifically, we adopt Low-Rank Adapters (LoRA) (Hu et al., 2021), gradient checkpointing (Chen et al., 2016), and Zero Redundancy Optimizer (ZeRO) (Rajbhandari et al., 2020) to fine-tune all Whisper models. We allow trainable decomposed weight matrices with a rank of 200 for the embedding layer, all the attention layers, and the first feed-forward layer in the transformer blocks, resulting in a total of 289,157,200 trainable parameters, approximately 16% of the original model’s parameter count.
We apply conventional speech data augmentation in the fine-tuning process, including SpecAug (Park et al., 2019) and speed perturbation (Ko et al., 2015) with parameters 0.9, 1.0, 1.1.

heading: 4.3 NLLB fine-tuning
text: NLLB Team et al.’s (2022) NLLB is an encoderdecoder framework designed for extensive multilingual translation across more than 200 languages. It incorporates the sparsely gated mixture of experts (Du et al., 2022) to balance enhanced modeling capacity with efficient training and inference. Training of the NLLB model involves three objectives— translation loss, denoising loss, and language modeling loss—all calculated using the negative loglikelihood (NLL) loss function but with distinct
4Note that in this case, since the ST data is used for pseudotranslation, only 
 tags are used.
datasets. Translation loss utilizes clean parallel texts, while denoising loss employs techniques from denoising auto-encoders (Liu et al., 2020b) that introduce noise into the source text. The language modeling objective of NLLB uses monolingual data to train the decoder.
Vanilla Fine-tuning We fine-tune the opensource NLLB model5 with the released MT corpora for apc-eng, bem-eng, and que-spa. Specifically, we use the distilled 600M-parameter NLLB model as the base model and fine-tune the model with NLL loss. Following NLLB Team et al. (2022), we append language tokens on both source and target sequences during training and force decode the target language token during inference. We use a learning rate of 1× 10−4 and set the maximum number of target tokens per batch to 1600. We train all translation models on a single V100 machine and accumulate gradient updates every 4 steps.
Fine-tuning with Intra-distillation We also finetune with intra-distillation (ID), which is an effective task-agnostic training method, aiming to encourage all parameters to contribute equally (Xu et al., 2022, 2023). Given an input batch, ID needs to forward pass the model K times to obtain K outputs and each time a random subset of parameters is zeroed out. The core idea of ID is to minimize the difference of these K outputs to approximately minimizing the contribution gap of the parameters that are zeroed-out, because the K outputs are forced to be the same with different zeroed parameters. Let {p1, · · · , pi, · · · , pK} denote the K outputs. The ID loss is then formulated by the X-divergence (Xu et al., 2022) to minimize the difference of K outputs as
Lid = 1
K
K∑
i=1
KL(pi ∥ p̄) +KL(p̄ ∥ pi)
where p̄ = 1
K
K∑
i=1
pi
Let the original task loss be Li for the ith pass. Then, the total loss is a combination of the original task loss and ID loss, given as
min 1
K
K∑
i=1
Li + αLid
5Available at: https://huggingface.co/docs/ transformers/en/model_doc/nllb
where α is a hyper-parameter to control the strength of ID.

heading: 5 Results and Discussion
text: Table 2 displays the results for all of our MT systems. We calculate scores using the same BLEU (Papineni et al., 2002) configuration as the task organizers.6 We include scores from internal Dev and Test sets when available, as well as the official Eval scores. Details of data splitting are in Appendix A.2. The results show that SEAMLESSM4T v2 systems perform best for half of the language pairs: bho-hin, gle-eng, mar-hin, and mlt-eng. Cascaded systems employing Whisper and NLLB for MT performed best for the others: apc-eng, bem-eng, que-spa, and tmh-fra. (Note these first three language pairs employed Whisper for ASR and a fine-tuned NLLB model for MT, while tmh-fra employed Whisper for X-Eng ST and NLLB out of the box for MT into French.)

heading: 5.1 End-to-end ST
text: The SEAMLESSM4T v2 models’ poor performance on bem-eng, que-spa, and tmh-fra is likely due to the absence of Bemba, Quechua, or Tamasheq in its pre-training corpus. We include zero-shot results for SEAMLESSM4T v2 out of the box in Table 3, which illustrate that the pre-trained model already performs well on mlt-eng and gle-eng,7 but poorly on unseen language pairs. We remark that our fine-tuning process brings notable improvements for bho-hin, mar-hin, and mlt-eng. In particular, SEAMLESSM4T v2 is successful for bho-hin despite not being pre-trained explicitly on Bhojpuri data, possibly because the Hindi pretraining data contains some Bhojpuri, or because SEAMLESSM4T v2 is capable of extrapolating fairly well to Bhojpuri given its high linguistic similarity to Hindi. Interestingly, the mixed data training (comb.) for language pairs sharing a target language does not significantly improve performance for either source language, though we expected it to benefit the lower-resource pair. In the case of gle,mlt-eng, there are domain differences (read speech vs. telephonic speech) between the
6With sacrebleu signature nrefs:1 | case:lc | eff:no | tok:13a | smooth:exp | version:2.0.0.
7There is a considerable discrepancy between the gle-eng dev and test scores from IWSLT 2023, with the latter being suspiciously high. Mbuya and Anastasopoulos (2023) suggest that the inflated test scores may be due to overlap between train and test sets.
fine-tuning corpora, possibly resulting in unhelpful or negative interference; Irish and Maltese are also not linguistically related, limiting cross-lingual transfer. On the other hand, with bho,mar-hin, Marathi and Bhojpuri both belong to the Indic subfamily of languages, and the speech translation data for both respective language pairs is from the news domain, averaging about 7 seconds each. The lack of success of joint fine-tuning for both these setups resonates with the findings of Sun et al. (2023), which presents several experiments showing that multilingual training for speech translation may not always benefit low-resource languages. We also note that curriculum training likewise did not improve performance for mlt-eng.
In our evaluation of Whisper systems, we emphasize two significant observations. Firstly, as anticipated, the BLEU scores for the mar-hin and bho-hin language pairs validate the efficacy of the proposed pseudo-translation method. This finding not only demonstrates that the model is capable of handling non-English translations with minimal fine-tuning, but also underscores its adaptability to linguistically similar language pairs. Secondly, the consistent performance gain observed with Whisper MTL over Whisper E2E as illustrated by the mar-hin results underscores the advantages of multi-task learning. This method treats finetuning on multiple tasks as involving one primary task and several auxiliary tasks, which collectively contribute to enhanced outcomes on all tasks involved.

heading: 5.2 Cascaded ST
text: Cascaded ST via fine-tuned Whisper for ASR and fine-tuned NLLB for MT is our best-performing approach for apc-eng, bem-eng, and que-spa, though it is much better for apc-eng and bem-eng than for que-spa. The relatively low performance of que-spa can be possibly attributed to it being a non-English-centric translation direction.
Table 4 presents the ASR performance of the fine-tuned Whisper models on 5 language pairs with different objectives. Those trained with the ASR-only objective are used solely as the ASR module in cascaded systems, while the systems trained with the multi-task learning objective are used for both direct translation and ASR for cascaded systems. Interestingly, we observe that for Bemba, the CERs (25.1 for dev and 17.9 for the test1 set) are significantly lower than the WERs. We find through manual inspection that the model
tends to make minor spelling errors, presumably due to its unfamiliarity with the language’s writing system, as suggested by the decent proficiency in its translation performance. This may cause error propagation in cascaded ST.
In our MT module, we implemented intradistillation to enhance ST results by balancing the contributions of the model parameters. Consistent with prior studies Xu et al. (2022, 2023), intradistillation consistently improves performance across all evaluated translation directions, with the most significant enhancement observed for
que-spa. MT performance was reasonably high for the three language pairs for which we employed cascaded ST. The cascaded approach for mlt-eng performs poorly, likely because our Maltese bitexts were noisy. Additionaly, NLLB has already been pre-trained on Maltese and may not benefit further from the noisy post-training.

heading: 6 Conclusion and Future Work
text: In this work, we describe our submitted systems for all eight language pairs in the IWSLT 2024 LowResource Language Track. We explore various fine-tuning approaches for large publicly available pre-trained models, compare end-to-end and cascaded systems, as well as investigate the benefits of joint and curriculum training, multitask learning, as well as intra-distillation. We find that the best-performing strategy is language-pair dependent, with fine-tuned SEAMLESSM4T v2 generally performing best on languages that are included in its pretraining corpus. Fine-tuned Whisper generally performed better with multi-task fine-tuning than standard fine-tuning, and better still when employed in a cascaded system with fine-tuned NLLB (with best results employing intra-distillation).
For future improvements, augmenting MT finetuning data with ASR hypotheses, as in Gow-Smith et al. (2023), could equip NLLB better for cascaded ST. Future work could also employ data augmentation of text and speech data, as in Shanbhogue et al. (2023), via textual back-translation (Sennrich et al., 2016), speech synthesis for augmentation (Rossenbach et al., 2020; Robinson et al., 2022), or other methods. Lastly, future research could employ the use of SSLR, or employ the large amounts of raw
audio available—particularly for Tamasheq—to train SSLR systems, following Gow-Smith et al. (2023).

heading: A Additional Experimental Details
text: A.1 SEAMLESSM4T v2 hyperparameters For SEAMLESSM4T v2 models, the longest audio length is truncated at 30 seconds. To ensure full reproducibility of the result, a random seed of 42 is deployed. We perform a minimum hyperparameter search for each language pair between the learning rate of {10−5, 10−6, 10−7}. For each language pair, we fine-tune a SEAMLESSM4T v2-large for
four epochs, with a learning rate of 1× 10−6 and batch size of 32. For Quecha-to-Spanish (que-spa) translation, a learning rate of 1× 10−8 is used for training 15 epochs due to its small dataset size. For all the training trials, a constant learning rate scheduler and a warm-up step of 50 is used. During inference, the maximum generation length is constrained to 256 tokens with greedy decoding.
A.2 Split details
We split data into train, dev, and test when possible, for tuning and internal evaluation. We split Makhoul et al.’s (2005) Levantine Arabic ASR data, Sikasote et al.’s (2023) Bemba ST data, He et al.’s (2020) Marathi ASR data, Cardenas et al.’s (2018) Quechua ASR data, and Tiedemann’s (2012) que-spa MT bitext ourselves using a 90-5-5 split. We split Sellat et al.’s (2023) apc-eng MT bitext ourselves with a 90-5-5 split but then performed our internal test on a 1000-line subset of the held out data. For the large mlt-eng MT bitexts from Bañón et al. (2023, 2020), we split the data ourselves with a 99-0.5-0.5 and a 98-1-1 split, respectively. We also split Bhogale et al.’s (2022) large Marathi ASR dataset ourselves with a 99-0.5-0.5 split. We used the creator’s own splits for Sikasote and Anastasopoulos’s (2022) Bemba ASR data, Agarwal et al.’s (2023) mar-hin E2E data, Tiedemann’s (2012) que-spa MT bitext, Zanon Boito et al.’s (2022) tmh-fra E2E data, and the Hindi ASR data from Common Voice. We did the same with Agarwal et al.’s (2023) gle-eng E2E data, using the test set from the 2023 challenge as our internal test set. For the mlt-eng ST data from Common Voice and Hernandez Mena et al. (2020) and the que-spa ST data from Ortega et al. (2020), we used their own train and dev splits and then split the dev set in half to create an internal test set. We used Agarwal et al.’s (2023) own train and dev splits without creating an internal test set.
B Instance Length Distribution
We show the length distribution in Figure 2 and Figure 3. Overall, most datasets show a normal distribution with a slightly skewed tail except for que-spa, the amount of instances for which is the smallest. However, we identify some extraordinarily long instances in bem-eng training set. These outlier instances can lead to out-of-memory instances if left untreated. Therefore, we truncate
the instances that are over 30 seconds when training SEAMLESSM4T v2 and limit the generation length to 256 new tokens.

heading: Future Work
text: For future improvements, augmenting MT finetuning data with ASR hypotheses, as in Gow-Smith et al. (2023), could equip NLLB better for cascaded ST. Future work could also employ data augmentation of text and speech data, as in Shanbhogue et al. (2023), via textual back-translation (Sennrich et al. , 2016), speech synthesis for augmentation (Rossenbach et al. , 2020; Robinson et al. , 2022), or other methods. Lastly, future research could employ the use of SSLR, or employ the large amounts of raw
audio available—particularly for Tamasheq—to train SSLR systems, following Gow-Smith et al. (2023)