NAVER LABS Europe’s Multilingual Speech Translation Systems for the IWSLT 2023 Low-Resource Track
Year: 2023

Abstract:
This paper presents NAVER LABS Europe’s systems for Tamasheq-French and QuechuaSpanish speech translation in the IWSLT 2023 Low-Resource track. Our work attempts to maximize translation quality in low-resource settings using multilingual parameter-efficient solutions that leverage strong pre-trained models. Our primary submission for Tamasheq outperforms the previous state of the art by 7.5 BLEU points on the IWSLT 2022 test set, and achieves 23.6 BLEU on this year’s test set, outperforming the second best participant by 7.7 points. For Quechua, we also rank first and achieve 17.7 BLEU, despite having only two hours of translation data. Finally, we show that our proposed multilingual architecture is also competitive for high-resource languages, outperforming the best unconstrained submission to the IWSLT 2021 Multilingual track, despite using much less training data and compute.

heading: 1 Introduction
text: The vast majority of speech pipelines are developed for high-resource languages, a small percentage of languages that have ample amounts of annotated data available (Joshi et al., 2020). However, the assessment of systems’ performance based only on high-resource settings can be problematic, since it fails to reflect the real-world performance these approaches will have in diverse and smaller datasets. Moreover, as around half of the world’s languages are considered to be not only low-resource, but also from oral tradition (i.e., without a written form), there is an urgent need for speech technology that can operate robustly in such low-resource settings (Bird, 2011). In this context, the IWSLT conference1 proposes low-resource speech translation (ST) challenges that allow the speech community to realistically benchmark ST approaches
∗Work done during an internship at NAVER LABS Europe.
†Equal contribution 1https://iwslt.org/
using diverse and representative datasets. This paper describes NAVER LABS Europe’s (NLE) submission to two of the language pairs from the IWSLT 2023 (Agarwal et al., 2023) Low-Resource Track: Tamasheq-French (Taq-Fr) and QuechuaSpanish (Que-Es).
Most successful approaches for tackling scenarios where ST data is scarce perform transfer learning across languages and modalities, leveraging multilingual pre-trained models for both speech and text (Anastasopoulos et al., 2022). However, due to the large number of parameters of current Transformer-based (Vaswani et al., 2017) approaches, training such systems is computationally expensive and not accessible to everyone. NLE’s submission focuses on a multilingual parameterefficient training solution that allows us to leverage strong pre-trained speech and text models to maximize performance in low-resource languages.
We present new SOTA results for the TaqFr pair (17 hours of training data) that represent a 57% BLEU increase compared to the results achieved by Khurana et al. (IWSLT 2022 postevaluation).2 This same system achieves 23.6 BLEU on the IWSLT 2023 test set, an improvement of 7.71 BLEU compared to the second best result submitted this year. We also present SOTA results in the unconstrained setting for the QueEs pair (2 hours of training data), while maintaining most of the performance in the Taq-Fr pair. In addition, to showcase the usefulness of our parameter-efficient multilingual solution we evaluate it on the high-resource setting of the IWSLT 2021 Multilingual Task (Anastasopoulos et al., 2021). We find that our approach outperforms the best IWSLT 2021 submission (FAIR, Tang et al., 2021), despite training considerably fewer parameters (-64%), and using substantially
2https://www.clsp.jhu.edu/ jsalt-2022-closing-presentations/
144
less training data and compute. This paper is organized as follows. We first describe the architecture and training settings of our multilingual ST systems in Section 2. We next list the resources we use in Section 3. Section 4 presents our results in both low and high-resource settings. Lastly, we highlight the zero-shot potential of our approach in Section 5 and present our concluding remarks in Section 6.

heading: 2 System Description
text: In this work we focus on a parameter-efficient training solution that allows us to input the features from a pre-trained speech representation model into a pre-trained multilingual MT model, producing translations from both speech and text in multilingual settings. This setting also allows us to leverage automatic speech recognition (ASR; i.e. speech-to-transcript) data. The general architecture is presented in Figure 1. The architecture is considered parameter-efficient because a small portion of its parameters are trained (bottom encoder layers and small adapters layers).
Architecture. We initialize our models with a pre-trained multilingual MT model, which we adapt to the ST task by inputting features extracted with a frozen pre-trained speech representation model. The MT model is also frozen, except for the bottom 2 or 3 encoder layers and small adapter modules (those introduced by Bapna and Firat (2019), with bottleneck dimension 64) added after each encoder and decoder layer. As we show in our results, the fine-tuned encoder layers are able
to map the speech features into the representation space of the pre-trained MT model and the adapters can help with domain adaptation (and possibly help alleviate the length mismatch). At inference, this model can be used for MT with very little memory overhead: the convolutional layers and adapters are disabled, and the bottom encoder layers are swapped with those of the initial pre-trained model.
Training settings. We train on 4 V100 GPUs (80GB) for up to 200 000 updates, with a maximum batch size of 4 000 source features (or 80 seconds of audio) and accumulated gradients over two batches.3 We sample language pairs with a temperature of 3.4 We validate every 5 000 updates and perform early stopping on valid BLEU for the language pair(s) of interest, with a patience of 5, averaging model weights across the last 3 checkpoints.5 We find best results using a single convolutional layer with stride 2, which downsamples the sequence of speech features by a factor of 2. The other hyperparameters are listed in Appendix Section A.1.
3This corresponds to a total of 32 000 features per update, or 640 seconds of audio. In practice, with padding, each update corresponds to approximately 80 utterances or 530 seconds of audio.
4pk = u 1/3 k / ∑ u 1/3 i where uk is the utterance count for language pair k. 5While all the configurations presented in this paper use checkpoint averaging, we later re-trained our contrastive submission for Taq-Fr and found virtually the same results without it.

heading: 3 Resources

heading: 3.1 Pre-trained Speech Representation Models
text: We experiment with different versions of two speech representation models: HuBERT (Hsu et al., 2021) and wav2vec 2.0 (Baevski et al., 2020). We do not fine-tune these models in any of our configurations, but instead use them as feature extractors (see Figure 1). Because of this, our models are sensitive to the layer we extract features from. Pasad et al. (2021) argue that, for wav2vec 2.0 models that are not fine-tuned on ASR, speech features from middle layers tend to have a higher abstraction from the speech signal, which is beneficial to downstream tasks. The results from Boito et al. (2022b) seem to confirm this observation holds for low-resource ST. To the best of our knowledge, there is no similar investigation for HuBERT models.6
Table 1 presents the speech representation models we experiment with. The Tamasheq model is a monolingual wav2vec 2.0 Base model trained on 243 h of Tamasheq speech. The Niger-Mali is a wav2vec 2.0 Base model trained on the same Tamasheq speech data plus 111 h of French, 109 h of Fulfulde, 100 h of Hausa, and 95 h of Zarma. This gives 658 h in total. The data for both models is sourced from the Niger-Mali audio collection (Boito et al., 2022a). The unreleased mHuBERT-Tamasheq model uses this same audio collection for training, while also including Common Voice (Ardila et al., 2020) data in four other languages (English, French, Arabic and Kabyle), resulting in 5 069 h of speech. XLSR-53 (56k hours) and XLS-R (500k hours) are massively multilingual wav2vec 2.0 Large models covering 53 and 128 languages, respectively. Neither of these two multilingual models have seen Tamasheq or Quechua
6We hypothesize that layer selection is less important for HuBERT architectures due to the multi-iteration approach that increases signal abstraction at each iteration.
speech during training.7

heading: 3.2 Pre-trained Multilingual MT Models
text: To initialize our ST models, we first experimented with mBART for many-to-many translation (mBART50NN; Tang et al., 2020), but found the NLLB-200 models (Costa-jussà et al., 2022) to give better results. We experiment with the dense NLLB models of various sizes: the distilled 600M-parameter and 1.3B-parameter versions, and the 3.3B-parameter version. We end up using the larger versions in our submissions (1.3B and 3.3B). Note that NLLB covers 202 languages, including Tamsheq and Quechua, which is not the case for mBART. At the same model size, despite covering more languages, NLLB is also a stronger machine translation model overall than mBART. Also, unlike mBART, it is not English-centric.
Contrary to Tang et al. (2021), we keep the original mBART or NLLB vocabularies of size 250k and do not train any embeddings. Instead, like Berard et al. (2021), we find that it is possible to filter the vocabulary at test time to only cover the languages of interest, significantly reducing the memory footprint of the model with a minor reduction in performance.8 We can also filter the vocabulary and embeddings before ST fine-tuning and achieve the same performance as with the full vocabulary without needing to train any embeddings. See Table 14 in Appendix for a comparison of these approaches. In order to study the zero-shot translation capabilities of our models (i.e., translating to languages and language pairs unseen at training), we do not apply vocabulary filtering to the configurations presented in the main paper.
7Appendix Table 16 lists all models with links for downloading checkpoints, when available.
8With NLLB, 44k tokens are enough for a 100% coverage of the training data (mTEDx, TED-LIUM, Quechua, Tamasheq), or 35k when restricting to our Taq-Fr setting. This represents a reduction of more than 200M parameters.

heading: 3.3 Datasets
text: We tackle the low-resource setting by building multilingual systems that utilize both ASR and ST data in the languages of interest (Tamasheq and Quechua), and in high-resource directions whose target language is of interest (French and Spanish). Note that we also include X→English data, as we initially planned to participate in the IrishEnglish task. Including more data in high-resource languages has several advantages. Firstly, it has a regularization effect that prevents us from immediately overfitting the low-resource training data. Secondly, this enables knowledge transfer from common target languages and from similarly-sounding source languages.9 Thirdly, as we build multilingual ST systems by mapping the speech representation vectors into the same space as the multilingual MT model, our goal is to produce a model that is as multilingual as possible, not specializing in one specific language. Our results show that training on multiple languages at once achieves this effect, while also producing good zero-shot ST results.
Table 2 presents statistics for the datasets provided by the IWSLT 2023 organizers. The Que-Es dataset10 is an unreleased dataset prepared for this year’s challenge. It corresponds to a translated subset of the Quechua ASR data (“Siminchik”) from Cardenas et al. (2018). The Taq-Fr dataset was introduced by Boito et al. (2022a). Table 3 presents statistics for the datasets in high-resource languages. English ASR data comes from TEDLIUMv2 (Rousseau et al., 2014), and the other data comes from mTEDx (Salesky et al., 2021). Appendix Table 15 lists the datasets used in each of our submissions. In Section 4.3, we also run
9Manual inspection revealed that audio from both datasets presents some degree of target language borrowing (e.g., Spanish words present in the Quechua speech, French words present in the Tamasheq speech).
10We are aware the dataset reference is Que-Spa. We chose to use the ISO 639-1 two letters abbreviation for Spanish for consistency with the other datasets used in this work.
experiments in the setting of the IWSLT 2021 Multilingual Task to measure how good our approach is on high-resource languages. The datasets used for this setting are presented in Appendix Table 10.

heading: 4 Experiments and Results
text: All our submissions to the low-resource ST task are in the unconstrained setting, due to the use of pre-trained models, and from training on data in other languages. The datasets used in each submission are listed in Appendix Table 15. This section is organized as follows. We present our Taq-Fr results (4.1) with a detailed ablation study justifying our architectural choices. We then present our QueEs results (4.2). Lastly, we evaluate and analyze our approach in a high-resource setting (4.3).

heading: 4.1 Tamasheq-French Results
text: We submit two systems that have Taq-Fr as the only low-resource language pair (primary and contrastive 1). Additionally, we take our primary submission for Que-Es, which has also been trained on Taq-Fr, and submit this as contrastive 2. The top portion of Table 4 gives the test BLEU scores, and the top portion of Appendix Table 11 presents the valid BLEU scores. Table 12 shows statistics (average and standard deviation) over multiple runs when applicable.
System description. The contrastive 1 model uses as a speech feature extractor the Niger-Mali wav2vec 2.0 model (8th layer). It was initialized with NLLB 1.3B, whose bottom 3 encoder layers were finetuned. We took three runs of this setting with different random seeds and picked the best performing one on the validation set (in terms of
Taq-Fr BLEU) as our contrastive submission. We then ensembled the three runs as our primary submission. Finally, constrastive 2 is the ensemble model used as primary submission to the Que-Es task, which covers both low-resource languages, and combines XSL-R Large with NLLB 3.3B.
Results. Our primary submission significantly outperforms the previous state of the art of 13.2 BLEU (+7.5 BLEU) on the IWSLT 2022 test set by Khurana et al. (2022).11 It also ranks first in this year’s edition, with +7.7 BLEU over the second best primary submission. Our contrastive submissions rank second and third (beating the second best primary submission by +5.4 and +2.8 BLEU).

heading: 4.1.1 Ablation Study
text: In Appendix Table 18 we compare our contrastive 1 model (the non-ensembled version of our primary submission) with other architectures trained on the same data to validate our choice of hyperparameters.
Speech features. The wav2vec 2.0 models trained with Tamasheq (Niger-Mali and Tamasheq) largely outperform the well-known massively multilingual models (XLSR-53 and XLS-R) on Taq-Fr (e.g. +2.5 BLEU Tamasheq compared to XLS-R L). These models are larger and trained on considerably more data, but do not include any Tamasheq speech. Similar to previous works (Pasad et al., 2021; Boito et al., 2022b), when extracting features from wav2vec 2.0 we find that the 8th layer gives better results than the 11th (penultimate) layer (+2.5 BLEU for Niger-Mali).
For HuBERT, on the contrary, features from the 11th layer give the best results (+0.2 BLEU compared to 8th layer). When using the right layer, we find that wav2vec 2.0 outperforms HuBERT (+2.7 BLEU Niger-Mali compared to mHuBERT-Taq).
Finally, Niger-Mali is as good on Taq-Fr as the Tamasheq wav2vec 2.0, but performs considerably better on Fr-En (+4.1 BLEU), probably because it was trained with French audio. The best Fr-En performance is achieved with XLS-R L. We find worse performance on Fr-En with XLS-R XL (-2.0 BLEU), but this may be due to layer selection.
Pre-trained MT model. The larger the model used for initialization, the better the perfor-
11Here we are referencing the model pre-trained using the Niger-Mali dataset that was presented at JSALT 2022: https://www.clsp.jhu.edu/ jsalt-2022-closing-presentations/
mance (even more so for Fr-En). However, we find that the gain from using NLLB 3.3B over NLLB 1.3B is too small to justify the increase in model size and decoding latency (3 times slower). At the same model size, NLLB 600M performs considerably better than mBART (+1.7 BLEU on Taq-Fr, +3.6 BLEU on Fr-En).
Trained parameters. Fine-tuning too many encoder layers results in overfitting, which hurts Taq-Fr and Fr-En performance. On the other hand, fine-tuning just 1 or 2 layers instead of 3 does not result in a large BLEU drop. Similarly, adapter modules are not always needed. Disabling decoder adapters does not degrade TaqFr performance (+0.2 BLEU), but results in a slight drop in Fr-En performance (-0.9 BLEU), which could be attributed to a domain adaptation effect (to the mTEDx domain). Disabling encoder adapters has more impact on performance for Taq-Fr (-0.8 BLEU), with similar effect on performance for Fr-En (-1.0 BLEU). Section 4.3 shows that these adapters are important for domain adaptation.
Convolutions. The number of convolutional layers does not impact performance much (range of 1.1 BLEU on Taq-Fr and 3.2 BLEU on Fr-En for 0 to 3 layers), but it can have a large impact on decoding speed: each layer divides the input length by a factor of 2 resulting in a roughly 3.5× speedup from 0 to 3 layers. Interestingly, even though it was trained on much shorter sequences, the MT model seems to adapt quite well to any input length, even without any convolutions – we achieve a better Taq-Fr result without any convolutions, but a worse Fr-En result.12 However, models with fewer convolutional layers seem to converge faster (as shown in Appendix Figure 2).
Stacked layers. While our approach described in Section 2 fine-tunes some parameters of the pretrained MT model, we can instead plug new Transformer layers at the bottom of the encoder, without changing any existing parameter. These “stacked layers” result in slightly larger models but are conceptually simpler, as they try to map the speech features into the same representation space as the input text embeddings of the MT model. Appendix Table 17 compares this architecture with the one used in our submission to the Taq-Fr task. We see
12Without any convolution, the speech feature to target token ratio is 12:1.
that it performs similarly well (sometimes better) and that it does not add any noticeable decoding latency. We can even reach the same Taq-Fr performance as our contrastive submission by just adding a single Transformer layer plus one convolution layer and small adapters (28M trained parameters in total). Finally, disabling all adapters only results in a small BLEU drop, suggesting that it is indeed possible to map the speech features into the text input space, with only one Transformer layer. This is surprising, considering that the input to this layer is 6 times as long as the target sequence on average.

heading: 4.2 Quechua-Spanish Results
text: The test and validation scores of our submissions to the Que-Es task are reported in the second half of Table 4 and 11, respectively. Because these models are also trained on Taq-Fr data, we additionally report their performance on that task.
System description. As we do not have a speech feature extractor specialized to Quechua speech, our contrastive 1 submission uses a massively multilingual wav2vec 2.0 model: XLS-R Large (18th layer). Compared to our Tamasheq submission, it is also initialized with a larger MT model (NLLB 3.3B), which we found to perform better in this setting. The training settings are the same as for the Tamasheq models, except that we only fine-tune the bottom 2 encoder layers (instead of 3) and validate every 2 500 updates, since this larger model tends to converge faster. Another difference is that we train on both Tamasheq and Quechua data (in addition to the mTEDx and TEDLIUM data). Like in our Tamasheq submission, we train 3 models with different random seeds and ensemble them as our primary submission. Our constrastive 2 submission uses a single model with the same training settings, but starts from a smaller pre-trained MT model (NLLB 1.3B).
Results. Our primary submission in the Que-Es task also ranked first, with 17.7 BLEU on the official test set. The full ranking results were not communicated in time to this camera-ready. They will be made available later through the conference findings paper (Agarwal et al., 2023).
Data contamination. We found shortly after our submission that all the audio files used in the official test and validation sets are also present in the ASR training data shared by the organizers for the unconstrained setting. This means that our
Que-Es ST models are evaluated in an unrealistic setting, where they are tasked to translate Quechua utterances of which they already know the transcription into Quechua. For this reason, we filtered the ASR data to remove all audio files also present in the validation and test sets for Que-Es, and we re-trained models on this filtered data.13 While our official submission results presented in Table 4 use the “contaminated” dataset for comparison with the other submissions, we think any future comparison to our work should be done with the updated results in Appendix Table 11. Note that similar care should be taken with the results of other participants.

heading: 4.3 Results and Analysis in a High-Resource Setting
text: The results of our ablation studies (Section 4.1.1) seem to indicate that our models are reasonably good on Fr-En translation, even though we do early stopping and tune our hyper-parameters based on Taq-Fr performance. Here, we further investigate the performance of our approach on highresource ST by training models in the setting of the IWSLT 2021 Multilingual Task (Anastasopoulos et al., 2021). This task evaluates the performance of multilingual ST models in 4 training directions, for which in-domain training data is provided, and 3 zero-shot directions, for which no training data is provided.
We use XLS-R Large as the speech feature extractor, experiment with both NLLB 1.3B and NLLB 3.3B as the MT model, and perform early stopping based on the average validation BLEU across the 4 official training directions. We train our models on all the mTEDx language pairs that are not zero-shot, along with TED-LIUM (English ASR) and the Tamasheq and Quechua data (see Table 15). Note that the use of pre-trained models and English ASR means our models fall into the unconstrained setting.
Table 5 presents our results on this task, compared with the best unconstrained submission (FAIR; Tang et al., 2021).14 We find that both our models outperform FAIR’s ensemble submission in the training directions, even though they require substantially less compute and data to train, and they are not ensembled. In the zero-shot direc-
13In the updated version, we use NLLB 1.3B by default instead of NLLB 3.3B, like for Taq-Fr. Appendix Table 11 presents uncontaminated results.
14SacreBLEU signature (Post, 2018): nrefs:1| case:mixed|eff:no|tok:13a|smooth:exp|version:2.1.0
tions, our NLLB 1.3B version performs worse than FAIR’s ensemble, which is not surprising since they used training data for the zero-shot language directions (from other datasets), whilst we do not.15 We find that using the larger NLLB 3.3B model for initialization considerably improves our zero-shot results.

heading: 4.3.1 Incremental Learning
text: A limitation of our approach for low-resource ST is that we need to know in advance (when training the multilingual ST model) the set of low-resource languages to cover. Here, we show that it is possible to add a new low-resource language into an existing model without re-training it, similar to what has been previously done by Berard (2021) for text-to-text MT. We train a model following the IWSLT 2021 setting presented above, but without any Tamasheq or Quechua data. Then, we attempt to adapt it to Taq-Fr using four different approaches: 1) adding adapters of dimension 64 in the bottom layers and training all adapters (including in the decoder layers and top encoder layers); 2) adding adapters of dimension 256 in the bottom layers and fine-tuning all adapters; 3) adding adapters
15NLLB has been pretrained on these language pairs for MT, but we do not train on ST data for them.
of dimension 256 in the bottom layers and training only those; 4) adding adapters of dimension 256 in the bottom layers and training both those and the convolutional layer.
We keep the same training settings as before, except that: we train on Taq-Fr data only; we train only the parameters mentioned above; we validate more often (every 1 000 updates); and we disable checkpoint averaging. Table 6 shows the performance of these four incremental training methods, compared to training on the entire language set from scratch. Even though incremental training does not perform quite as well, it appears to be a viable option that can achieve decent results. Lastly, we highlight that our experiments were limited to these four incremental learning settings (without hyper-parameter search), and that better results may be obtained with other parameter-efficient adaptation methods, or with more regularization.

heading: 4.3.2 Multimodality and Domain Transfer
text: Since our systems are initialized with an MT model, of which just a few encoder layers are modified, it is straightforward to use our ST models for text-totext translation: we just need to store both the MT and ST bottom layers and route tokens through the MT ones (see Figure 1). However, one question that remains is whether the ST adapters can be used for text-to-text decoding.
As an investigation of this, Appendix Table 19 measures the MT performance (NLLB 1.3B) on the IWSLT 2021 test sets (same domain as the mTEDx training data) with and without the ST adapters. Surprisingly, we see that not only can we use these adapters for both text and speech modalities, but they actually improve the MT scores (+2.7 BLEU on average), even though they were only trained with ST and ASR data. This suggests that the fine-tuned bottom layers are able to fully map the speech representations into the text represen-
tation space and that the adapters further improve performance by allowing domain adaptation of the MT model (which is hard to do at the very bottom layers). Note that the encoder adapters seem to be the most important ones, which is consistent with the findings of Cooper Stickland et al. (2021) that adapting the encoder is the most effective strategy for domain adaptation. Lastly, we highlight that adapting the MT model directly with MT data (mTEDx’s transcriptions and translations) gives even better results (+4.6 BLEU on average), but this cross-modality domain transfer is an interesting by-product of our parameter-efficient approach.

heading: 5 Zero-Shot Capabilities
text: Throughout this paper we have argued that one advantage of the multilingual models we propose is their potential for zero-shot translation, a setting in which a system produces translation in an unseen language pair by leveraging its existing knowledge of both languages. In Section 4.3 we showed that our models are competitive with the best submission to IWSLT 2021 on the three zero-shot highresource language pairs, despite the fact that these pairs were not truly zero-shot for that system. In this section, we further illustrate the zero-shot capabilities of our models by translating Tamasheq speech in two settings: 1) target language seen during both MT pre-training and ST adaptation (English); 2) target language only seen during MT pre-training (Korean).
Evaluation settings. To score BLEU and chrF16 in the chosen target languages, we use a commercial translation service to translate the French side of the IWSLT 2022 test set to English and Korean.
16SacreBLEU signature: nrefs:1|case:mixed| eff:no|tok:X|smooth:exp|version:2.3.1, (En: X=13a, Ko: X=ko-mecab-0.996/ko-0.9.2-KO). chrF signature: nrefs:1|case:mixed| eff:yes|nc:6|nw:0|space:no|version:2.3.1
Note that this is only a silver-standard made of synthetic data, and thus the evaluation will inevitably be biased.17 Our goal is solely to assess whether our systems have some zero-shot ST abilities. We evaluate our Taq-Fr contrastive 1 system, and variants of this system with fewer or larger adapters. We compare with a cascade baseline, in which we first perform Taq-Fr ST, followed by Fr-En or FrKo MT using the text-to-text path from Figure 1. In this setting, the adapters are disabled during MT.
Results. In Table 7, we measure the zero-shot translation capabilities of our approach on this silver-standard test set. We evaluate four models: our contrastive 1 submission presented in Section 4.1, and variants of this model with increased adapter size, adapters only in the encoder, or no adapters. We compare against a cascade baseline that is not zero-shot, which consists in translating the Tamasheq speech into French text and then translating this text into English or Korean.
We observe that, in the case of English, which was seen during ST adaptation, adapters can be helpful (+2 BLEU over the cascade baseline). On the other hand, for Korean, unseen during ST adaptation, systems with adapters in the decoder (first two rows) perform worse, as they likely bring some degree of language confusion. Results are even worse with larger adapters, with over 40% of output sentences being in the wrong language. In this setting, the best results are achieved with only encoder adapters or no adapters at all (-1 BLEU compared to the baseline).
Appendix Table 13 measures the percentage of output sentences in the correct language and the percentage of Hangul versus Latin character in each system’s outputs. We find that models with
17For instance, we observe that these generated translations contain both the Korean transliteration in Hangul of named entities and the original version in the Latin script. This will likely penalize our produced translation during scoring.
adapters in the decoder (first two rows) generate more Latin characters. Note that the ideal translation is not necessarily 100% Hangul, as it might sometimes be best to keep the foreign named entities in the Latin alphabet. Table 8 illustrates this with a few examples of translations from our contrastive 1 system.

heading: 6 Conclusion
text: In this paper we presented our parameter-efficient multilingual systems as submissions to the IWSLT 2023 Low-Resource Task in the TamasheqFrench and Quechua-Spanish language pairs. The architecture we propose has several advantages: it is computationally and data efficient, it allows the same model to do both speech-to-text and textto-text translation (or transcription), it maximizes knowledge transfer to improve low-resource performance, and it has good zero-shot translation capabilities. Our submissions reach a new state of the art performance, winning both speech translation challenges, especially for Tamasheq-French, where we outperform the previous state of the art by more than 7 BLEU points.
Future work will include a comprehensive evaluation of the ASR capabilities of our architecture, and the investigation of adapters inside the speech representation model. Moreover, when the speech representation model is frozen, a more in-depth analysis of the optimal layer is needed.

heading: Acknowledgements
text: This work was partially funded by the European Horizon 2022 project UTTER (Unified Transcription and Translation for Extended Reality), under grant agreement No 101070631.

heading: A Appendix
text: A.1 Hyperparameters
A.2 Additional Results

heading: Future Work
text: Future work will include a comprehensive evaluation of the ASR capabilities of our architecture, and the investigation of adapters inside the speech representation model. Moreover, when the speech representation model is frozen, a more in-depth analysis of the optimal layer is needed