Pushing the Limits of Zero-shot End-to-End Speech Translation (2024)

Abstract: Data scarcity and the modality gap between the speech and text modalities are two major obstacles of end-to-end Speech Translation (ST) systems, thus hindering their performance. Prior work has attempted to mitigate these challenges by leveraging external MT data and optimizing distance metrics that bring closer the speech-text representations. However, achieving competitive results typically requires some ST data. For this reason, we introduce ZEROSWOT, a method for zero-shot ST that bridges the modality gap without any paired ST data. Leveraging a novel CTC compression and Optimal Transport, we train a speech encoder using only ASR data, to align with the representation space of a massively multilingual MT model. The speech encoder seamlessly integrates with the MT model at inference, enabling direct translation from speech to text, across all languages supported by the MT model. Our experiments show that we can effectively close the modality gap without ST data, while our results on MUST-C and COVOST demonstrate our method’s superiority over not only previous zero-shot models, but also supervised ones, achieving state-of-the-art results.1

heading: 1 Introduction
text: Traditionally, the standard approach for Speech translation (ST) has been the cascade model (Ney, 1999; Mathias and Byrne, 2006), where Automatic Speech Recognition (ASR) and Machine Translation (MT) systems were chained together to produce the desired output. However, in recent years, there has been a paradigm shift towards end-to-end models, which aim to directly map the input speech to the target text without the intermediate transcription step (Bérard et al., 2016). Such models offer several advantages, including reduced error propagation, more compact architectures, and faster inference times (Sperber and Paulik, 2020).
1https://github.com/mt-upc/ZeroSwot
Despite their advantages, end-to-end models require parallel ST data, which are often limited. To mitigate this data scarcity, several strategies aim to leverage the more abundant ASR and MT data (Bérard et al., 2018; Anastasopoulos and Chiang, 2018). Then, another challenge that arises is the modality gap, which emerges from the inherent differences between the speech and text modalities, manifesting in length mismatch and representation mismatch. Convolutional Length Adapters (Li et al., 2021) or Connectionist Temporal Classification (CTC) (Graves et al., 2006; Liu et al., 2020) can alleviate the length mismatch to some extent. To ease the representation gap, previous works have proposed a shared encoder (Ye et al., 2021) and optimizing distance metrics to bring the speech-text representation closer to the shared semantic space (Fang et al., 2022; Ye et al., 2022). Due to its ability to handle representations of different lengths, the Wasserstein distance (Frogner et al., 2015) using Optimal Transport, has emerged as an effective distance metric, specifically during pretraining (Le et al., 2023; Tsiamas et al., 2023b). Although these methods show promise in reducing the modality gap and increasing translation quality, they still require at least some ST data.
14245
To this end, we propose Zero-shot ST with Subword compression and Optimal Transport (ZEROSWOT), a novel approach that aims to bridge the modality gap in a zero-shot fashion, only requiring ASR and MT data. Using Optimal Transport, we train a speech encoder based on WAV2VEC 2.0 (Baevski et al., 2020) to produce representations akin to the embedding space of a massively multilingual MT model, thus enabling zero-shot ST inference across all its supported target languages (Fig. 1). To map the two embedding spaces, our method learns the tokenization of the MT model through CTC, and compresses the speech representations to the corresponding subwords, thus closing the length mismatch. Experiments on MUST-C (Cattoni et al., 2021) reveal that our method is not only better than existing zero-shot models, by a large margin, but also surpasses supervised ones, achieving state-of-the-art results. On CoVoST (Wang et al., 2020b), ZEROSWOT outperforms the original version of the multimodal SEAMLESSM4T (Seamless-Communication, 2023a), while evaluations on the 88 target languages of FLEURS (Conneau et al., 2023) showcase the massively multilingual capacity of our method. ZEROSWOT is also vastly superior to comparable CTC-based cascade ST models, and while it is on par with cascades that utilize strong attention-based encoder-decoder ASR models, it ranks better in terms of efficiency. Finally, we provide evidence of our method’s ability to close the modality gap, both in terms of length and representation.

heading: 2 Relevant Research

heading: 2.1 End-to-end Speech Translation
text: Data Scarcity. To alleviate data scarcity, end-toend ST methods usually utilize the more abundant data of ASR and MT. Such methods include pretraining (Bérard et al., 2018; Bansal et al., 2019; Wang et al., 2020a; Alinejad and Sarkar, 2020), multi-task learning (Anastasopoulos and Chiang, 2018; Indurthi et al., 2021), data augmentation (Jia et al., 2019; Pino et al., 2019; Lam et al., 2022; Tsiamas et al., 2023a), and knowledge distillation (Liu et al., 2019; Gaido et al., 2020). Furthermore, many methods indirectly leverage external data through large foundation models (Bommasani et al., 2022). Initializing parts of the ST model with WAV2VEC 2.0 (Baevski et al., 2020) and MBART50 (Tang et al., 2020) is a quite common practice (Li et al., 2021; Ye et al., 2021; Han et al., 2021; Tsiamas
et al., 2022), while more recently Gow-Smith et al. (2023) utilized a massively multilingual MT model (NLLB et al., 2022) and Zhang et al. (2023) a large language model (LLaMa2) (Touvron et al., 2023). Similarly, here employ WAV2VEC 2.0 and NLLB (NLLB et al., 2022) to ease the data scarcity issue. Modality Gap. Implicit alignment techniques to bridge the representation gap involve a shared semantic encoder in a multitasking framework (Liu et al., 2020; Ye et al., 2021; Tang et al., 2021b), which can be improved by mixup (Fang et al., 2022; Zhou et al., 2023). In addition, explicit alignment methods optimize a distance metric to bring the speech-text representations closer in the shared semantic space. Such distance metrics include the euclidean (Liu et al., 2020; Tang et al., 2021a), cosine (Chuang et al., 2020), and contrastive (Ye et al., 2022; Ouyang et al., 2023), but they typically require some transformation in the representations, such as mean-pooling, while our approach optimizes a distance that does not alter the representation space. Methods to reduce the length discrepancy usually include sub-sampling the speech representation using convolutional length adaptors (Li et al., 2021; Gállego et al., 2021; Fang et al., 2022; Zhao et al., 2022) or character/phonemebased CTC compression (Liu et al., 2020; Gaido et al., 2021; Xu et al., 2021a). Several methods have also used phonemized text in order to better match the representations in both length and content (Tang et al., 2021a, 2022; Le et al., 2023), but potentially limiting the quality of the text branch due to noise. In this work, we introduce a novel CTC-based compression to subword units, directly aligning with the tokenization of MT models.

heading: 2.2 Optimal Transport
text: Optimal Transport (OT) (Peyré and Cuturi, 2019), traditionally used in NLP and MT (Chen et al., 2019; Alqahtani et al., 2021), has recently also been applied to ST. Zhou et al. (2023) used OT to find the alignment between speech and text features to apply Mixup. Le et al. (2023) proposed to use OT in a siamese pretraining setting in combination with CTC, yielding improvements compared to the standard ASR pretraining, while also proposing a positional regularization in order to make OT applicable to sequences. Tsiamas et al. (2023b) extended this pretraining in the context of foundation models, while also freezing the text branch and using CTC compression in order to achieve
better adaptation with the text decoder during ST finetuning. Inspired by these works, we also follow the siamese paradigm, but as a means of directly integrating the speech encoder to the text space of an MT model, thus not requiring any ST data.

heading: 2.3 Zero-shot Speech Translation
text: Escolano et al. (2021b) were the first to achieve zero-shot ST, by incorporating a speech encoder to the representation space of an MT model using language-specific encoders-decoders (Escolano et al., 2021a). Dinh et al. (2022) combined multitask learning on ASR and MT, introducing an auxiliary loss to minimize the euclidean distance between mean-pooled speech and text representations. T-modules (Duquenne et al., 2022) harnessed mined data to learn a fixed-size multilingual and multimodal representation space, subsequently enabling zero-shot ST encoding and decoding from that space. DCMA (Wang et al., 2022) projects both speech and text into a fixed-length joint space using an attention-driven shared memory module, after which it enforces the speech distribution of a codebook to closely mirror its textual counterpart. Gao et al. (2023b) proposed a strategy that employs a multimodal variant of cross-lingual consistency regularization (Gao et al., 2023a), training a model with WAV2VEC 2.0 and a shared encoder on both MT and ASR data. Recently, Yang et al. (2023) introduced a zero-shot ST training method by adapting a new speech encoder to bilingual MT models using OT and a CTC module that shares the vocabulary of the MT model, attaining improvements over zero-shot methods and CTC-based cascade systems. In contrast, our approach focuses on achieving multilingual zero-shot ST, by adapting a large CTC-based speech encoder (Baevski et al., 2020) to a massively multilingual MT model (NLLB et al., 2022), aiming for high-quality results comparable to those of state-of-the-art cascade and end-to-end models. To address the unique challenges associated with adapting to the large vocabulary size of multilingual models (e.g., 250k tokens for (NLLB et al., 2022)), we propose a novel compression method that effectively learns the subword tokenization of the MT model, rather than predicting actual tokens. This decoupling from the vocabulary size ensures efficiency and scalability, overcoming limitations that may arise in previous methods (Yang et al., 2023), due to the need of sharing the MT vocabulary with the CTC module.

heading: 3 Methodology

heading: 3.1 Problem Definition & Proposed Solution
text: End-to-end Speech Translation. A speech translation corpus consists of speech-transcriptiontranslation triplets, D = {(s,x,y)}, where s is the speech signal, x is the transcription, and y the translation. End-to-end ST systems aim to learn a model MST that generates the translation y directly from the speech s, using the parallel ST data DST = {(s,y)}. The transcription x is optionally utilized, as in pretraining or multi-task learning. Zero-shot Speech Translation. In a zero-shot ST setting, we aim to learn a model MZS-ST that generates the translation y with access to corpora DMT = {(x,y)} and DASR = {(s,x)}, but without directly utilizing the parallel corpus DST. Proposed Solution. Assuming access to an MT model MMT obtained via DMT, we utilize DASR to learn a speech encoder S-Enc that produces representations akin to those expected by MMT. On inference time, we achieve zero-shot ST by replacing the embedding layer of MMT with S-Enc.

heading: 3.2 Model Architecture
text: Our model architecture includes a speech and a text branch. The text branch consists of a text encoder, which remains frozen during training. The speech branch consists of an acoustic encoder, a CTC module, a compression adapter, a speech embedder and a semantic encoder, of which the parameters are shared with the frozen text encoder (Figure 2a).

heading: 3.2.1 Text Encoder
text: The text encoder is a Transformer, of which the parameters are initialized from a massively multilingual MT model (NLLB et al., 2022), and kept frozen during training. The tokenized2 source text x with length m is passed through an embedding layer Emb : |B| → d to obtain ex ∈ Rm×d, where B is a multilingual subword-level vocabulary. Sinusoidal positional encodings Pos(·) are added after an up-scaling (Vaswani et al., 2017), and a Transformer encoder T-Enc with L layers is used to obtain a semantic representation hxL ∈ Rm×d.
ex = √ d ·Emb(x) +Pos(m) (1)
hxL = T-Enc(e x) (2)

heading: 3.2.2 Acoustic Encoder
text: The acoustic encoder A-Enc takes as input the speech signal s ∈ Rℓ and produces an acoustic
2With prepended <lang> and appended 
 tokens.
representation a ∈ Rn×d. It consists of a series of strided convolutional layers that downsample the signal by r (thus, n= ℓ/r), followed by a Transformer encoder with N layers. It is initialized from WAV2VEC 2.0 and is finetuned during training.
a = A-Enc(s) (3)

heading: 3.2.3 Connectionist Temporal Classification
text: We use a CTC loss (Graves et al., 2006) to maintain a meaningful acoustic representation and provide the necessary information to the compression adapter (§3.2.4). A CTC module, which is a linear softmax layer produces probabilities p ∈ Rn×|V|, where V is a letter-based vocabulary.3
p = softmax(a ·Wctc + bctc) (4)
Where Wctc ∈ Rd×|V| and bctc ∈ R|V| are trainable parameters, initialized from WAV2VEC 2.0.
Given the CTC labels z, and their conditional probability P (z|p), which is obtained by summing over the probability of all possible alignment paths between the acoustic representation a and z, the CTC loss is defined as:
3Incl. blank <blank>, unknown <unk>, separator <sep>.
The labels z are usually obtained from the transcription x via splitting on words and removing characters absent from V , such as punctuation. But this implies a word-level tokenization and text normalization, which is inconsistent with the text branch (§3.2.1). Thus, in order to minimize the discrepancies between the two branches, we propose to split on subwords using the text branch tokenizer, and keep the positions of characters not in V , by replacing them with the unknown token (Table 1). This is particularly important for the effectiveness of our compression to subwords (§3.2.4).

heading: 3.2.4 Compression Adapter
text: The compression adapter (Fig. 2b) takes as input the acoustic representation a and the CTC probabilities p, to produce a compressed acoustic representation asw ∈ Rnsw×d (where nsw ≤ n) that resembles in length and content the text embedding ex (Eq. 1). It involves two stages: compression to characters (Liu et al., 2020), followed by a novel
compression to subwords. Character Compression. Each vector ai ∈ Rd of the acoustic representation is labeled according to the corresponding greedy CTC prediction πi=argmax(pi). Then, consecutive same-labeled representations are grouped together, and combined via mean-pooling, while removing those labeled as blanks, thus having a character-like representation achar ∈ Rnchar×d, where nchar ≤ n. Similarly, p is compressed to pchar ∈ Rnchar×|V|.
π = argmax(p) (6)
pchar = char-compress(p|π) (7) achar = char-compress(a|π) (8)
Subword Compression. The character representation achar is split into chunks c(1), . . . , c(n sw) according to the positions labeled as separators, where nsw is the number of predicted separators, c(i) ∈ Rki×d is the i−th chunk representation (with ki number of characters), and nchar = ∑nsw i=0 ki. Since the CTC is predicting the tokenization of the text branch through the proposed target labels, each chunk is combining information that resemble subword tokens from V . Each chunk of character representations is processed independently with Subword-Enc : Rki×d → Rd, which is Transformer encoder that prepends a trainable <cls> token to each chunk, which is then used a subword representation of the whole chunk (Devlin et al., 2019). Then, by concatenation we obtain a subword-like compressed representation asw ∈ Rnsw×d, where nsw≤nchar≤n (Fig. 3).
c(1), . . . , c(n sw) = split(achar|πchar) (9) aswi = Subword-Enc ( c(i) ) (10)

heading: 3.2.5 Speech Embedder
text: We concatenate to the compressed acoustic representation two vectors ϵ<lang>, ϵ
∈Rd that function as the source language and end of sentence embeddings. Similar to the text branch, we scale the representation by √ d and add sinusoidal positional encodings with Pos(·) (Eq. 1), thus obtaining a speech embedding es ∈ Rn′×d, where n′=nsw+2. The special embeddings are initialized from the text embedding (Eq. 1), and remain frozen.

heading: 3.2.6 Semantic Encoder
text: The speech embedding es is passed through a Transformer encoder T-Enc, which is frozen and shared
with the text branch (Eq. 2), to obtain a semantic speech representation hsL ∈ Rn ′×d.
es = √ d[ϵ<lang>,asw, ϵ
] +Pos(n′) (11) hsL = T-Enc(e s) (12)

heading: 3.3 Optimal Transport
text: To align a speech representation hs ∈ Rn′×d to the text representation hx ∈ Rm×d, we are minimizing their Wasserstein loss (Frogner et al., 2015) using Optimal Transport (OT) (Peyré and Cuturi, 2019), as in Le et al. (2023); Tsiamas et al. (2023b).
We assume two uniform probability distributions ϕs,ϕx, with ϕsi = 1/n
′ and ϕxj = 1/m, that define the mass of each position in the speech and text representations. Using a squared euclidean cost function we obtain a pairwise cost matrix C ∈ Rn′×m+ , where Cij = ||hsi − hxj ||22. Then, the Wasserstein distance Wδ ∈ R+ is defined as the minimum transportation cost over all possible transportation plans Z ∈ Rn′×m+ of the total masses ϕs,ϕx. The optimization is thus subjected to constraints ∑m j=1 Z:,j=1/n ′ and ∑n′ i=1 Zi,:=1/m.
Wδ = min Z
n′∑
i=1
m∑
j=1
ZijCij (13)
In practice, to make the Wasserstein distance differentiable and efficient to compute using the Sinkhorn algorithm (Knopp and Sinkhorn, 1967), we use its entropy-regularized upper-bound estimation. Furthermore, since the Wasserstein distance is permutation invariant, we follow Le et al. (2023) and incorporate two positional regularization vectors vs ∈ Rn′ and vx ∈ Rm in hs and hx accordingly, thus penalizing the transportation of mass between two distant positions i and j.
h̄ s = [hs;µvs] , where vsi = i− 1 n′ − 1 (14)
h̄ x = [hx;µvx] , where vxj = j − 1 m− 1 (15)
And thus for the positional regularized cost matrix C̄, the upper-bound Wasserstein distance that we use as a loss function is defined as:
LWASS = min Z
( n′∑
i=1
m∑
j=1
ZijC̄ij − λH(Z) ) (16)
Where λ > 0 is a hyperparameter, and H(·) denotes the von Neumann entropy of a matrix.

heading: 3.4 Final Loss
text: We apply a Wasserstein loss (Eq. 16), not only at the final layer but also at intermediate ones, since they also carry important information, and can thus help producing more robust speech representations. Thus, the final loss is defined as:
L = ∑
l∈I
α
|I|L (l) WASS + (1− α)LCTC (17)
Where I is the set of shared encoder layers for which we apply Wasserstein losses, and α > 0 is a hyperparameter to balance the relative importance between the Wasserstein and CTC losses.

heading: 3.5 Zero-shot Speech Translation Inference
text: To enable zero-shot ST inference, we simply replace the MT embedding layer with the trained speech encoder. (ZEROSWOT, Fig. 2c).

heading: 4 Experimental Setup
text: Data. For training we are using Common Voice (Ardila et al., 2020), the speech-transcription pairs of MUST-C V1.0 (Cattoni et al., 2021), and LibriSpeech (Panayotov et al., 2015). We evaluate our models on three multilingual ST benchmarks: MUST-C V1.0 (En → 8 lang.), COVOST 2 (Wang et al., 2020b) (En → 15 lang.), and FLEURS (Conneau et al., 2023) (En → 88 lang.) (Appendix A). Model Architecture. The acoustic encoder follows WAV2VEC 2.0 LARGE (Baevski et al., 2020) with N = 24 layers and dimensionality d= 1024 (315M parameters). The CTC module uses a vocabulary with size |V| = 30, and together with the acoustic encoder are initialized from a CTCfinetuned version of WAV2VEC 2.0 on LibriSpeech (Panayotov et al., 2015). The subword encoder has 3 layers with dimensionality of 1024 (35M). The text encoder has either L=12 or L=24, both with dimensionality 1024, and an embedding size of |B|=256k. We initialize its parameters from two dense versions of NLLB (NLLB et al., 2022), the 600M (MEDIUM) or the 1.3B (LARGE). Based on the size of NLLB, the resulting ZEROSWOT models used for inference have 0.95B (MEDIUM) or 1.65B (LARGE) parameters, while both of them the have the same number of training parameters (350M). We also train a SMALL version based on WAV2VEC 2.0 BASE and NLLB-MEDIUM (Appendix C.1). Training details. We use AdamW (Loshchilov and Hutter, 2019) with a learning rate of 3e-4, inverse square root scheduler with warmup, and batch size
of 32M tokens. We apply dropout of 0.1 as well as time and channel masking (Baevski et al., 2020). Speech input is 16kHz waveforms, and text is tokenized with sentencepiece (Kudo and Richardson, 2018), using the NLLB model (Appendix C.2). Loss. We set α = 0.9 and apply Wasserstein losses for I = {6, 7, 8, 9, 10, 11, 12} (MEDIUM) and I= {12, 14, 16, 18, 20, 22, 24} (LARGE) (Eq. 17). Positional and entropy regularization are set to µ=10 (Eq. 14, 15), and λ=1 (Eq. 16). Evaluation. We apply checkpoint averaging and use a beam search of size 5. We evaluate with casesensitive detokenized BLEU (Papineni et al., 2002) using sacreBLEU (Post, 2018) (Appendix B).

heading: 5 Results
text: MUST-C V1.0. In Table 4 we present the results of our proposed ZEROSWOT in MUST-C and compare them with other SOTA zero-shot and supervised methods. We train ZEROSWOT under three scenarios, regarding data usage (Table 2), and in three different sizes depending on the WAV2VEC 2.0 and NLLB versions (Table 3). For scenario C we first multilingually finetuned NLLB on MUSTC En-X MT data (Appendix D), before adapting to it the speech encoder with MUST-C ASR data.
ZEROSWOT Model Size Train/Infer Parameters WAV2VEC 2.0 NLLB
In the upper part of Table 4 we compare our small ZEROSWOT model using only ASR data from MUST-C (scenario B) to previously proposed zeroshot methods. Our results show that our model can surpass all previous methods, even though some of them use both ASR and MT data from MuSTC, and also have more training parameters. Then, comparing our larger versions to supervised methods, ZEROSWOT sets new SOTA in 7/8 directions, with multiple versions of our model outperforming
our proposed (small) model. Lower part: Underscored is the best supervised score, highlighted are zero-shot scores that are better than the best supervised ones, and in bold is best overall. † NLLB parameters are finetuned separately. ∗ Parameters estimated from methodology description. Extended results in Table 14.
the previous best. Notably, ZEROSWOT-LARGE (C) is better in En-De and En-Es than the recently proposed LST (Zhang et al., 2023), which is based on LLaMA2-13B (Touvron et al., 2023), and compared to our method has 40× / 8× more training/inference parameters. COVOST 2. In Table 5 we present the results of ZEROSWOT in COVOST 2, where we used Common Voice for training, and a multilingually finetuned NLLB4 on the MT data of COVOST En-X. For comparison, we provide results from other supervised SOTA methods, which are XLS-R (Babu et al., 2021) and SEAMLESSM4T (SeamlessCommunication, 2023a,b), and group them according to their size based on inference parameters. For the medium-sized models, we observe that although our method is evaluated on zero-shot ST and has less training parameters, it surpasses the previous SOTA SEAMLESSM4T in 12/15 directions and on average by a large margin of 3.5 BLEU points. For the large models, ZEROSWOT ranks better than the original LARGE configuration of SEAMLESSM4T, with an average of 31.1 BLEU,
4Similar to scenario C of Table 2. Details in Appendix D.
outperforming it by 0.6 points. Compared to the updated v2, our method is 0.5 BLEU points behind, but still obtains better results in 7/15 directions. How does ZEROSWOT compare with cascades? To answer this, we train multilingual cascade ST systems based on WAV2VEC 2.0 and NLLB in MUST-C. The cascades are trained in the same data scenarios and data sizes as ZEROSWOT (Tables 2, 3), and have two versions based on the ASR model, which is either a CTC encoder or an attention-based encoder-decoder (AED), in which case we pair WAV2VEC 2.0 with a Transformer decoder (Appendix E). We present BLEU scores by data usage and model size, and also measure the average inference time per example. The results of Table 6 highlight that although ZEROSWOT closely resembles a CTC-based cascade, in that it softly connects a CTC encoder and an MT model, its performance is consistently better, with gains from 2.7 to 7.2 BLEU points5. This can be expected since CTC ASR models are error-prone, leading to
5The improvements are more evident than the zero-shot ST method of Yang et al. (2023), which surpassed CTC-based cascades by 0.8 BLEU, further highlighting the effectiveness of our method complimented by the proposed compression.
compounding errors through the MT model (Bentivogli et al., 2021), where our method solves this by directly mapping the encoder representation to the MT embedding space via OT and our proposed compression. Compared to strong cascades that utilize AED-based ASR models, we observe that ZEROSWOT is noticeably better in the absence of in-domain data (A, +0.8/0.9), while the cascade becomes slightly better when in-domain data become fully available (C, -0.3/0.3). More importantly though, we find that ZEROSWOT excibits better efficiency, being 18% and 5% faster in the MEDIUM and LARGE models accordingly.
Does our compression reduce the length gap? We replace our proposed compression adapter (§3.2.4), with either a length adaptor (Li et al., 2021) that down-samples by a factor of 4, or CTC compression (Gaido et al., 2021), or no compression at all.6 We also consider different types of tokenization than the proposed "Subword+Unk" (Table 1), which affect the placement of the separator tokens in the CTC target labels, and thus which representations are combined by the compression adapter. Apart from translation quality in MUST-C, we measure the length gap between the speech and text representations at the output of the shared encoder, in terms of absolute differences |∆len| = |len(speech) − len(text)|, and relative ratios r(len) = len(speech)/len(text). Our findings in Table 7 show that our proposed method (final row) surpasses previous methods (rows 1-3), by 0.6-0.7 points, and closes the absolute length gap to an average of 1.4 tokens. We also find that without learning the text branch tokenization (rows 4-5), our compression falls behind previous methods. The length ratio of each method reveals that over-compression, i.e. r(len) < 1, is what worsens performance, leading to information loss, as even the model without compression (r(len)≈11) achieves scores higher than our method without proper tokenization with respect to the text branch. Does ZEROSWOT close the modality gap? Apart from the competitive zero-shot performance we provide further evidence by designing a crossmodal retrieval experiment. In Table 8 we present the speech-text retrieval accuracy in the 2551 examples of MUST-C En-De tst-COMMON, and compare against ConST (Ye et al., 2022), which used contrastive learning. We perform two retrieval experiments. For each speech representation we retrieve the text representation with the (a) highest cosine similarity after temporal mean-pooling or (b)
6Details on these experiments can be found in App. C.3.
the lowest Wasserstein distance. Our results show that ZEROSWOT with Wasserstein-based retrieval achieves an improvement of 3.5 points compared to ConST, approaching a perfect accuracy score7, which highlights the degree of alignment between the text and the learned speech representations.
Is ZEROSWOT massively multilingual? We evaluate on FLEURS (Conneau et al., 2023), from English to 88 target languages. In Table 9 we present the BLEU scores of ZEROSWOT trained on Common Voice using the original NLLB versions. Since our model can benefit from ASR data which are in general accessible for English, we additionally train with MUST-C(ASR) and LibriSpeech. We also compare with a strong cascade by training an AED-ASR model based on WAV2VEC 2.0 (similar to Table 6) on Common Voice and coupling it with the original NLLB. Although our models are zero-shot ST systems, we find that they obtain results that are competitive to SEAMLESSM4T, which used 400k hours of audio data (SeamlessCommunication, 2023a). Furthermore ZEROSWOT is comparable to cascades, and additional ASR data can bring some further improvements.
Can ZEROSWOT benefit from ST finetuning? We initialize ST models with the zero-shot models trained on MUST-C (Table 4) with medium/large sizes and B/C data usage scenarios, and perform supervised ST finetuning on MUST-C En-De (Ap-
7Manual inspection of the 32 wrong predictions, showed that most of them are either due to many positives (several "Thank you.") or noisy examples (misaligned speech and text).
pendix C.4). The results of Table 10 hint that ZEROSWOT when trained on both ASR and MT data (scenario C) is likely in a local minima, which is not easily adaptable to supervised ST, having only marginal improvements (+0.2 BLEU). On the contrary, models trained only with ASR MUST-C data (scenario B) are more flexible for adaptation, with gains of 1.9-2.3 BLEU points. Notably, by finetuning ZEROSWOT-LARGE (B) we obtain an even higher new SOTA for MUST-C En-De at 32 BLEU points, improving the previously best published result (Zhang et al., 2023) by 1.6 points (Table 4).
Ablations. In the ablations of Table 11 we observe that the speech embedder (§3.2.5) is critical to the effectiveness of ZEROSWOT, possibly due to reasons regarding over-compression (absence of <lang> and 
). Auxiliary Wasserstein losses (§3.4) have less impact, but come with a negligible computational cost due to batching.

heading: 6 Conclusions
text: We presented ZEROSWOT, a novel method for zeroshot end-to-end Speech Translation, that works by adapting a speech encoder to the representation space of a massively multilingual MT model. Our method sets new SOTA in MuST-C, while it also surpasses larger, supervised models in COVOST. We additionally showed that ZEROSWOT outperforms cascades, both in performance and efficiency. Finally, we provided evidence of closing the modality gap, with respect to both length and representation. Future research will expand on low-resource scenarios and speech-to-speech translation.
Limitations
Our method obtains state-of-the-art results in Speech Translation, despite being a zero-shot model. Still there are a couple of limitations inherent to the model, and on its evaluation. Although ZEROSWOT is an end-to-end model, in the sense that there is not intermediate transcription step and solves error-propagation, it suffers from some of the caveats of cascade systems. Since the speech encoder "mimics" the representation space of an MT model, no acoustic information is retained. This means that the resulting translation model can’t use acoustic information, like prosody, and can thus be sometimes limited in each ability to carry out correct translations. Furthermore, given the requirement of ASR data to train the speech encoder, this methodology can’t be used for spoken-only languages, at least in its current form.
Finally, in our evaluations we did not test for a different language, other than English, in the source. This was partly due to space constrains in the paper. Although we hypothesize that our proposed method can handle equally well other source languages, it would be interesting to be applied for low-resource scenarios, which we leave as future research.

heading: Acknowledgements
text: Work at UPC was supported by the Spanish State Research Agency (AEI) project PID2019107579RB-I00 / AEI / 10.13039/501100011033.

heading: A Data Processing and Filtering
text: A.1 MUST-C V1.0 For the train set of each language pair, we perform text cleaning on both the transcription and translation, including space and punctuation normalization as well as removal of non-utf characters. We also remove speaker names and events such as "(Laughing)". Following, we remove examples that contain less than 4 characters, or have a transcription-to-translation ratio smaller than 0.5 or larger than 2. For the Zero-shot ST training we remove the translations, and process the transcriptions by replacing numbers and symbols with their spelled out forms. Then we concatenate all eight training sets and remove duplicates according to the triplet of (speaker, text, duration). Due to alignment errors during the creation of the dataset, the same text can be paired with different parts of audio for different language pairs. Thus, for duplicates of (speaker, text) we keep the example that is closer to the speaking ratio of speaker8. Finally, we remove examples with extreme words-per-second ratio (larger than 10). A summary of the dataset sizes is available at Table 13. We also similarly construct a validation set, with a total of 2,081 examples, to be used during training, by concatenating the individual dev sets (without filtering), and simple removal of hard duplicates according to the triplet of (speaker, text, duration).
A.2 Common Voice We use version 8.09, and more specifically the train and dev splits. For the train split, we
8Calculated as the average words-per-second of all the examples for this speaker in the concatenated dataset.
9commonvoice.mozilla.org/en/datasets
apply text normalization, as well as replace numbers and symbols with their spelled-out forms.
A.3 LibriSpeech
For training we use the train-clean-100, train-clean-360, and train-other500 splits, and for evaluation the dev-clean one. Since LibriSpeech has normalized transcriptions, we restore the casing and punctuation with a finetuned BERT-base model (Devlin et al., 2019) using rpunct.10
A.4 CTC Labels
For all the datasets, to obtain the target labels for CTC, first we tokenize the transcription with the NLLB sentencepiece model11, then remove casing, replace all spaces with the <sep> token, and substitute all characters not present in the WAV2VEC 2.0 letter-based vocabulary12 with the <unk> token.

heading: B Evaluation
text: The evaluation of all models, either MT, ZEROSWOT, ASR, Cascade ST, or Supervised ST remains the same. We apply checkpoint averaging to the 10 best checkpoints according to the corresponding validation metric, and generate with a beam search of 5. We measure translation quality with BLEU (Papineni et al., 2002) using sacreBLEU (Post, 2018) with signature BLEU|nrefs:1|case:mixed|eff:no|tok:13a| smooth:exp|version:2.3.1. For comparing
10github.com/Felflare/rpunct 11fairseq/nllb/flores200_sacrebleu_tokenizer_spm.model 12fairseq/wav2vec/dict.ltr.txt
with SEAMLESSM4T, we follow their evaluation, and using a char tokenization for Mandarin Chinese, Japanese, Thai, Lao and Burmese. We do not carry statistical significance testing since in almost all results we are comparing average BLEU scores across many language directions. Although BLEU as a metric has its limitations (Mathur et al., 2020), we opted to use it for comparison with previous methods from the literature.

heading: C ZEROSWOT Models
text: C.1 Model Architecture
The acoustic encoder is composed of a feature extractor with 7 strided convolutional layers that sub-sample the signal by r = 320, followed by N=24 transformer layers with d=1024 (Baevski et al., 2020). Each layer has feed-forward dimension of 4096, 16 attention heads, GELU activations (Hendrycks and Gimpel, 2020), and use the prelayernorm configuration (Xiong et al., 2020). The acoustic encoder and the linear layer of the CTC with size |V|=30 are initialized from a large version of WAV2VEC 2.013, with around 315M parameters, which was pretrained with self-supervised learning and self training (Xu et al., 2021b), and finetuned on ASR with Librispeech (Panayotov et al., 2015) and Libri-light (Kahn et al., 2020).
The subword encoder in the compression adapter is a Transformer encoder with 3 layers and dimensionality d=1024, which have the same format as the ones in the acoustic encoder. We also add sinusoidal positional encodings to the chunked input.
The text embedding uses a vocabulary with size |B|=256k, and for the text encoder we are either using the MEDIUM version with L = 12 layers or the LARGE version with L= 24 layers. Each layer in the MEDIUM version has a dimensionality of 1024, feed-forward dimension of 4096, ReLU activation, 16 attention heads, and pre-layernorm configuration. The only difference of the LARGE version apart from the number of layers is the feedforward dimension which is 8192. For initialization we are using two dense versions of NLLB (NLLB et al., 2022), which have been distilled from the 54B parameter sparse model: the 600M version14 and the 1.3B version15. The SMALL version of our method uses the
13fairseq/wav2vec/wav2vec_vox_960h_pl.pt 14fairseq/nllb/nllb-200-distilled-600M.pt 15fairseq/nllb/nllb-200-distilled-1.3B.pt
BASE16 version of WAV2VEC 2.0 (12 layers and dimensionality 768) and NLLB-MEDIUM. The subword encoder has 3 layers, with dimensionality 768. We use a linear layer to project its output to the d = 1024, which is the dimensionality of the text encoder due to NLLB-MEDIUM. The total number of training parameters is 110M, while the inference parameters are approximately 0.7B.
C.2 Training details We finetune the parameters of the acoustic encoder, CTC module and compression adapter, while the parameters of the speech embedder, text embedding and text encoder are kept frozen. We are using AdamW (0.9, 0.98) (Loshchilov and Hutter, 2019) with a base learning rate of 3e-4, an inverse square root scheduler with a linear warmup (2k for MUST-C(ASR), 4k for Common Voice), and a batch size 32M tokens. The input to the speech branch is raw waveforms sampled at 16kHz and the input to the text branch is English text, tokenized with the NLLB sentencepiece model17.
For the Wasserstein loss we are using the geomloss package18 using the Sinkhorn distance (Knopp and Sinkhorn, 1967) with the default parameters. When it for multiple layers, we batch them together for efficient computation. The Wasserstein losses for the intermediate layers are applied after the first layer-norm of the next layer, since we are using a pre-layernorm setting. Applying the losses on representations that have not been layernormed made the training unstable.
We use an activation dropout of 0.1 in the acoustic encoder, a dropout of 0.1 in the CTC module, and a dropout of 0.1 in the compression adapter. We furthermore apply masking to the speech signal (Baevski et al., 2020), with a probability of 0.3 to mask 10 consecutive frames, and a probability of 0.2 to mask 64 consecutive frequency channels.
To speed-up the training, we extract offline the text encoder representations from NLLB, and can thus completely remove the frozen text branch during the training of the speech encoder.
We train until convergence, and apply early stopping if the Wasserstein loss of the last layer in the development set using has not improved for 10 epochs. Models on LibriSpeech or MUST-C(ASR) usually train for around 50k steps, while on Common Voice they train for around 150k.
16fairseq/wav2vec/wav2vec_small_960h_pl.pt 17fairseq/nllb/flores200_sacrebleu_tokenizer_spm.model 18kernel-operations.io/geomloss
To accelerate the training for models in Common Voice, we trained a seed model using the original MEDIUM version of NLLB, and then adapted the resulting speech encoder to different NLLB versions (LARGE and/or finetuned). For adaptation, only required 30k steps (compared to 150k of the seed model), where we used learning of 3e-5, warm-up of 1k steps and cosine annealing.
All models are implemented and trained on FAIRSEQ (Ott et al., 2019; Wang et al., 2020a) with memory-efficient-fp16. Training for 50k steps takes around 36 hours on a machine with 8 NVIDIA 3090s (for either MEDIUM or LARGE).
C.3 Models trained with different type of compression
Here we provide details for the models trained to obtain the results of Table 7. To ensure a fair comparison we keep the same number of parameters for all models, by adding transformer layers or MLPs. Architecture changes are done only in the part of the compression, and everything else remains the same (including Speech Embedder). For the model without compression we add two large MLPs with residual connections at the output of the acoustic encoder. The MLPs consist of up-projection to 8192, GELU activation, and down-projection back to 1024. For the model with the Length Adaptor (Li et al., 2021), we use 2 strided convolutional layers that sub-sample the output of the acoustic encoder by a factor of 4, followed by a transformer layer. For the model using CTC-based compression (Liu et al., 2020; Gaido et al., 2021) we first meanpool consecutive same-labeled representations as in §3.2.4, followed by 3 transformer layers. For the models using the proposed compression adapter but different tokenizations, we just change the CTC target labels, and everything else remains the same.
C.4 Supervised ST finetuning Here we provide details for the ST-finetuned versions of ZEROSWOT from Table 10. We train for 20k steps with a learning rate of 4e-5, linear warmup for 1k steps, fixed for another 4k steps, and reduced with cosine annealing for the rest. For MEDIUM models we finetune the parameters of the compression adapter, text encoder and text decoder, and thus keep frozen the acoustic encoder, speech embedder, and text embedding. For the LARGE models, we only finetune the compression adapter and apply LNA to the text encoder-decoder (Li et al., 2021). We hypothesize that better results
could be obtain with more sophisticated finetuning approaches (Hu et al., 2022). For inference we average the 10 best checkpoints according to BLEU on MUST-C En-De dev.

heading: D Machine Translation Models
text: Model Architecture. For the MT models used in this research we used NLLB-600M (MEDIUM) and NLLB-1.3B (LARGE), which are dense transformers distilled from a 54B parameter Mixture-ofExperts (MoE) model (NLLB et al., 2022). NLLB supports many-to-many translation between 202 languages. The MEDIUM model has 12 layers in both encoder and decoder, model dimensionality of 1024, feed-forward dimension of 4096, 16 attention heads, ReLU activations, and using pre-layernorm (Xiong et al., 2020). The only difference of the LARGE model is that has 24 layers in both encoder and decoder, and a feed-forward dimension of 8192. Both use a multilingual shared vocabulary of 256k tokens learned with sentencepiece (Kudo and Richardson, 2018). Finetuning hyperparameters. In order to adapt them to the domain and languages of MUST-C and COVOST 2, we did multilingual finetuning of NLLB using their transcription-translation pairs. We used batch size of 32k tokens for the MEDIUM, and 64k for LARGE. We used the AdamW optimizer (Loshchilov and Hutter, 2019) with 2k warm-up updates and an inverse square root scheduler. The learning rate and dropout rate were selected from a grid search to {7.5e-5, 1e-4, 2e-4} and {0.1, 0.2, 0.3} accordingly19. We also use a dropout of 0.1 for the attention, 0.0 for the activations and a label smoothing of 0.1 for the cross entropy loss, as was done for training the original versions (NLLB et al., 2022). We evaluate every 500 steps on the dev sets, and early stop if the loss has not improved for 20 consecutive evaluations. At the end of the training we average the best 10 checkpoints according to the dev sets loss. Models were trained with fp16, while to train the LARGE version on an NVIDIA 3090, we used memory-efficient-fp16 (Ott et al., 2019). Models in MUST-C required 40k steps to converge, while the ones in COVOST 2 required around 100k. Results. BLEU scores for both the original and finetuned versions are available at Table 14 for MUST-C and Table 16 for COVOST 2.
19For each NLLB version we run 9 experiment in MUST-C and used the optimal combinations also for COVOST 2.

heading: E Cascade Speech Translation Models
text: Here we provide the training details and hyperparameters used for the Cascade ST systems (Table 6). For the MT models we used either NLLB-MEDIUM or NLLB-LARGE, which were optionally finetuned, as described in Appendix D. For the ASR models we either used CTC or attention-based encoderdecoder (AED) models, as described below. CTC ASR Models. The architecture is the same with the acoustic encoder used in the ZEROSWOT models, followed by a CTC module (Appendix C). We initialize the model with the same version used to initialize the ZEROSWOT models, and finetune it with CTC on either MUST-C(ASR)or Common Voice. For each dataset, we use exactly the same filtering and preprocessing (Appendix A), and as targets for the CTC we use the standard labeling (Table 1, 1st row). We use AdamW (Loshchilov and Hutter, 2019) with a learning rate of 1e-4, warmup of 1k steps and an inverse square root scheduler, and the batch size is set to 32M tokens. We use a dropout of 0.1 for the activations and at the CTC module, and a time masking with probability 0.5 for length of 10, and channel masking with probability of 0.25 for size of 64 (Baevski et al., 2020). We evaluate every 250 steps and stop the training when the validation loss did not improve for 10 consecutive evaluations. We average the 10 best checkpoints according to the validation loss. The size of the model is around 315 million parameters. Results for these models are available at Table 15. AED ASR Models. The architecture of the encoder is the same as with the CTC ASR model. For the decoder we use 6 transformer layers with dimensionality of 768, feed-forward dimension of 3072, 8 attention heads, GeLU activations (Hendrycks and Gimpel, 2020), and pre-layernorm (Xiong et al., 2020). We initialize the encoder with a pretrained WAV2VEC 2.020, train the whole model with cross entropy using LibriSpeech and then finetune it either on MUST-C(ASR) or Common Voice. The target vocabulary has a size of 16k and is learned jointly on the three ASR datasets with sentencepiece. For each dataset, we use exactly the same filtering and preprocessing (Appendix A), and the target text is the same we used for training the ZEROSWOT models. The hyperparameters are the same for both training on LibriSpeech and finetuning on MUST-C(ASR) or Common Voice. We use AdamW (Loshchilov and Hutter, 2019) with learn-
20fairseq/wav2vec/wav2vec_vox_new.pt
ing rate of 2e-4, an inverse square root scheduler with a warmup of 1k steps, and a batch size of 32M tokens. The masking and dropout for the encoder are as in the CTC model, while for the decoder we use a dropout of 0.1, and a label smoothing of 0.1. We evaluate every 500 steps and stop training when the validation loss did not improve for 10 consecutive updates. We average the 10 best checkpoints according to the validation loss. The size of the model is around 390 million parameters. Results for these models are available at Table 15. Cascade ST inference. For generation, we use a beam search of size 5 for either the CTC or AED ASR model, and then pass the ASR generated text to the MT model, which generates the translation with a beam search of size 5.

heading: F Detailed Results in MUST-C V1.0
text: In Table 14 we provide a summary of all obtained results in MUST-C, with our ZEROSWOT models (including ablations), our cascade models, and our finetuned NLLB models. We also provide more results from the literature for both zero-shot and supervised ST models. Note on performance drop of CTC-based Cascade when moving from the original NLLB model (scenario B) to the finetuned one (scenario C). We hypothesize that the original NLLB can better handle the noisy English text produces by the CTC ASR model, due to the abundance of data it was trained on. When finetuning it using the bitext of MUST-C, the model losses some of this capacity.
In Table 15 we provide the results in MUST-C of the ASR models we trained for the CTC-based and AED-based cascades (Tables 6, 14). Models trained on Common Voice correspond to scenario (A), while models trained on MUST-C correspond to scenarios B and C.

heading: G Detailed Results in COVOST 2
text: Here we provide a more extended version of Table 5, including MT results of NLLB (original & finetuned), ZEROSWOT results with the original NLLB, and also add XLS-R-0.3B (Babu et al., 2021).

heading: H Detailed Results in FLEURS
text: In Table 17 we present the per-language21 results for FLEURS En-X test. ZEROSWOT is trained on Common Voice, while "+ More Data" indicates
21github.com/openlanguagedata/flores
that it was additionally trained on MUST-C(ASR) and LibriSpeech. NLLB results are obtained by us, by running the original versions, while SEAMLESSM4T (Seamless-Communication, 2023a) results are obtained from their repository22.
22facebookresearch/seamless_communication
Language Code
MEDIUM LARGE
NLLB Cascade ZEROSWOT + More Data SEAMLESSM4T NLLB Cascade ZEROSWOT + More Data SEAMLESSM4T
amh-Ethi 12.2 9.2 9.9 10.0 10.0 13.2 11.5 9.4 9.4 12.2 arb-Arab 23.1 18.1 18.7 18.9 20.0 26.4 20.5 21.2 21.9 22.9 asm-Beng 6.8 4.9 6.6 6.7 6.0 6.8 6.2 7.1 6.7 6.7 azj-Latn 11.0 8.3 8.7 8.6 9.2 13.2 10.2 10.4 10.4 11.0 bel-Cyrl 11.3 8.7 8.3 8.5 8.6 13.3 10.2 9.5 9.9 10.3 ben-Beng 14.8 12.0 13.5 13.7 13.2 17.6 13.6 15.0 15.2 15.1 bos-Latn 26.9 20.8 20.9 21.7 24.3 31.2 23.2 24.5 25.4 26.8 bul-Cyrl 36.7 29.4 29.8 30.7 30.4 40.4 33.4 33.2 33.5 35.5 cat-Latn 37.1 28.6 30.6 30.7 32.7 40.4 32.0 33.6 34.2 35.8 ceb-Latn 28.6 22.2 21.6 22.2 21.6 29.6 25.0 22.2 22.3 22.2 ces-Latn 28.4 21.1 20.7 21.7 22.6 30.2 24.6 23.3 24.4 24.8 ckb-Arab 8.3 5.7 7.5 7.7 8.2 10.8 7.7 8.3 8.5 10.0 zho-Hans 28.2 27.1 24.4 24.9 25.8 31.7 30.2 26.1 25.4 29.8 cym-Latn 33.2 25.3 26.2 26.7 33.7 41.5 32.4 33.0 33.4 37.4 dan-Latn 40.3 32.8 33.9 34.4 34.7 42.5 35.9 35.6 36.1 39.2 deu-Latn 34.8 28.1 27.7 28.7 28.6 37.4 31.2 31.0 31.4 32.5 ell-Grek 24.0 20.0 20.4 20.7 19.4 26.5 22.2 22.5 23.0 22.0 est-Latn 18.4 14.2 14.1 14.7 17.6 23.4 18.9 18.5 18.4 21.0 fin-Latn 18.3 13.9 14.0 14.4 15.8 22.6 18.2 18.5 19.0 20.3 fra-Latn 46.2 37.7 35.5 36.3 37.4 48.7 40.4 37.4 38.3 41.4 fuv-Latn 1.2 0.3 0.5 0.6 0.7 2.3 0.4 1.0 1.1 0.7 gaz-Latn 3.4 1.9 1.7 1.7 2.8 3.9 3.2 2.0 2.0 4.9 gle-Latn 22.5 16.6 17.0 17.7 20.7 27.4 21.0 21.1 21.3 23.4 glg-Latn 30.9 24.7 25.3 25.8 27.1 33.5 26.5 27.4 28.1 29.0 guj-Gujr 22.1 16.9 17.6 17.8 17.7 23.4 18.1 18.4 18.4 19.7 heb-Hebr 24.2 18.9 19.3 19.9 19.6 29.3 23.4 24.3 24.2 24.9 hin-Deva 29.7 24.7 24.6 25.3 27.1 30.8 26.5 25.8 26.0 28.8 hrv-Latn 23.7 18.3 18.9 19.0 21.0 28.8 21.2 22.8 23.2 22.8 hun-Latn 21.1 15.8 16.4 16.5 16.7 24.4 19.7 19.3 19.8 19.7 hye-Armn 15.9 13.1 14.7 15.2 14.8 17.9 15.1 17.1 17.1 16.2 ibo-Latn 15.6 13.1 11.6 11.7 13.6 16.7 14.5 12.0 11.9 13.8 ind-Latn 42.3 30.8 31.9 32.4 33.7 44.7 33.2 34.2 34.4 36.8 isl-Latn 19.5 14.1 14.7 15.2 15.6 22.3 17.7 16.8 17.0 20.6 ita-Latn 28.0 21.5 22.1 22.4 21.9 29.1 22.6 23.2 23.7 23.9 jav-Latn 25.5 16.9 17.8 17.8 19.8 28.0 19.1 19.4 19.6 20.4 jpn-Jpan 34.1 31.6 30.5 30.8 31.9 36.2 34.3 32.9 33.3 35.6 kan-Knda 17.7 13.1 13.4 13.6 13.5 19.4 14.3 14.3 14.2 15.2 kat-Geor 13.0 10.4 9.9 9.8 9.8 14.0 11.1 10.6 10.7 12.1 kaz-Cyrl 18.8 14.6 14.3 14.5 15.5 20.9 17.3 15.8 15.4 18.8 khk-Cyrl 9.1 6.9 7.5 7.7 9.8 11.4 9.4 9.5 9.7 12.2 khm-Khmr 0.8 0.1 2.5 2.7 0.7 1.4 0.4 2.6 2.8 0.4 kir-Cyrl 12.3 9.0 9.2 9.6 9.8 13.0 10.5 10.1 10.2 11.4 kor-Hang 12.0 9.2 9.8 10.1 10.4 12.2 11.5 10.8 10.7 11.7 lao-Laoo 54.3 49.4 53.1 53.0 54.5 56.1 53.3 54.7 54.8 55.2 lit-Latn 19.0 14.1 14.0 14.2 16.1 22.1 18.1 16.8 16.9 19.0 lug-Latn 6.6 3.6 3.6 3.4 6.4 8.6 5.2 4.9 5.0 6.8 luo-Latn 10.8 7.2 7.4 7.5 9.6 11.2 8.4 8.2 8.1 9.2 lvs-Latn 18.1 14.5 15.2 15.5 20.1 23.2 19.7 19.8 19.9 23.0 mal-Mlym 14.2 10.4 9.8 9.8 11.0 14.7 13.0 10.7 10.7 13.3 mar-Deva 13.6 10.2 10.8 10.7 11.4 16.1 11.8 12.0 12.4 13.0 mkd-Cyrl 28.9 23.5 23.7 23.5 26.5 33.3 27.4 26.9 27.2 28.8 mlt-Latn 24.2 20.9 14.8 14.6 24.0 28.8 24.0 18.0 18.0 27.6 mya-Mymr 41.4 43.1 40.4 40.3 41.5 43.3 45.8 43.1 43.3 43.0 nld-Latn 25.3 19.9 20.8 21.2 21.7 26.6 20.7 22.6 22.8 24.2 nob-Latn 30.8 24.7 25.0 25.6 26.8 33.2 26.4 26.3 27.1 28.5 npi-Deva 16.7 12.7 13.2 13.3 15.3 16.9 14.1 13.1 13.5 15.7 nya-Latn 14.3 9.6 9.2 9.1 10.4 14.2 10.9 9.6 10.1 10.8 ory-Orya 14.2 11.3 12.3 12.4 12.6 15.6 13.5 12.5 12.5 13.5 pan-Guru 22.6 17.7 18.8 19.5 19.5 24.4 19.0 21.1 21.1 21.7 pbt-Arab 12.7 10.5 10.9 11.0 11.1 13.8 10.4 11.4 11.5 11.6 pes-Arab 21.7 17.4 22.5 22.9 18.3 23.2 19.6 23.5 24.3 21.1 pol-Latn 18.3 13.6 13.6 14.1 14.6 20.4 15.9 15.9 16.2 16.9 por-Latn 45.5 37.0 37.5 38.2 38.0 47.0 39.2 39.4 40.0 40.5 ron-Latn 33.5 28.0 26.1 26.4 29.3 35.6 32.0 27.8 28.4 31.9 rus-Cyrl 27.5 21.7 22.1 22.3 22.1 29.8 24.2 24.6 25.2 25.8 slk-Latn 27.8 21.0 21.5 21.7 22.6 32.3 25.1 25.4 25.7 27.1

heading: Future Work
text: Future research will expand on low-resource scenarios and speech-to-speech translation. Limitations
Our method obtains state-of-the-art results in Speech Translation, despite being a zero-shot model. Still there are a couple of limitations inherent to the model, and on its evaluation. Although ZEROSWOT is an end-to-end model, in the sense that there is not intermediate transcription step and solves error-propagation, it suffers from some of the caveats of cascade systems. Since the speech encoder "mimics" the representation space of an MT model, no acoustic information is retained. This means that the resulting translation model can’t use acoustic information, like prosody, and can thus be sometimes limited in each ability to carry out correct translations. Furthermore, given the requirement of ASR data to train the speech encoder, this methodology can’t be used for spoken-only languages, at least in its current form. Finally, in our evaluations we did not test for a different language, other than English, in the source. This was partly due to space constrains in the paper. Although we hypothesize that our proposed method can handle equally well other source languages, it would be interesting to be applied for low-resource scenarios, which we leave as future research