# SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations

Paul-Ambroise Duquenne, Hongyu Gong, Ning Dong, Jingfei Du, Ann Lee, Vedanuj Goswami, Changhan Wang, Juan Pino, Benoît Sagot, Holger Schwenk

<figure><img src="image_1.png" /></figure>

## To cite this version:

Paul-Ambroise Duquenne, Hongyu Gong, Ning Dong, Jingfei Du, Ann Lee, et al., SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations. ACL 2023 - 61st Annual Meeting of the Association for Computational Linguistics, Aug 2023, Toronto, Canada. 10.18653/v1/2023.acl-long.899. hal-04264040

HAL Id: hal-04264040

https://inria.hal.science/hal-04264040v1

Submitted on 29 Oct 2023

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>
SpeechMatrix: A Large-Scale Mined Corpus
of Multilingual Speech-to-Speech Translations

Paul-Ambroise Duquenne<fnref n="1" />,<fnref n="2" />,* and Hongyu Gong<fnref n="1" />,*

Ning Dong<fnref n="1" /> and Jingfei Du<fnref n="1" />

Ann Lee<fnref n="1" /> and Vedanuj Goswami<fnref n="1" /> and Changhan Wang<fnref n="1" />

Juan Pino<fnref n="1" /> and Benoît Sagot<fnref n="2" /> and Holger Schwenk<fnref n="1" />

¹ Meta AI Research, <fnref n="2" /> Inria

padqn,hygong,dnn,jingfeidu,schwenk@meta.com

Abstract

We present SpeechMatrix, a large-scale multilingual corpus of speech-to-speech translations (S2ST) mined from real speech of European Parliament recordings. It contains speech alignments in 136 language pairs with a total of 418 thousand hours of speech. To evaluate the quality of this parallel speech, we train bilingual speech-to-speech translation models on mined data only and establish extensive baseline results on Europarl-ST, VoxPopuli and FLEURS test sets. Enabled by the multilinguality of SpeechMatrix, we also explore multilingual speech-to-speech translation, a topic which was addressed by few other works. We also demonstrate that model pre-training and sparse scaling using Mixture-of-Experts bring large gains to translation performance. We are open-sourcing the mined data, speech encoders used for mining, multilingual HuBERT models in four language families for target unit generation, language-specific vocoders for speech synthesis from discrete units, and S2S models trained and presented in this work.<fnref n="1" />

# 1 Introduction

Research has progressed in the area of speech-to-speech translation with the goal of seamless communication among people who speak different languages. Direct S2ST models attract increasing research interest, e.g. (Jia et al., 2019). Compared to conventional cascaded models, direct models do not rely on intermediate text representations which make them applicable to the translation of languages without a well-defined writing script. Moreover, direct S2ST have the advantage of higher training and inference efficiency (Lee et al., 2022a).

Despite the benefits of direct approaches, their training is faced with the major issue of data

scarcity in parallel speech. Human labeled speech data is expensive to create, there are very few data resources providing speech alignments, and the data amount is quite limited. To mitigate the data scarcity, some works have leveraged multi-task learning (Jia et al., 2019; Lee et al., 2022a), data augmentation with speech variations (Jia et al., 2019), or with synthesized speech (Jia et al., 2022a; Popuri et al., 2022; Nguyen et al., 2022). It is also shown useful to leverage knowledge transferred from pre-trained models (Lee et al., 2022b; Popuri et al., 2022) such as HuBERT (Hsu et al., 2021), wav2vec 2.0 (Baevski et al., 2020) and mBART (Liu et al., 2020).

Recently, Duquenne et al. (2021) is the first work to make speech mining efforts by learning a shared multilingual speech and text embedding space. Speech content is encoded by speech encoders into fixed-size representations which are then used for aligning speech and text across different languages. It demonstrates good empirical gains to train direct speech-to-text and speech-to-speech translation systems with the mined data (Duquenne et al., 2021; Lee et al., 2022b).

In this work, we trained speech encoders for 17 languages<fnref n="2" /> and mined speech-to-speech alignments for all possible language pairs from VoxPopuli (Wang et al., 2021a), a collection of European Parliament recordings. To the best of our knowledge, SpeechMatrix is by far the largest freely available speech-to-speech translation corpus, with 136 language directions and an average of 1,537 hours of source speech in each direction for a total of 418 thousand hours. We demonstrate that strong S2ST models can be trained with these mined data and validate the good quality of the speech alignments across languages. We are open-sourcing

<footnote>*Equal contributions</footnote>

<footnote n="1">¹https://github.com/facebookresearch/fairseq/tree/ust/examples/speech_matrix</footnote>

<footnote n="2">²Czech (cs), German (de), English (en), Spanish (es), Estonian (et), Finnish (fi), French (fr), Croatian (hr), Hungarian (hu), Italian (it), Lithuanian (lt), Dutch (nl), Polish (pl), Portuguese (pt), Romanian (ro), Slovak (sk) and Slovenian (sl).</footnote>
the mined data and the speech encoders used for
mining, which could pave the way for future re-
search on S2ST. Moreover, for reproducibility, we
will release model components including multilin-
gual HuBERT models in four language families for
target unit generation, language-specific vocoders
for speech synthesis from discrete units, and S2S
models trained and presented in this work.

## 2 Related Works

**From bitext mining to speech mining.** Bitext mining is to find parallel sentences from monolingual resources, which provides a large amount of training data for machine translation models. Early works on bitext mining used document meta-information (Resnik, 1999), cross-lingual document retrieval (Munteanu and Marcu, 2005) or information retrieval (Abdul-Rauf and Schwenk, 2009; Bouamor and Sajjad, 2018). More recent works use multilingual sentence embeddings (Artetxe and Schwenk, 2018; Yang et al., 2019; Schwenk et al., 2021a). The embedding based approach can be extended to new languages (Reimers and Gurevych, 2020; Heffernan et al., 2022) or the speech modality (Duquenne et al., 2021; Khurana et al., 2022) with knowledge distillation, also called teacher-student approach. These multilingual and multimodal sentence embeddings enabled us to perform large-scale speech-text mining, or speech-speech mining for a small set of languages.

**Speech-to-speech translation (S2ST).** S2ST started from cascaded systems consisting of automatic speech recognition (ASR), machine translation (MT) and text-to-speech synthesis (TTS) (Nakamura et al., 2006; Do et al., 2015). The reliance on intermediate text outputs poses limitations on cascaded models to support efficient inference and unwritten languages. Given these challenges, there has been a recent surge of research interest in direct approaches to speech translation without the need of texts. Translatotron (Jia et al., 2019) and Translatotron2 (Jia et al., 2022b) train end-to-end S2ST to generate target spectrograms with multitask learning. Another line of research replaces the target spectrograms in S2ST modeling with discrete units which are learned from a large amount of unlabeled speech (Lee et al., 2022a,b). Discrete units have shown to better capture linguistic content than spectrograms. Despite these progress on direct S2ST, it is faced with the challenge of limited parallel speech.

**Speech translation corpora.** The Fisher dataset, a collection of approximately 170 hours of telephone conversations in Spanish (Post et al., 2014), is commonly used as training data for Spanish-English S2ST. However, it does not provide parallel English speech. Previous works generate synthesized English speech from English text translations provided by Fisher. Another S2S dataset containing synthesized speech is CVSS, which covers parallel S2ST translations from 21 languages into English. It is derived from Common Voice (Ardila et al., 2020) and CoVoST 2 (Wang et al., 2021b), and it synthesizes speech from translated texts. The release of VoxPopuli dataset provides the largest S2S translations in real speech so far (Wang et al., 2021a). It covers pairwise speech-to-speech translations among 15 languages, and each direction has less than 500 hours of speech. In another initiative named FLEURS, the text-to-text evaluation data of the FLoRes-101 benchmark (Goyal et al., 2022) was extended to the speech modality. Supporting 102 languages, FLEURS has a larger language coverage than VoxPopuli, but it only contains around 12 hours of speech per language and it is intended to be used as N-way parallel test data.

In this work, we present SpeechMatrix, a large-scale multilingual speech-to-speech corpus mined from VoxPopuli (Wang et al., 2021a). It contains speech alignments in 136 language pairs with an average of 1,537-hour source speech per direction. The main characteristics of these speech corpora are summarized in Table 1.

## 3 Speech-to-Speech Mining

The mining approach of this work is built upon the idea of encoding multilingual speech utterances into a shared embedding space. Speech encoders project utterances with similar semantic content to fixed-size representations which are close in the embedding space regardless of their languages. The closeness of embeddings reflects the similarity of speech content, and is used as the alignment score in the mining process. In this section, we discuss speech encoders and speech mining.

### 3.1 Speech Encoders

We followed the teacher-student approach introduced in (Duquenne et al., 2021) and trained speech encoders with the supervision of the multilingual LASER text encoder (Schwenk et al., 2021b). Tran-
<table><caption>Table 1: A comparison of existing speech-to-speech datasets.</caption><thead><tr><th>Dataset</th><th># of Languages</th><th>Avg. duration (h)</th><th>Source speech</th><th>Target speech</th></tr></thead><tbody><tr><td>Fisher (Post et al., 2014)</td><td>2</td><td>127</td><td>Telephone conversation</td><td>Synthetic</td></tr><tr><td>MaSS (Boito et al., 2020)</td><td>8</td><td>20</td><td>Bible reading</td><td>Bible reading</td></tr><tr><td>VoxPopuli (Wang et al., 2021a)</td><td>15</td><td>82</td><td>European Parliament speech</td><td>Simultaneous interpretation</td></tr><tr><td>CVSS (C+T) (Jia et al., 2022c)</td><td>21</td><td>181</td><td>Read</td><td>Synthetic</td></tr><tr><td>FLEURS (Conneau et al., 2022)</td><td>102</td><td>12</td><td>Read</td><td>Read</td></tr><tr><td>SpeechMatrix (ours)</td><td>17</td><td>1537</td><td>European Parliament speech</td><td>European Parliament speech</td></tr></tbody></table>

criptions or written translation of the audio utterances are encoded with LASER text encoder as target vectors for speech encoder training. During training, we minimize the cosine loss between fixed-size representations output by speech encoders, and the outputs of LASER text encoder (whose weights are frozen during training). Speech encoders are initialized with the 2B-parameter XLS-R model (Babu et al., 2021), which was pre-trained on nearly half a million hours of publicly available audios in 128 languages. Following (Duquenne et al., 2022), the fixed-size representation for speech is obtained with max pooling of the encoder outputs which appeared to work better compared to other pooling methods. We summarize the architecture of the speech encoder in Figure 1.

We used various publicly available ASR data sets which cover our languages to train the speech encoders, including CoVoST 2 (Wang et al., 2020, 2021b), Common Voice (Ardila et al., 2020), Europarl (Ardila et al., 2020), mTedx (Salesky et al., 2021), Must-C (Di Gangi et al., 2019) and VoxPopuli (Wang et al., 2021a), as well as speech translation data from the foreign languages into English and from English into German. We removed training samples whose transcription or the written translation consisted of multiple sentences, as LASER has been trained on single sentences only. For better training efficiency, we trained speech encoders for each language family instead of each

<figure><img src="image_3.png" /><figcaption>Figure 1: Architecture of speech encoders training.</figcaption></figure>

language. The language grouping is provided in Appendix. To better handle imbalanced training data, we sample the training data from different languages with the same approach as (Duquenne et al., 2021). For English (en), Slovenian (sl), Lithuanian (lt) and Dutch (nl), we also trained separate monolingual speech encoders that had lower valid cosine loss compared to multilingual encoders, and these four monolingual encoders were used for mining.

## 3.2 Evaluation of speech encoders

Similarity search is frequently used to evaluate multilingual text encoders (Artetxe and Schwenk, 2018; Feng et al., 2020; Heffernan et al., 2022). We use the following score to measure similarity between the source audio, and the target transcriptions or translations:

$$ \begin{aligned} \text{sim}(x,y) &= \cos(x, y) - \left( \sum_{z \in NN_k(x)} \frac{\cos(x, z)}{2k} + \sum_{z \in NN_k(y)} \frac{\cos(y, z)}{2k} \right) \end{aligned} \quad (1) $$

where x and y are the source and target embeddings, and $NN_k(x)$ denotes the k nearest neighbors of x. We used $k = 4$. We evaluated similarity search of audios against transcriptions on VoxPopuli ASR test set in Table 2, which is our target domain as we plan to mine unlabeled speech from VoxPopuli (see subsection 3.3). We also evaluated similarity search of audio against written translations or transcriptions on CoVoST 2 test set in order to compare with speech encoders in previous work (see detailed analysis in Appendix A). Finally, we report text-to-text similarity search using the LASER text encoder as lower bound for the speech translation similarity search error rate since we use gold transcriptions to search against written translations. We report error rates (in %) that are percentage of audio utterances incorrectly matched with text transcripts from the same test set. We note that error rates are very low for all languages (below 5% and around 1 or 2% for most languages), which is an initial validation of good-quality speech encoders before the large-scale mining.
<table><caption>Table 2: Similarity search error rates (in %) on VoxPopuli ASR test set.</caption><thead><tr><th>Sim Search</th><th>cs</th><th>de</th><th>en</th><th>es</th><th>et</th><th>fi</th><th>fr</th><th>hr</th><th>hu</th><th>it</th><th>lt</th><th>nl</th><th>pl</th><th>pt</th><th>ro</th><th>sk</th><th>sl</th></tr></thead><tbody><tr><td># test sentences</td><td>1k</td><td>1.7k</td><td>1.5k</td><td>1.4k</td><td>47</td><td>0.4k</td><td>1.5k</td><td>0.3k</td><td>1k</td><td>1k</td><td>39</td><td>1k</td><td>1.6k</td><td>—</td><td>1.3k</td><td>0.6k</td><td>0.3k</td></tr><tr><td>Audio vs. transcriptions</td><td>0.6</td><td>1.0</td><td>0.2</td><td>0.7</td><td>0.0</td><td>0.7</td><td>0.5</td><td>0.3</td><td>1.1</td><td>4.9</td><td>0.0</td><td>0.8</td><td>0.9</td><td>—</td><td>0.9</td><td>0.7</td><td>3.1</td></tr></tbody></table>

## 3.3 Large-scale speech mining

We used VoxPopuli as our source of unlabeled un-segmented speech for 17 languages in focus. In principle, performing speech-to-speech or speech-to-text mining can be done with exactly the same pipeline as text-to-text mining but with different encoders. We follow the global mining approach as described in Schwenk et al. (2021a) and compare all segments in the source language with all segments in the target language. Similarity scores are calculated in both directions using the margin as described in Equation 1 considering $k = 16$ neighbors. Segments are considered to be parallel if the margin score exceeds a threshold, we use 1.06 if not specified otherwise. The reader is referred to Schwenk et al. (2021a) for a detailed description of the generic mining pipeline.

There is however one important difference when processing speech: it is not straightforward to segment the audio signal into parts which have the optimal granularity for mining. The VoxPopuli recordings have a rather long duration, e.g. one hour and a half on average for English. We apply Voice Activity Detection (VAD) using Silero-VAD (Silero-Team, 2021) which supports over 100 languages. The resulting segments do not necessarily correspond to complete sentences. On one hand, there may be silence in the middle of an utterance, e.g. a hesitation. On the other hand, two sentences may follow each other without a long silence separating them. We follow the “over segmentation” approach outlined in Duquenne et al. (2021): several possible segments are created and we let the mining algorithm decide which ones match the best. Initial experiments suggest that segments shorter than 1 second or longer than 20 seconds are unlikely to be aligned and therefore were excluded.

After mining, the resulting speech alignments may have overlap as we over-segment the unlabeled speech. A post-processing method Duquenne et al. (2021) is introduced to remove overlaps between mined speech segments on the source speech side. We relax the post-processing of the mined data, allowing for some overlap between mined speech segments: for two audio segments that overlap on

the source side, if the overlap represents more than 20% of the first segment and of the second segment, we discard the alignment with the lowest mining score. We did an ablation study on different thresholds of overlap ratio for one low-resource, one mid-resource and one high-resource direction and found that 20% was the best overlap threshold in all settings.

We report the statistics of the mined speech-to-speech translation pairs in Table 3, with a mining score threshold of 1.06. The mined data totals 418k hours of parallel speech with an average of 1,537 hours of source speech in all translation directions. While some high resource languages like English (en), Spanish (es) or French (fr) can reach up to 5k hours of aligned speech with other spoken languages; lower resource languages such as Estonian (et) and Lithuanian (lt) obtain much fewer alignments, with only a few hours of aligned speech for Lithuanian. We also performed mining of the source speech in sixteen languages against more than twenty billion English sentences from Common Crawl. This yielded speech-text alignments between 827 and 3,966 hours (c.f. the last column of Table 3). Training and evaluation of speech-to-text translation are left for future research.

## 3.4 Evaluation Data

Besides the speech-to-speech data mined as the train set, we leverage labeled public speech datasets as the evaluation sets.

**Test set.** In our experiments, we derive test sets in speech translation from three public corpora, evaluating translation models trained on mined data across different domains.

(1) Europarl-ST (EPST) (Iranzo-Sánchez et al., 2020). It is a multilingual speech-to-text translation corpus built on recordings of debates from the European Parliament, containing 72 translation directions in 9 languages.<fnref n="3" />

(2) VoxPopuli (Wang et al., 2021a). S2S data, as part of VoxPopuli release, provides aligned source and target speech together with source transcriptions. We prepare the speech-to-text data with

<footnote n="3">³en, fr, de, it, es, pt, pl, ro and nl</footnote>
<table><caption>Table 3: Duration statistics (hours of source speech) of speech-to-speech alignments for each pair of 17 languages (for mining threshold of 1.06). The last column provides statistics for alignments of source speech against 21.5 billion sentences of English texts. The last row provides duration of raw speech from VoxPopuli used for mining.</caption><thead><tr><th rowspan="2">Src/Tgt</th><th rowspan="2">cs</th><th rowspan="2">de</th><th rowspan="2">en</th><th rowspan="2">es</th><th rowspan="2">et</th><th rowspan="2">fi</th><th rowspan="2">fr</th><th colspan="8">Speech targets</th><th rowspan="2">Text<br>en</th></tr><tr><th>hr</th><th>hu</th><th>it</th><th>lt</th><th>nl</th><th>pl</th><th>pt</th><th>ro</th><th>sk</th><th>sl</th></tr></thead><tbody><tr><td>cs</td><td>-</td><td>2381</td><td>3208</td><td>2290</td><td>952</td><td>1312</td><td>2476</td><td>726</td><td>1396</td><td>2410</td><td>84</td><td>2377</td><td>2516</td><td>1867</td><td>1190</td><td>2146</td><td>452</td><td>2528</td></tr><tr><td>de</td><td>2386</td><td>-</td><td>4734</td><td>3113</td><td>901</td><td>1477</td><td>3536</td><td>498</td><td>1871</td><td>3476</td><td>41</td><td>3384</td><td>2632</td><td>2250</td><td>1281</td><td>1646</td><td>361</td><td>3073</td></tr><tr><td>en</td><td>3172</td><td>4676</td><td>-</td><td>4715</td><td>1585</td><td>2169</td><td>5178</td><td>824</td><td>2266</td><td>4897</td><td>82</td><td>4422</td><td>3583</td><td>3572</td><td>2258</td><td>2306</td><td>586</td><td>-</td></tr><tr><td>es</td><td>2240</td><td>3041</td><td>4708</td><td>-</td><td>862</td><td>1373</td><td>4446</td><td>528</td><td>1599</td><td>4418</td><td>47</td><td>3067</td><td>2646</td><td>3484</td><td>1857</td><td>1603</td><td>308</td><td>3966</td></tr><tr><td>et</td><td>943</td><td>892</td><td>1593</td><td>877</td><td>-</td><td>1201</td><td>934</td><td>265</td><td>1119</td><td>1019</td><td>39</td><td>1055</td><td>949</td><td>721</td><td>419</td><td>780</td><td>196</td><td>1578</td></tr><tr><td>fi</td><td>1296</td><td>1463</td><td>2180</td><td>1393</td><td>1197</td><td>-</td><td>1449</td><td>306</td><td>1473</td><td>1599</td><td>47</td><td>1654</td><td>1350</td><td>1128</td><td>621</td><td>977</td><td>260</td><td>1969</td></tr><tr><td>fr</td><td>2424</td><td>3457</td><td>5171</td><td>4455</td><td>923</td><td>1435</td><td>-</td><td>560</td><td>1711</td><td>4618</td><td>50</td><td>3273</td><td>2822</td><td>3384</td><td>1991</td><td>1657</td><td>326</td><td>3966</td></tr><tr><td>hr</td><td>736</td><td>507</td><td>854</td><td>553</td><td>273</td><td>317</td><td>588</td><td>-</td><td>328</td><td>615</td><td>24</td><td>546</td><td>660</td><td>433</td><td>277</td><td>586</td><td>136</td><td>1311</td></tr><tr><td>hu</td><td>1417</td><td>1897</td><td>2346</td><td>1672</td><td>1140</td><td>1507</td><td>1787</td><td>328</td><td>-</td><td>1855</td><td>68</td><td>1839</td><td>1566</td><td>1315</td><td>808</td><td>1064</td><td>311</td><td>2301</td></tr><tr><td>it</td><td>2404</td><td>3460</td><td>4948</td><td>4500</td><td>1028</td><td>1614</td><td>4700</td><td>607</td><td>1823</td><td>-</td><td>103</td><td>3414</td><td>2848</td><td>3421</td><td>1995</td><td>1656</td><td>474</td><td>2891</td></tr><tr><td>lt</td><td>78</td><td>38</td><td>79</td><td>46</td><td>37</td><td>44</td><td>48</td><td>21</td><td>61</td><td>95</td><td>-</td><td>77</td><td>80</td><td>35</td><td>18</td><td>64</td><td>6</td><td>827</td></tr><tr><td>nl</td><td>2322</td><td>3305</td><td>4396</td><td>3066</td><td>1040</td><td>1633</td><td>3269</td><td>521</td><td>1768</td><td>3355</td><td>80</td><td>-</td><td>2459</td><td>2399</td><td>1352</td><td>1646</td><td>458</td><td>2708</td></tr><tr><td>pl</td><td>2530</td><td>2646</td><td>3662</td><td>2735</td><td>967</td><td>1378</td><td>2913</td><td>656</td><td>1554</td><td>2883</td><td>88</td><td>2540</td><td>-</td><td>2121</td><td>1301</td><td>1892</td><td>431</td><td>2871</td></tr><tr><td>pt</td><td>1849</td><td>2224</td><td>3606</td><td>3525</td><td>722</td><td>1131</td><td>3421</td><td>421</td><td>1279</td><td>3403</td><td>37</td><td>2436</td><td>2087</td><td>-</td><td>1579</td><td>1358</td><td>247</td><td>3540</td></tr><tr><td>ro</td><td>1187</td><td>1275</td><td>2290</td><td>1894</td><td>423</td><td>627</td><td>2024</td><td>271</td><td>789</td><td>1996</td><td>19</td><td>1384</td><td>1288</td><td>1592</td><td>-</td><td>870</td><td>125</td><td>2784</td></tr><tr><td>sk</td><td>2127</td><td>1628</td><td>2329</td><td>1631</td><td>781</td><td>982</td><td>1685</td><td>574</td><td>1038</td><td>1650</td><td>69</td><td>1676</td><td>1869</td><td>1361</td><td>867</td><td>-</td><td>370</td><td>2090</td></tr><tr><td>sl</td><td>436</td><td>350</td><td>579</td><td>307</td><td>192</td><td>254</td><td>324</td><td>128</td><td>295</td><td>461</td><td>6</td><td>454</td><td>413</td><td>241</td><td>121</td><td>359</td><td>-</td><td>1267</td></tr></tbody><tfoot><tr><td colspan="20"># hours of unlabeled speech</td></tr><tr><td></td><td>18.7k</td><td>23.2k</td><td>24.1k</td><td>21.4k</td><td>10.6k</td><td>14.2k</td><td>22.8k</td><td>8.1k</td><td>17.7k</td><td>21.9k</td><td>14.4k</td><td>19.0k</td><td>21.2k</td><td>17.5k</td><td>17.9k</td><td>12.1k</td><td>11.3k</td><td></td></tr></tfoot></table>

target speech and source transcription as our test set. To ensure that there is no overlap between the mined data and VoxPopuli test sets, we need to remove speech from mined alignments which are from the same session as test samples. In order to keep as much mined data as possible, we use VoxPopuli test set only when a language direction is not covered by EPST considering their domain similarity. Moreover, similarity scores are provided to indicate the quality of VoxPopuli samples. To choose high-quality data, we sort all sessions in the VoxPopuli S2S data in a decreasing order of the average similarity score of their samples. We keep adding samples from highly ranked sessions to the test set until the test size reaches 1000.

### (3) FLEURS (Conneau et al., 2022).
Built upon N-way text translations from FLoRes (Goyal et al., 2022), FLEURS provides speech for aligned texts and creates speech-to-speech data covering all mined directions. We take its source speech and target texts as the test data. In the case where multiple utterances correspond to one piece of source text, we generate one test pair for each source utterance respectively. FLEURS texts are from English Wikipedia, which is a different domain from VoxPopuli and EPST.

**Valid set.** Valid sets are prepared for S2S modeling using VoxPopuli and FLEURS data in a similar way as test sets. For VoxPopuli, we extract a valid set of about 1000 samples by adding data from highly scored sessions which are not in the test set. FLEURS valid set is derived from its valid samples.

We prepare speech-to-unit data from these selected valid samples by transforming the target speech into target units for speech-to-unit modeling, which will be discussed in section 4.

## 4 Experiments & Results

To evaluate the quality of the mined data, we trained S2ST models on SpeechMatrix data and report the translation performance. We hope that these results will serve as baselines for future studies in speech-to-speech translation.

<figure><img src="image_4.png" /><figcaption>Figure 2: A Pipeline of Speech-to-Speech Translation and Evaluation.</figcaption></figure>

<table><caption>Table 4: BLEU scores on EPST test sets by S2ST models with different training data.</caption><thead><tr><th>Train set</th><th></th><th>Es-En</th><th>Fr-En</th><th>En-Es</th><th>En-Fr</th></tr></thead><tbody><tr><td>VoxPopuli</td><td>Hours</td><td>532</td><td>523</td><td>415</td><td>451</td></tr><tr><td>S2S</td><td>BLEU</td><td>13.1</td><td>15.4</td><td>16.4</td><td>15.8</td></tr><tr><td>SpeechMatrix</td><td>Hours</td><td>1,353</td><td>1,507</td><td>1,366</td><td>1,518</td></tr><tr><td>(t = 1.09)</td><td>BLEU</td><td>20.4</td><td>20.7</td><td>21.9</td><td>19.3</td></tr></tbody></table>
<table><caption>Table 5: BLEU scores of bilingual S2S models on EP/VP test sets. EPST score is underscored.</caption><thead><tr><th></th><th>cs</th><th>de</th><th>en</th><th>es</th><th>et</th><th>fi</th><th>fr</th><th>hr</th><th>hu</th><th>it</th><th>lt</th><th>nl</th><th>pl</th><th>pt</th><th>ro</th><th>sk</th><th>sl</th></tr></thead><tbody><tr><td>cs</td><td>-</td><td>12.9</td><td>22.7</td><td>16.7</td><td>-</td><td>0.6</td><td>21.1</td><td>4.4</td><td>0.5</td><td>10.2</td><td>0.1</td><td>6.1</td><td>8.5</td><td>-</td><td>4.3</td><td>16.9</td><td>3.0</td></tr><tr><td>de</td><td>7.3</td><td>-</td><td><u>16.3</u></td><td><u>11.7</u></td><td>-</td><td>1.2</td><td><u>10.7</u></td><td>4.5</td><td>0.6</td><td><u>3.8</u></td><td>0.1</td><td><u>10.4</u></td><td><u>3.5</u></td><td><u>7.1</u></td><td><u>5.2</u></td><td>3.0</td><td>4.1</td></tr><tr><td>en</td><td>8.2</td><td><u>10.1</u></td><td>-</td><td><u>21.9</u></td><td>-</td><td>1.9</td><td><u>19.2</u></td><td>8.4</td><td>1.1</td><td><u>11.5</u></td><td>0.3</td><td><u>15.1</u></td><td><u>8.2</u></td><td><u>11.8</u></td><td><u>7.6</u></td><td>5.7</td><td>5.5</td></tr><tr><td>es</td><td>5.2</td><td><u>6.1</u></td><td><u>20.4</u></td><td>-</td><td>-</td><td>1.3</td><td><u>16.3</u></td><td>3.6</td><td>0.7</td><td><u>11.1</u></td><td>0.1</td><td><u>8.0</u></td><td><u>3.9</u></td><td><u>13.3</u></td><td><u>5.2</u></td><td>2.2</td><td>2.2</td></tr><tr><td>et</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>fi</td><td>3.0</td><td>9.0</td><td>19.7</td><td>11.4</td><td>-</td><td>-</td><td>14.1</td><td>1.5</td><td>0.0</td><td>5.8</td><td>0.1</td><td>6.6</td><td>4.5</td><td>-</td><td>4.4</td><td>1.7</td><td>1.6</td></tr><tr><td>fr</td><td>5.4</td><td><u>6.3</u></td><td><u>20.7</u></td><td><u>18.4</u></td><td>-</td><td>0.8</td><td>-</td><td>5.4</td><td>0.7</td><td><u>10.2</u></td><td>0.1</td><td><u>8.4</u></td><td><u>4.8</u></td><td><u>13.4</u></td><td><u>5.6</u></td><td>1.6</td><td>1.5</td></tr><tr><td>hr</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>hu</td><td>2.6</td><td>7.3</td><td>15.3</td><td>9.5</td><td>-</td><td>0.7</td><td>13.8</td><td>1.9</td><td>-</td><td>6.3</td><td>0.1</td><td>3.0</td><td>1.6</td><td>-</td><td>2.4</td><td>0.9</td><td>1.2</td></tr><tr><td>it</td><td>6.4</td><td><u>4.9</u></td><td><u>18.9</u></td><td><u>19.6</u></td><td>-</td><td>0.4</td><td><u>15.3</u></td><td>5.2</td><td>0.7</td><td>-</td><td>0.1</td><td><u>6.5</u></td><td><u>3.6</u></td><td><u>12.4</u></td><td><u>3.7</u></td><td>2.1</td><td>2.8</td></tr><tr><td>lt</td><td>0.2</td><td>0.0</td><td>3.1</td><td>0.8</td><td>-</td><td>0.0</td><td>0.7</td><td>0.1</td><td>0.0</td><td>0.6</td><td>-</td><td>0.7</td><td>0.1</td><td>-</td><td>0.0</td><td>0.0</td><td>0.1</td></tr><tr><td>nl</td><td>3.5</td><td><u>8.1</u></td><td><u>18.0</u></td><td><u>13.2</u></td><td>-</td><td>0.5</td><td><u>13.0</u></td><td>3.3</td><td>0.4</td><td><u>5.2</u></td><td>0.1</td><td>-</td><td><u>3.4</u></td><td><u>6.7</u></td><td><u>4.1</u></td><td>1.7</td><td>2.1</td></tr><tr><td>pl</td><td>7.2</td><td><u>2.8</u></td><td><u>4.9</u></td><td><u>6.3</u></td><td>-</td><td>1.0</td><td><u>5.5</u></td><td>4.5</td><td>0.5</td><td><u>5.8</u></td><td>0.2</td><td><u>1.6</u></td><td>-</td><td><u>6.1</u></td><td><u>3.2</u></td><td>4.7</td><td>2.4</td></tr><tr><td>pt</td><td>-</td><td><u>4.7</u></td><td><u>21.2</u></td><td><u>23.2</u></td><td>-</td><td>-</td><td><u>18.1</u></td><td>-</td><td>-</td><td><u>4.4</u></td><td>-</td><td><u>5.0</u></td><td><u>3.6</u></td><td>-</td><td><u>4.4</u></td><td>-</td><td>-</td></tr><tr><td>ro</td><td>4.6</td><td><u>6.5</u></td><td><u>22.6</u></td><td><u>20.1</u></td><td>-</td><td>0.8</td><td><u>18.6</u></td><td>2.4</td><td>0.4</td><td><u>8.7</u></td><td>0.1</td><td><u>3.5</u></td><td><u>4.6</u></td><td><u>10.3</u></td><td>-</td><td>2.3</td><td>0.7</td></tr><tr><td>sk</td><td>28.2</td><td>10.7</td><td>21.4</td><td>15.5</td><td>-</td><td>1.0</td><td>19.2</td><td>5.0</td><td>0.5</td><td>4.7</td><td>0.1</td><td>4.2</td><td>5.3</td><td>-</td><td>4.4</td><td>-</td><td>3.6</td></tr><tr><td>sl</td><td>4.0</td><td>11.1</td><td>19.5</td><td>8.6</td><td>-</td><td>0.8</td><td>13.2</td><td>4.8</td><td>0.4</td><td>6.0</td><td>0.1</td><td>4.5</td><td>6.7</td><td>-</td><td>1.1</td><td>1.7</td><td>-</td></tr></tbody></table>

## 4.1 Experimental Setup

The training and evaluation pipeline of speech-to-speech translation is shown in Figure 2. Recent progress in speech-to-speech modeling suggests to discretize the target speech waveform into a unit sequence, relieving models from the complexity of predicting continuous waveform values. We borrow the idea of training speech-to-unit (S2U) model where units are pre-generated from target speech with a pre-trained HuBERT model (Lee et al., 2022a). During S2U training, models are periodically evaluated on the valid set of speech-to-unit samples, and the best checkpoint with the lowest valid loss is saved for model inference.

When it comes to inference, speech could be synthesized from the predicted units with a vocoder, as the output of the S2S pipeline. It is then transcribed into texts by an off-the-shelf ASR model. The BLEU score is calculated by comparing the transcriptions against the ground truth target texts, which serves as the quantitative metric of mined data quality. We note that the ASR BLEU score is not a perfect metric for data quality, as it is unavoidably affected by the quality of ASR models. Next we discuss each module of the pipeline.

**Speech-to-Unit.** The S2U model takes the source speech and predicts a sequence of target units. It typically has an encoder-decoder architecture, where the encoder consists of convolutional and Transformer encoder layers, and the decoder is a Transformer decoder. We have experimented with different model variants, and discuss bilingual

and multilingual training in section 5 and section 6.

**HuBERT.** HuBERT is used to extract speech features of audio frames, which are then grouped into *k*-means clusters. The continuous features are thus mapped to corresponding clusters. In this way, speech could be discretized into unit sequence where units are basically indices of clusters. We reuse the same HuBERT model and *k*-means clusters for English, Spanish and French as in (Lee et al., 2022b) for a fair comparison with existing results. We also train multilingual HuBERT models to cover other languages in SpeechMatrix, and more HuBERT training details can be found in Appendix B.1.

**Vocoder.** Unit-based HiFi-GAN vocoders are trained to synthesize speech from unit sequence (Polyak et al., 2021). In our experiments, vocoders are separately trained from S2U model. We train vocoders on three datasets:

(1) CSS10 (Park and Mulc, 2019). It is a single-speaker corpus which we use to train vocoders in German, Finnish, Hungarian and Dutch.

(2) VoxPopuli (Wang et al., 2021a). Given its ASR data with speaker id, we sort speakers based on their speech duration, and keep adding the top speakers until the speech is more than 20 hours.

(3) Common Voice (Ardila et al., 2020). Portuguese and Estonian are not covered by the two corpora above, and thus we turn to Common Voice. Again, we select top speakers and prepare 12-hour and 10-hour speech for the vocoder training in Portuguese and Estonian respectively.
Data preprocessing and training are included in Appendix B.3.

**ASR.** We use off-the-shelf ASR models to transcribe the speech generated by vocoders. Details about the ASR models and their benchmark results of word error rates are provided in Appendix B.2.

# 5 Bilingual Speech-to-Speech Baselines

In this part, we discuss the bilingual S2S models trained in each of 272 language directions in SpeechMatrix. The architecture of Textless model is used for bilingual translation in our experiments (Lee et al., 2022a). A Textless model consists of a speech encoder, Transformer encoder and decoder.

**Training.** For a given direction, we extract units for source and target speech with their corresponding HuBERT models (Hsu et al., 2021). Taking source speech, the model is trained to predict target unit sequence with cross-entropy loss as well as source unit reconstruction as an auxiliary task.

For the training efficiency of extensive S2ST experiments, we use a subset of mined data as the train set. Mined samples are selected if their alignment scores are above a preset threshold. We performed an analysis of the threshold selection in subsection B.4.

**Comparison with existing results.** Since we adopt the same model as the previous work (Lee et al., 2022a) and the only difference lies in the train set, it is straightforward to compare with existing results. Table 4 shows the results of S2ST models which are trained on our SpeechMatrix mined data compared to VoxPopuli S2S data in each of four language directions: es-en, fr-en, en-es and en-fr. The threshold of mined data is set as 1.09 to these four directions, yielding an average of 1, 436-hour train set. Compared with 480-hour labeled speech from VoxPopuli, SpeechMatrix achieves an average improvement of 5.4 BLEU, indicating the good quality and usefulness of the mined data.

## 5.1 Large-Scale Bilingual Evaluation

A large-scale evaluation is launched covering 272 mined languages directions, and bilingual models are trained for each direction to establish baseline results in speech-to-speech translation.

Table 5 and Table 6 summarize performance of bilingual S2ST models on three test sets. In each direction, Table 5 reports BLEU scores in European Parliament domain, either EPST or VoxPopuli set. EPST BLEU is underlined to be distinguished

from VoxPopuli BLEU. Table 6 reports BLEU in Wikipedia domain, i.e., FLEURS test data.

**Bilingual results.** Empirically we find that translations into high-resource languages such as en, es and fr outperform those into low-resource languages such as lt and sl based on the speech amount of these languages in Table 3. Another observation is the performance difference across test domains, and BLEU on FLEURS is lower than that on EPST and VoxPopuli data, likely because of the domain mismatch between train and test data.

It is also found that translation results are not symmetric for some language pairs, for example, ro-en has a BLEU of 22.6 while en-ro BLEU is only 7.6 on EPST. Besides different complexity levels of target languages and test sets, such asymmetry also results from the dependency of BLEU scores on the speech synthesis quality of the vocoder and transcription quality of the ASR model. For languages whose vocoder and ASR models are not good, they are likely to receive low BLEU scores. In this case, Romanian vocoder and ASR are not as strong as English models as reflected by its higher word error rate in speech resynthesis as reported in Appendix B.3.

# 6 Multilingual Speech-to-Speech Translation

Multilingual modeling has been explored in tasks of language understanding and machine translation, demonstrating knowledge transfer among languages. However, to our best knowledge, there are few studies of multilingual S2ST on real speech, partially due to the lack of multilingual speech-to-speech resources. With the massively multilingual data we have mined, we are able to explore multilingual S2ST training.

In this work, we focus on many-to-English translation, studying the translation from 6 Slavic languages to English in subsection 6.1 and the translation from all 16 languages in SpeechMatrix to English in subsection 6.2. English-to-many or many-to-many translation are left to future work. We present here multilingual models used in our experiments (more details can be found in Appendix C).

(1) **Textless model.** The same model with 70M parameters that we use for bilingual evaluation is reused in the multilingual experiments. Given diverse multilingual data, we increase the model size for larger model capacity, trying multilingual models with 70M and 260M parameters.
<table><caption>Table 6: BLEU scores of bilingual S2S models on FLEURS sets.</caption><thead><tr><th></th><th>cs</th><th>de</th><th>en</th><th>es</th><th>et</th><th>fi</th><th>fr</th><th>hr</th><th>hu</th><th>it</th><th>lt</th><th>nl</th><th>pl</th><th>pt</th><th>ro</th><th>sk</th><th>sl</th></tr></thead><tbody><tr><th>cs</th><td>-</td><td>2.0</td><td>4.2</td><td>4.6</td><td>0.1</td><td>0.2</td><td>7.5</td><td>2.1</td><td>0.2</td><td>2.5</td><td>0.1</td><td>1.0</td><td>2.3</td><td>2.8</td><td>1.4</td><td>3.5</td><td>1.7</td></tr><tr><th>de</th><td>2.3</td><td>-</td><td>8.3</td><td>3.8</td><td>0.1</td><td>0.2</td><td>6.5</td><td>2.2</td><td>0.2</td><td>1.8</td><td>0.0</td><td>1.2</td><td>0.9</td><td>3.1</td><td>2.1</td><td>0.8</td><td>1.0</td></tr><tr><th>en</th><td>2.7</td><td>2.7</td><td>-</td><td>6.0</td><td>0.7</td><td>0.6</td><td>10.4</td><td>2.4</td><td>0.3</td><td>3.6</td><td>0.1</td><td>3.8</td><td>1.3</td><td>5.1</td><td>2.0</td><td>1.2</td><td>1.2</td></tr><tr><th>es</th><td>1.9</td><td>1.8</td><td>7.5</td><td>-</td><td>0.1</td><td>0.2</td><td>9.2</td><td>1.0</td><td>0.2</td><td>4.2</td><td>0.1</td><td>1.5</td><td>1.4</td><td>5.9</td><td>2.3</td><td>0.9</td><td>0.8</td></tr><tr><th>et</th><td>2.1</td><td>0.7</td><td>8.2</td><td>3.0</td><td>-</td><td>0.7</td><td>6.3</td><td>1.0</td><td>0.7</td><td>2.3</td><td>0.1</td><td>1.5</td><td>1.2</td><td>1.7</td><td>1.4</td><td>0.4</td><td>0.8</td></tr><tr><th>fi</th><td>1.5</td><td>0.9</td><td>5.5</td><td>3.8</td><td>0.5</td><td>-</td><td>6.2</td><td>0.5</td><td>0.0</td><td>1.2</td><td>0.0</td><td>0.8</td><td>1.2</td><td>2.0</td><td>1.1</td><td>0.7</td><td>0.7</td></tr><tr><th>fr</th><td>1.5</td><td>2.1</td><td>9.8</td><td>7.6</td><td>0.1</td><td>0.2</td><td>-</td><td>1.7</td><td>0.2</td><td>3.1</td><td>0.1</td><td>1.3</td><td>1.5</td><td>5.8</td><td>2.4</td><td>0.6</td><td>0.6</td></tr><tr><th>hr</th><td>2.5</td><td>0.9</td><td>7.7</td><td>3.1</td><td>0.2</td><td>0.1</td><td>5.8</td><td>-</td><td>0.2</td><td>1.1</td><td>0.0</td><td>0.9</td><td>1.1</td><td>2.0</td><td>0.6</td><td>0.9</td><td>0.8</td></tr><tr><th>hu</th><td>1.3</td><td>1.0</td><td>4.6</td><td>3.0</td><td>0.1</td><td>0.2</td><td>5.7</td><td>0.7</td><td>-</td><td>1.2</td><td>0.0</td><td>0.1</td><td>0.4</td><td>2.3</td><td>0.9</td><td>0.2</td><td>0.3</td></tr><tr><th>it</th><td>1.3</td><td>1.0</td><td>6.3</td><td>8.3</td><td>0.1</td><td>0.1</td><td>11.3</td><td>1.3</td><td>0.2</td><td>-</td><td>0.0</td><td>0.9</td><td>1.1</td><td>5.6</td><td>1.9</td><td>0.4</td><td>0.6</td></tr><tr><th>lt</th><td>0.1</td><td>0.0</td><td>0.9</td><td>0.2</td><td>0.0</td><td>0.0</td><td>0.2</td><td>0.0</td><td>0.0</td><td>0.4</td><td>-</td><td>0.1</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><th>nl</th><td>1.4</td><td>3.1</td><td>5.7</td><td>4.9</td><td>0.2</td><td>0.2</td><td>7.5</td><td>1.8</td><td>0.2</td><td>1.7</td><td>0.0</td><td>-</td><td>0.9</td><td>3.3</td><td>1.4</td><td>0.4</td><td>1.0</td></tr><tr><th>pl</th><td>1.6</td><td>1.6</td><td>4.9</td><td>4.4</td><td>0.1</td><td>0.2</td><td>5.4</td><td>1.2</td><td>0.1</td><td>1.5</td><td>0.0</td><td>0.3</td><td>-</td><td>2.5</td><td>1.2</td><td>1.1</td><td>0.7</td></tr><tr><th>pt</th><td>1.2</td><td>1.0</td><td>6.1</td><td>8.7</td><td>0.1</td><td>0.3</td><td>11.1</td><td>1.1</td><td>0.1</td><td>1.1</td><td>0.1</td><td>0.6</td><td>0.8</td><td>-</td><td>1.5</td><td>0.6</td><td>0.6</td></tr><tr><th>ro</th><td>1.9</td><td>2.2</td><td>7.8</td><td>7.0</td><td>0.4</td><td>0.3</td><td>11.3</td><td>0.9</td><td>0.2</td><td>3.8</td><td>0.1</td><td>0.9</td><td>1.1</td><td>6.0</td><td>-</td><td>0.7</td><td>0.2</td></tr><tr><th>sk</th><td>9.1</td><td>2.1</td><td>5.5</td><td>5.1</td><td>0.3</td><td>0.2</td><td>7.8</td><td>3.0</td><td>0.4</td><td>2.1</td><td>0.0</td><td>0.7</td><td>1.9</td><td>2.3</td><td>1.9</td><td>-</td><td>1.5</td></tr><tr><th>sl</th><td>2.2</td><td>2.0</td><td>7.3</td><td>3.4</td><td>0.2</td><td>0.3</td><td>4.5</td><td>1.1</td><td>0.1</td><td>1.2</td><td>0.0</td><td>1.0</td><td>1.2</td><td>1.5</td><td>0.1</td><td>0.3</td><td>-</td></tr></tbody></table>

(2) **XM Transformer.** Inspired by the recent finding that crossmodal pre-training is beneficial for speech translation (Popuri et al., 2022), we apply XM Transformer to multilingual training, whose encoder is initialized from pre-trained XLS-R model with 1B parameters (Babu et al., 2021) and decoder is initialized from a unit decoder pre-trained in an mBART style (Popuri et al., 2022). With multilingual speech-to-unit data, the model is further finetuned to minimize the cross-entropy loss in target unit prediction.

(3) **XM Transformer with Sparsity.** Sparse modeling, in particular Mixture-of-Experts (MoE), has been widely studied in multilingual machine translation. MoE increases the number of parameters without sacrificing computation efficiency.

**GShard.** GShard is a sparse scaling technique proposed in (Lepikhin et al.). We replace every other Transformer layer with an MoE layer. FFN modules in an MoE transformer layer are shared across experts. A learnable gating function routes input tokens to different experts (NLLB Team et al., 2022). We apply GShard architecture on the decoder of XM Transformer, and expert weights are all initialized with the pretrained unit mBART.

## 6.1 Slavic-to-English Translation

The six Slavic languages include Czech (cs), Croatian (hr), Lituanian (lt), Polish (pl), Slovak (sk), and Slovenian (sl). In the multilingual setting, all mined data into English are combined from each

<table><caption>Table 7: Average BLEU of Slavic-to-English models in EP/VP and FLEURS (FL) domains.</caption><thead><tr><th rowspan="2"></th><th colspan="4">Bilingual</th><th colspan="4">Multilingual</th></tr><tr><th>EP/VP</th><th>FL</th><th>EP/VP</th><th>FL</th><th>EP/VP</th><th>FL</th><th>EP/VP</th><th>FL</th></tr></thead><tbody><tr><th>Textless</th><td>70M</td><td></td><td>260M</td><td></td><td>70M</td><td></td><td>260M</td><td></td></tr><tr><th>Avg.</th><td>14.3</td><td>5.1</td><td>16.8</td><td>6.5</td><td>14.1</td><td>2.5</td><td>22.4</td><td>11.2</td></tr><tr><th>XM</th><td>Dense(1.2B)</td><td></td><td></td><td></td><td>Dense (1.2B)</td><td></td><td>GShard (4.3B)</td><td></td></tr><tr><th>Avg.</th><td>18.1</td><td>10.1</td><td></td><td></td><td>26.0</td><td>15.2</td><td>27.0</td><td>15.5</td></tr></tbody></table>

Slavic language as the train set.

We summarize ASR BLEU scores of different models averaged over six Slavic-to-English directions in Table 7. For completeness, we report BLEU of each direction in Appendix C. As is shown, Textless model benefits from the parameter increase to 260M, and multilingual training further brings BLEU gains of 5.6 and 4.7 in EP/VP and FLEURS. We tried larger models than 260M but didn't see more gains.

Comparing against bilingual Textless model (70M), bilingual XM Transformer achieves +3.8 BLEU in EP/VP and +5.0 BLEU in FLEURS. Multilingual training further improves dense XM Transformer by 7.9 and 5.1 BLEU. GShard with 64 experts brings +1.0 BLEU over dense XM Transformer to EP/VP, and +0.3 BLEU to FLEURS. Overall the best Slavic-to-English translation is achieved by XM Transformer with GShard trained in multilingual setting. This demonstrates that multilinguality, pre-training and model sparsity are of help to speech-to-speech translation.
<table><caption>Table 8: BLEU of All-to-English multilingual models across FLEURS (FL) and EP/VP domains (for EP/VP column, underlined scores are on EPST data, and others on VoxPopuli data).</caption><thead><tr><th rowspan="2"></th><th colspan="2">Dense (1.2B)</th><th colspan="2">GShard (4.3B)</th></tr><tr><th>EP/VP</th><th>FL</th><th>EP/VP</th><th>FL</th></tr></thead><tbody><tr><td>cs</td><td>29.9</td><td>18.7</td><td>30.9</td><td>18.2</td></tr><tr><td>de</td><td><u>18.8</u></td><td>19.0</td><td><u>19.3</u></td><td>20.3</td></tr><tr><td>es</td><td><u>22.8</u></td><td>15.2</td><td><u>23.3</u></td><td>15.9</td></tr><tr><td>et</td><td>-</td><td>16.7</td><td>-</td><td>16.7</td></tr><tr><td>fi</td><td>26.8</td><td>14.1</td><td>28.2</td><td>14.0</td></tr><tr><td>fr</td><td><u>23.5</u></td><td>18.3</td><td><u>24.1</u></td><td>18.9</td></tr><tr><td>hr</td><td>-</td><td>16.6</td><td>-</td><td>16.8</td></tr><tr><td>hu</td><td>20.2</td><td>12.0</td><td>21.3</td><td>12.5</td></tr><tr><td>it</td><td>36.3</td><td>16.2</td><td>37.8</td><td>14.9</td></tr><tr><td>lt</td><td>21.9</td><td>9.8</td><td>23.8</td><td>10.3</td></tr><tr><td>nl</td><td><u>21.4</u></td><td>16.4</td><td><u>22.1</u></td><td>17.3</td></tr><tr><td>pl</td><td><u>21.2</u></td><td>12.4</td><td><u>21.3</u></td><td>13.4</td></tr><tr><td>pt</td><td><u>23.8</u></td><td>21.8</td><td><u>24.2</u></td><td>22.3</td></tr><tr><td>ro</td><td><u>25.1</u></td><td>19.7</td><td><u>25.0</u></td><td>19.8</td></tr><tr><td>sk</td><td>30.8</td><td>19.6</td><td>32.2</td><td>18.2</td></tr><tr><td>sl</td><td>28.3</td><td>13.7</td><td>29.9</td><td>13.7</td></tr><tr><td>avg</td><td>25.1</td><td>16.3</td><td>26.0</td><td>16.5</td></tr></tbody></table>

## 6.2 All-to-English Translation

We move forward to a larger-scale multilinguality by extending from Slavic language family to all languages in SpeechMatrix. We adopt the best models in Slavic-to-English translation, i.e., multilingual XM Transformer with both dense and sparse architectures.

**Results.** Compared with XM Transformer (1.2B) dense model, MoE-GShard64 (4.3B) with the same forward computation time brings gains of +0.9 and +0.2 BLEU to EP/VP and FLEURS respectively. Similar to our findings in Slavic-to-English setting, increasing the capacity with sparse modeling benefits in-domain (EP/VP) more than out-of-domain FLEURS test set.

Given sparse architecture of XM Transformer with GShard, all-to-English model shows +0.6 and -0.4 BLEU difference compared with Slavic-to-English model on EP/VP and FLEURS respectively, averaged over Slavic languages. Multilingual sparse model benefits from the additional in-domain data in other languages when evaluated in EP/VP domain, while sees performance degradation in out-of-domain data.

## 7 Limitations and Risks

**Limitations.** The HuBERT model quality is critical to speech-to-speech translation performance, as its extracted units are used by both speech-to-unit model and vocoder. We have not explored the optimal strategy of multilingual HuBERT training. One research question is how to choose a group of languages so that a multilingual HuBERT model could be well trained. For example, it is arguable whether Lithuanian (lt) should be included in Slavic or Uralic family. Other questions could be whether a larger HuBERT with more model capacity should be used and how we should deal with language imbalance in multilingual training.

We provide benchmark results of bilingual speech translation with mined data selected by heuristics. One of our future directions is to come up with a better strategy of mined data selection to improve translation performance and training efficiency.

As mentioned in our results analysis, the reported BLEU scores are heavily dependent on the ASR quality, which may not reflect the speech translation performance accurately. Future directions could be improving ASR quality or exploring other evaluation metrics without reliance on ASR models.

**Potential Risks.** As a technology used for speech generation, the presented speech translation models or the translation models that will be trained with SpeechMatrix dataset might have systemic bias or produce inappropriate outputs.

## 8 Conclusion

In this paper, we introduce a large-scale multilingual speech-to-speech corpus mined from VoxPopuli. It is the largest resource of speech alignments with a coverage of 17 languages. We perform an extensive evaluation of the mined parallel speech, showing good quality of the speech alignments. Multilingual speech-to-speech models can be efficiently trained on this corpus and we suggest different methods, such as sparse scaling using Mixture-of-Experts, to further boost translation performance in the multilingual setting.

## References

Sadaf Abdul-Rauf and Holger Schwenk. 2009. On the Use of Comparable Corpora to Improve SMT performance. In EACL, pages 16–23.
Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis M. Tyers, and Gregor Weber. 2020. Common voice: A massively-multilingual speech corpus. In *Proceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, Marseille, France, May 11-16, 2020*, pages 4218–4222. European Language Resources Association.

Mikel Artetxe and Holger Schwenk. 2018. Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings. https://arxiv.org/abs/1811.01136.

Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, Alexei Baevski, Alexis Conneau, and Michael Auli. 2021. XLS-R: self-supervised cross-lingual speech representation learning at scale. CoRR, abs/2111.09296.

Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. In *Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual*.

Marcely Zanon Boito, William Havard, Mahault Gartnerin, Éric Le Ferrand, and Laurent Besacier. 2020. Mass: A large and clean multilingual corpus of sentence-aligned spoken utterances extracted from the bible. In *Proceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, Marseille, France, May 11-16, 2020*, pages 6486–6493. European Language Resources Association.

Houda Bouamor and Hassan Sajjad. 2018. H2@BUCC18: Parallel Sentence Extraction from Comparable Corpora Using Multilingual Sentence Embeddings. In *BUCC*.

Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. 2022. FLEURS: few-shot learning evaluation of universal representations of speech. CoRR, abs/2205.12446.

Alexandre Defossez, Gabriel Synnaeve, and Yossi Adi. 2020. Real time speech enhancement in the waveform domain. In *Interspeech*.

Mattia A. Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2019. MuST-C: a Multilingual Speech Translation Corpus. In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)*, pages 2012–2017, Minneapolis, Minnesota. Association for Computational Linguistics.

Quoc Truong Do, Sakriani Sakti, Graham Neubig, Tomoki Toda, and Satoshi Nakamura. 2015. Improving translation of emphasis with pause prediction in speech-to-speech translation systems. In *Proceedings of the 12th International Workshop on Spoken Language Translation: Papers, IWSLT 2015, Da Nang, Vietnam, December 3-4, 2015*.

Paul-Ambroise Duquenne, Hongyu Gong, Benoît Sagot, and Holger Schwenk. 2022. T-modules: Translation modules for zero-shot cross-modal machine translation. In *EMNLP*.

Paul-Ambroise Duquenne, Hongyu Gong, and Holger Schwenk. 2021. Multimodal and multilingual embeddings for large-scale speech mining. *Advances in Neural Information Processing Systems*, 34:15748–15761.

Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2020. Language-agnostic bert sentence embedding. *arXiv preprint arXiv:2007.01852*.

Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc'Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022. The flores-101 evaluation benchmark for low-resource and multilingual machine translation. *Trans. Assoc. Comput. Linguistics*, 10:522–538.

Kevin Heffernan, Onur Çelebi, and Holger Schwenk. 2022. Bitext mining using distilled sentence representations for low-resource languages. In *EMNLP*.

Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. *IEEE ACM Trans. Audio Speech Lang. Process.*, 29:3451–3460.

Javier Iranzo-Sánchez, Joan Albert Silvestre-Cerdà, Javier Jorge, Nahuel Roselló, Adrià Giménez, Albert Sanchís, Jorge Civera, and Alfons Juan. 2020. Europarl-st: A multilingual corpus for speech translation of parliamentary debates. In *2020 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2020, Barcelona, Spain, May 4-8, 2020*, pages 8229–8233. IEEE.

Ye Jia, Yifan Ding, Ankur Bapna, Colin Cherry, Yu Zhang, Alexis Conneau, and Nobuyuki Morioka. 2022a. Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translation. CoRR, abs/2203.13339.

Ye Jia, Michelle Tadmor Ramanovich, Tal Remez, and Roi Pomerantz. 2022b. Translatotron 2: High-quality direct speech-to-speech translation with voice preservation. In *International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA*, volume 162 of *Proceedings of Machine Learning Research*, pages 10120–10134. PMLR.
Ye Jia, Michelle Tadmor Ramanovich, Quan Wang, and Heiga Zen. 2022c. CVSS corpus and massively multilingual speech-to-speech translation. CoRR, abs/2201.03713.

Ye Jia, Ron J. Weiss, Fadi Biadsy, Wolfgang Macherey, Melvin Johnson, Zhifeng Chen, and Yonghui Wu. 2019. Direct speech-to-speech translation with a sequence-to-sequence model. In Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019, pages 1123–1127. ISCA.

Sameer Khurana, Antoine Laurent, and James Glass. 2022. Samu-xlsr: Semantically-aligned multimodal utterance-level cross-lingual speech representation. arXiv preprint arXiv:2205.08180.

Ann Lee, Peng-Jen Chen, Changhan Wang, Jiatao Gu, Sravya Popuri, Xutai Ma, Adam Polyak, Yossi Adi, Qing He, Yun Tang, Juan Pino, and Wei-Ning Hsu. 2022a. Direct speech-to-speech translation with discrete units. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3327–3339. Association for Computational Linguistics.

Ann Lee, Hongyu Gong, Paul-Ambroise Duquenne, Holger Schwenk, Peng-Jen Chen, Changhan Wang, Sravya Popuri, Yossi Adi, Juan Pino, Jiatao Gu, and Wei-Ning Hsu. 2022b. Textless speech-to-speech translation on real data. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pages 860–872. Association for Computational Linguistics.

Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021.

Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. 2021. Base layers: Simplifying training of large, sparse models. In International Conference on Machine Learning, pages 6265–6274. PMLR.

Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742.

Dragos Stefan Munteanu and Daniel Marcu. 2005. Improving Machine Translation Performance by Exploiting Non-Parallel Corpora. Computational Linguistics, 31(4):477–504.

Satoshi Nakamura, Konstantin Markov, Hiromi Nakaiwa, Gen-ichiro Kikui, Hisashi Kawai, Takatoshi Jitsuhiro, Jinsong Zhang, Hirofumi Yamamoto, Eiichiro Sumita, and Seiichi Yamamoto. 2006. The ATR multilingual speech-to-speech translation system. IEEE Trans. Speech Audio Process., 14(2):365–376.

Xuan-Phi Nguyen, Sravya Popuri, Changhan Wang, Yun Tang, Ilia Kulikov, and Hongyu Gong. 2022. Improving speech-to-speech translation through unlabeled text. CoRR, abs/2210.14514.

NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shanon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human-centered machine translation.

Kyubyong Park and Thomas Mulc. 2019. CSS10: A collection of single speaker speech datasets for 10 languages. In Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019, pages 1566–1570. ISCA.

Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Abdelrahman Mohamed, and Emmanuel Dupoux. 2021. Speech Resynthesis from Discrete Disentangled Self-Supervised Representations. In Proc. Interspeech 2021.

Sravya Popuri, Peng-Jen Chen, Changhan Wang, Juan Pino, Yossi Adi, Jiatao Gu, Wei-Ning Hsu, and Ann Lee. 2022. Enhanced direct speech-to-speech translation using self-supervised pre-training and data augmentation. pages 5195–5199.

Matt Post, Gaurav Kumar, Adam Lopez, Damianos Karakos, Chris Callison-Burch, and Sanjeev Khudanpur. 2014. Fisher and callhome spanish-english speech translation. LDC2014T23. Web Download. Philadelphia: Linguistic Data Consortium.

Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In EMNLP, pages 4512–4525.

Philip Resnik. 1999. Mining the Web for Bilingual Text. In ACL.

Elizabeth Salesky, Matthew Wiesner, Jacob Bremerman, Roldano Cattoni, Matteo Negri, Marco Turchi,
Douglas W. Oard, and Matt Post. 2021. The multilingual tedx corpus for speech recognition and translation. In *Interspeech 2021, 22nd Annual Conference of the International Speech Communication Association*, Brno, Czechia, 30 August - 3 September 2021, pages 3655–3659. ISCA.

Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, Armand Joulin, and Angela Fan. 2021a. CCMatrix: Mining billions of high-quality parallel sentences on the web. In *ACL*, pages 6490–6500.

Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, Armand Joulin, and Angela Fan. 2021b. Ccmatrix: Mining billions of high-quality parallel sentences on the web. In *Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021*, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 6490–6500. Association for Computational Linguistics.

Silero-Team. 2021. Silero vad: pre-trained enterprise-grade voice activity detector (vad), number detector and language classifier. https://github.com/snakers4/silero-vad.

Changhan Wang, Juan Pino, Anne Wu, and Jiatao Gu. 2020. Covost: A diverse multilingual speech-to-text translation corpus. In *Proceedings of the 12th Language Resources and Evaluation Conference*, pages 4197–4203.

Changhan Wang, Morgane Rivière, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Miguel Pino, and Emmanuel Dupoux. 2021a. Voxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. In *Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021*, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 993–1003. Association for Computational Linguistics.

Changhan Wang, Anne Wu, Jiatao Gu, and Juan Pino. 2021b. Covost 2 and massively multilingual speech translation. In *Interspeech*, pages 2247–2251.

Yinfei Yang, Gustavo Hernandez Abrego, Steve Yuan, Qinlan Shen Mandy Guo, Daniel Cer, Brian Strope Yun-hsuan Sun and, and Ray Kurzweil. 2019. Improving multilingual sentence embedding using bidirectional dual encoder with additive margin softmax. In *IJCAI*, pages 5370–5378.

Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. 2022. Designing effective sparse expert models. *arXiv preprint arXiv:2202.08906*.

# A Speech Encoder

## A.1 Similarity search on CoVoST

We compared our similarity search results with previous work (Duquenne et al., 2021) in Table 9. We notice that our new speech encoders have lower error rates compared to previous work.

<table><caption>Table 9: Similarity search error rates (in %) on CoVoST 2 test set.</caption><thead><tr><th>Audio vs. en translations</th><th>de</th><th>es</th><th>fr</th></tr></thead><tbody><tr><td>Previous work</td><td>3.36</td><td>1.66</td><td>2.05</td></tr><tr><td>This work</td><td><strong>3.27</strong></td><td><strong>1.26</strong></td><td><strong>1.55</strong></td></tr></tbody></table>

We also provide similarity search of audios against written translations or transcriptions on CoVoST 2 test set for other languages covered by our speech encoders in Table 10, in order to evaluate cross-modal similarity search.

<table><caption>Table 10: Similarity search error rates (in %) on CoVoST 2 test set.</caption><thead><tr><th># test sentences</th><th>de</th><th>en</th><th>es</th><th>et</th><th>fr</th><th>it</th><th>nl</th><th>pt</th><th>sl</th></tr></thead><tbody><tr><td><strong>Audio</strong></td><td>14k</td><td>16k</td><td>13k</td><td>2k</td><td>15k</td><td>9k</td><td>2k</td><td>4k</td><td>0.4k</td></tr><tr><td>vs. transcriptions</td><td>1.4</td><td>2.9</td><td>0.4</td><td>0.1</td><td>0.5</td><td>0.5</td><td>1.0</td><td>1.1</td><td>1.7</td></tr><tr><td>vs. en translations</td><td>3.3</td><td>—</td><td>1.3</td><td>1.0</td><td>1.5</td><td>1.7</td><td>4.4</td><td>1.9</td><td>4.4</td></tr><tr><td><strong>Text transcription</strong></td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>vs. en translations</td><td>2.0</td><td>—</td><td>1.0</td><td>0.1</td><td>1.0</td><td>1.3</td><td>2.4</td><td>0.7</td><td>0.8</td></tr></tbody></table>

# B Bilingual Speech-to-Speech Translation

We describe experiment details of bilingual speech-to-speech translation.

## B.1 HuBERT

<table><caption>Table 11: Language families in VoxPopuli data.</caption><thead><tr><th>Family</th><th>Languages</th></tr></thead><tbody><tr><td>Romance</td><td>es, fr, it, pt, ro</td></tr><tr><td>Slavic</td><td>cs, pl, sk, sl, hr, lt</td></tr><tr><td>Germanic</td><td>en, de, nl</td></tr><tr><td>Uralic</td><td>fi, et, hu</td></tr></tbody></table>

We train a multilingual HuBERT model for each family on the collection of speech in each component language as shown in Table 11. We collect unlabeled VoxPopuli speech for all languages of the same family as the training data. The HuBERT model consists of 7 convolutional layers and 12 Transformer encoder layers. Each encoder layer has 12 attention heads, the embedding dimension of 768 and the forward dimension of 3072. Models are trained for 3 iterations, and in each iteration
<table><caption>Table 12: HuggingFace ASR models for each language.</caption><thead><tr><th>Lang</th><th>cs</th><th>de</th></tr></thead><tbody><tr><td>ASR</td><td>comodoro/wav2vec2-xls-r-300m-cs-250</td><td>jonatasgrosman/wav2vec2-xls-r-1b-german</td></tr><tr><td>Lang</td><td>et</td><td>fi</td></tr><tr><td>ASR</td><td>RASMUS/wav2vec2-xlsr-1b-et</td><td>jonatasgrosman/wav2vec2-large-xlsr-53-finnish</td></tr><tr><td>Lang</td><td>hr</td><td>hu</td></tr><tr><td>ASR</td><td>classla/wav2vec2-xls-r-parlaspeech-hr</td><td>jonatasgrosman/wav2vec2-large-xlsr-53-hungarian</td></tr><tr><td>Lang</td><td>it</td><td>lt</td></tr><tr><td>ASR</td><td>jonatasgrosman/wav2vec2-large-xlsr-53-italian</td><td>sammy786/wav2vec2-xlsr-lithuanian</td></tr><tr><td>Lang</td><td>nl</td><td>pl</td></tr><tr><td>ASR</td><td>jonatasgrosman/wav2vec2-xls-r-1b-dutch</td><td>jonatasgrosman/wav2vec2-xls-r-1b-polish</td></tr><tr><td>Lang</td><td>pt</td><td>ro</td></tr><tr><td>ASR</td><td>jonatasgrosman/wav2vec2-xls-r-1b-portuguese</td><td>gigant/romanian-wav2vec2</td></tr><tr><td>Lang</td><td>sk</td><td>sl</td></tr><tr><td>ASR</td><td>anuragshas/wav2vec2-xls-r-300m-sk-cv8-with-lm</td><td>anuragshas/wav2vec2-xls-r-300m-sl-cv8-with-lm</td></tr></tbody></table>

pseudo-labels are prepared as the training target for utterances. In the first iteration, the target labels are MFCC features. In the second iteration, we extract speech features from the 6-th layer of the trained HuBERT model and apply *k*-means clustering to derive a set of 500 labels. In the third iteration, speech features from the 9-th layer are clustered into 500 labels. Lastly after these three iterations, we try feature extraction from different layers including layer 10, 11 and 12 of trained HuBERT. As for feature clustering, we also try different numbers of clusters, 800, 1000 and 1200, to derive multiple sets of target units.

To choose the optimal setup, we launch a resynthesis evaluation to select the HuBERT layer to extract speech features and the number of *k*-means clusters. We train a vocoder on each set of target units, i.e., vocoder takes the units and synthesizes target speech. The synthesized speech is sent to off-the-shelf ASR models, and Word Error Rate (WER) is reported to measure the speech quality. The resynthesis experiments are discussed in subsection B.3. The optimal HuBERT layer and label size is selected if their corresponding vocoder achieves the lowest WER.

## B.2 ASR models

We use ASR models publicly released on Hugging-Face to transcribe the generated speech in order to calculate WER or BLEU scores in comparison with ground truth texts. ASR models used in our evaluation are listed in Table 12.

## B.3 Vocoder

**Data preprocessing.** We applied a denoiser<fnref n="4" /> (Denossez et al., 2020) to the speech of VoxPopuli and Common Voice as the speech preprocessing to increase signal-to-noise ratio (SNR) given that they are noisier than CSS10 audios. Then we prepare vocoder labels with HuBERT models generating *k*-means cluster labels for each utterance. Single-speaker vocoders are trained in CSS10, and languages from VoxPopuli and Common Voice have multi-speaker vocoders where speaker embeddings are learned. During inference, we select the speaker with the longest speech duration to synthesize speech from predicted unit sequences, who has the most data for the vocoder to learn good speaker embeddings.

**Vocoder training and evaluation.** Vocoders are trained to synthesize speech from a given sequence of units. The train sets are speech data from CSS10, VoxPopuli and Common Voice. As mentioned before, units are derived from HuBERT models for these speech. Table 13 summarizes WER of ASR models, which reflects the transcription quality in each language. Besides, we report the training data, vocoder WER of synthesized speech from vocoders, and here we include the vocoder results obtained from the optimal HuBERT layer and *k*-means cluster size. Layer 11 is the best HuBERT layer for feature extraction in all languages, and most languages have the best *k*-means size of 1000 except Italian (it) whose best label size is 800.

As shown in Table 13, ASR models are of good quality for high-resource languages such as de, fi

<footnote n="4">⁴https://github.com/facebookresearch/denoiser</footnote>
<table><caption>Table 13: Benchmark results of ASR models and vocoder resynthesis.</caption><thead><tr><th>Lang</th><th>Data</th><th>ASR WER</th><th>HuBERT</th><th>Vocoder WER</th><th>Lang</th><th>Data</th><th>ASR WER</th><th>HuBERT</th><th>Vocoder WER</th></tr></thead><tbody><tr><td>de</td><td>CSS10</td><td>0.10</td><td>Germanic HuBERT layer 11, km 1000</td><td>0.16</td><td>nl</td><td>CSS10</td><td>0.19</td><td>Germanic HuBERT layer 11, km 1000</td><td>0.27</td></tr><tr><td>fi</td><td>CSS10</td><td>0.02</td><td>Uralic HuBERT layer 11, km 1000</td><td>0.15</td><td>hu</td><td>CSS10</td><td>0.21</td><td>Uralic HuBERT layer 11, km 1000</td><td>0.21</td></tr><tr><td>et</td><td>Common Voice</td><td>0.14</td><td>Uralic HuBERT layer 11, km 1000</td><td>0.44</td><td>it</td><td>VoxPopuli</td><td>0.23</td><td>Uralic HuBERT layer 11, km 800</td><td>0.27</td></tr><tr><td>pt</td><td>Common Voice</td><td>0.06</td><td>Uralic HuBERT layer 11, km 1000</td><td>0.31</td><td>ro</td><td>VoxPopuli</td><td>0.42</td><td>Uralic HuBERT layer 11, km 1000</td><td>0.50</td></tr><tr><td>cs</td><td>VoxPopuli</td><td>0.15</td><td>Slavic HuBERT layer 11, km 1000</td><td>0.23</td><td>pl</td><td>VoxPopuli</td><td>0.14</td><td>Slavic HuBERT layer 11, km 1000</td><td>0.23</td></tr><tr><td>hr</td><td>VoxPopuli</td><td>0.21</td><td>Slavic HuBERT layer 11, km 1000</td><td>0.29</td><td>lt</td><td>VoxPopuli</td><td>0.38</td><td>Slavic HuBERT layer 11, km 1000</td><td>0.57</td></tr><tr><td>sk</td><td>VoxPopuli</td><td>0.28</td><td>Slavic HuBERT layer 11, km 1000</td><td>0.41</td><td>sl</td><td>VoxPopuli</td><td>0.37</td><td>Slavic HuBERT layer 11, km 1000</td><td>0.46</td></tr></tbody></table>

and pt, while suffering from high error rates in languages such as ro, lt and sl. It is expected to have higher vocoder WER than ASR WER since the former is for synthesized speech. By measuring the gap between the two error rates, we can tell how good a vocoder is and also infer the quality of HuBERT units. For et, pt and lt, the gaps are obviously larger than other languages. It not surprising since we do not have much good-quality vocoder data for these languages. For example, there is only around 10-hour noisy speech from Common Voice for et and pt vocoder training.

## B.4 Training

**Textless model.** A Textless model consists of a speech encoder with 2 convolution layers and 12 Transformer encoder layers. Transformer layer has the embedding dimension of 512 and the forward dimension of 2048. It has two unit decoders with 6 and 2 Transformer decoder layers for target and source unit prediction respectively. The target unit decoder has the embedding dimension of 512 and the forward dimension of 2048, and the source unit decoder's dimensions are 256 and 2048.

**Hyperparameters.** We tried learning rates of 0.0003 and 0.0005, and dropout rates of 0.1 and 0.3. The best setup is a learning rate of 0.0005 and a dropout of 0.3 for bilingual Textless model training. Bilingual models are trained with a batch of 20k tokens for 400k steps. A label smoothing weight of 0.2 is applied to the cross-entropy loss.

As for decoding of speech-to-unit models, we set the beam size of 10 in all bilingual and multilingual experiments.

**Mined data selection.** We performed an analysis of translation performance varying with thresholds from 1.06 to 1.09 on three language pairs: es-en, ro-en and hr-en. Figure 3 shows the thresh-

<figure><img src="image_5.png" /><figcaption>Figure 3: Bilingual S2S BLEU by mined data at different thresholds.</figcaption></figure>

old, the corresponding speech data size and BLEU score.

For low-resource directions such as hr-en, it is best to include all the mined data. For high- and medium-resource directions, es-en and ro-en, the optimal amount of mined data is around 1k hours and it does not bring further gains to go beyond that data size. Given these observations, we choose the highest threshold that keeps the source speech duration in mined data more than 1k hour for each direction. For example, we use a threshold of 1.09 for es-en and of 1.06 for hr-en.

**Computation.** Each bilingual model is trained on 16 A100 GPUs for 3 days on average.

## C Multilingual Speech-to-Speech Translation

We provide details of models and experiment setups in multilingual speech-to-speech translation.

### C.1 Slavic-to-English Translation

**Textless model.** Textless model (260M) has a speech encoder with 4 convolution layers and 12
Transformer encoder layers with the embedding dimension of 1024 and the forward dimension of 4096. It has two unit decoders with 6 and 2 Transformer decoder layers for target and source unit prediction respectively. The target unit decoder has the embedding dimension of 1024 and the forward dimension of 4096, and the source unit decoder's dimensions are 256 and 2048.

For the Textless model (424M), its speech encoder contains 6 convolution layers and 16 Transformer encoder layers with the embedding dimension of 1024 and the forward dimension of 4096. It has two unit decoders with 12 and 2 Transformer decoder layers for target and source unit prediction respectively. The target unit decoder has the embedding dimension of 1024 and the forward dimension of 4096, and the source unit decoder's dimensions are 256 and 2048.

**XM Transformer.** XM Transformer (1.2B) is initialized from XLS-R encoder with 7 convolution layers and 48 Transformer encoder layers with the embedding dimension of 1280 and the forward dimension of 5120. Its unit decoder is initialized from a pre-trained mbart-style decoder with 12 layers, embedding dimension of 1024 and forward dimension of 4096.

**Hyperparameters.** For Textless model, we reuse a learning rate of 0.0005, a dropout of 0.3

and a label smoothing weight of 0.2 for Slavic-to-English training. The 70M model has 20k tokens in one batch. The 260M model has batch tokens of 6k and a update frequency of 4. The 424M model has tokens of 4000 and a update frequency of 6. For XM Transformer model, we use a learning rate of 0.0001, a dropout of 0.1 and a label smoothing weight of 0.2. In a batch, token sizes of 1500 and 9000 with update frequency of 15 and 2 are used for V100 and A100 training respectively.

**Results.** We first extend Textless model from the bilingual to multilingual setting. Translation results are presented for Textless models with different parameter sizes in Table 14. Multilingual Textless model works best with 260M parameters. Compared with its bilingual counterparts, an average gain of 5.6 BLEU is achieved in EP/VP and the gain of 4.7 BLEU in FLEURS.

With the Textless model size fixed as 70M, multilingual training hurts the performance of most languages compared with bilingual training. This is due to the insufficient model capacity, and the language interference is reflected by an average of -2.6 BLEU in FLEURS. We increase model parameters to 260M in both bilingual and multilingual settings. With a larger model capacity, bilingual models achieve gains in high-resource languages including cs, pl and sk, while suffering from performance loss in low-resource directions such as hr, lt and sl.

Given model sizes of 260M, we observe consistent gains of multilingual models over the bilingual models across different language directions and test domains. An average gain of 5.6 BLEU is achieved in EP/VP and the gain of 4.7 BLEU in FLEURS. It demonstrates the positive transfer enabled by multilingual training. As the multilingual model size continues to increase to 424M, we don't observe further gains likely due to the bottleneck of training data amount.

XM Transformer leveraging pre-trained modules is also trained on Slavic-to-English data. Pre-training is shown to be beneficial, and results are reported in Table 15. Comparing against bilingual Textless model (70M), bilingual XM Transformer outperforms it in all directions except lt-en. The gain in EP/VP is 3.8 BLEU on average, and a larger gain of 5.0 BLEU is achieved in FLEURS. Multilingual training brings further gains to XM Transformer with +7.9 and +5.1 BLEU over bilingual training in EP/VP and FLEURS test set

<table><caption>Table 14: BLEU of Slavic-to-English multilingual Textless model across FLEURS (FL) and EP/VP domains (for EP/VP column, underlined scores are on EPST data, and others on VoxPopuli data).</caption><thead><tr><th rowspan="3"></th><th colspan="6">Bilingual</th><th colspan="6">Multilingual</th></tr><tr><th colspan="2">70M</th><th colspan="2">260M</th><th colspan="2">70M</th><th colspan="2">260M</th><th colspan="2">424M</th></tr><tr><th>EP/VP</th><th>FL</th><th>EP/VP</th><th>FL</th><th>EP/VP</th><th>FL</th><th>EP/VP</th><th>FL</th><th>EP/VP</th><th>FL</th></tr></thead><tbody><tr><th>cs</th><td>22.7</td><td>4.2</td><td>24.7</td><td>11.2</td><td>19.7</td><td>2.3</td><td>27.5</td><td>13.7</td><td>25.3</td><td>10.2</td></tr><tr><th>hr</th><td>-</td><td>7.7</td><td>-</td><td>4.6</td><td>-</td><td>3.1</td><td>-</td><td>12.8</td><td>-</td><td>9.2</td></tr><tr><th>lt</th><td>3.1</td><td>0.9</td><td>0.2</td><td>0.0</td><td>2.8</td><td>0.3</td><td>14.7</td><td>4.8</td><td>10.7</td><td>3.3</td></tr><tr><th>pl</th><td>4.9</td><td>4.9</td><td>17.6</td><td>7.7</td><td>14.4</td><td>1.9</td><td>19.9</td><td>9.5</td><td>16.4</td><td>6.9</td></tr><tr><th>sk</th><td>21.4</td><td>5.5</td><td>24.4</td><td>11.0</td><td>18.9</td><td>4.1</td><td>27.2</td><td>15.4</td><td>24.9</td><td>11.1</td></tr><tr><th>sl</th><td>19.5</td><td>7.3</td><td>16.9</td><td>4.7</td><td>14.6</td><td>3.1</td><td>22.9</td><td>10.7</td><td>21.0</td><td>7.6</td></tr><tr><th>avg</th><td>14.3</td><td>5.1</td><td>16.8</td><td>6.5</td><td>14.1</td><td>2.5</td><td>22.4</td><td>11.2</td><td>19.7</td><td>8.1</td></tr></tbody></table>

<table><caption>Table 15: BLEU of Slavic-to-English multilingual XM Transformer models across FLEURS (FL) and EP/VP domains (for EP/VP column, underlined scores are on EPST data, and others on VoxPopuli data).</caption><thead><tr><th rowspan="2"></th><th colspan="2">Bilingual (1.2B)</th><th colspan="2">Multiling. Dense (1.2B)</th><th colspan="2">Multiling. GShard (4.3B)</th></tr><tr><th>EP/VP</th><th>FL</th><th>EP/VP</th><th>FL</th><th>EP/VP</th><th>FL</th></tr></thead><tbody><tr><th>cs</th><td>28.3</td><td>17.8</td><td>29.7</td><td>18.2</td><td>30.6</td><td>19.3</td></tr><tr><th>hr</th><td>-</td><td>12.1</td><td>-</td><td>17.1</td><td>-</td><td>17.6</td></tr><tr><th>lt</th><td>0.0</td><td>0.0</td><td>20.9</td><td>9.6</td><td>22.2</td><td>10.2</td></tr><tr><th>pl</th><td>17.4</td><td>7.4</td><td>21.1</td><td>12.9</td><td>21.4</td><td>12.6</td></tr><tr><th>sk</th><td>24.7</td><td>14.5</td><td>30.8</td><td>19.3</td><td>31.8</td><td>20.0</td></tr><tr><th>sl</th><td>20.1</td><td>8.5</td><td>27.4</td><td>14.0</td><td>29.1</td><td>13.0</td></tr><tr><th>avg</th><td>18.1</td><td>10.1</td><td>26.0</td><td>15.2</td><td>27.0</td><td>15.5</td></tr></tbody></table>
respectively.

Comparing against dense XM Transformer, GShard with 64 experts has 1.0 BLEU gains on average over 5 directions on EP/VP, and +0.3 BLEU gains for FLEURS. We believe that it is due to a phenomena mentioned in (Zoph et al., 2022), i.e., MoE specializes in multilingual settings but not by language. GShard in our setting brings larger improvements to in-domain test sets.

**Computation.** Textless models used 32 A100 GPUs, the 70M model was trained for 3 days, the 260M model was for 5 days, and the 424M model was for 6 days. It took 2 days to train XM Transformer on 32 A100 GPUs for Slavic-to-English translation.

## C.2 All-to-English Translation

In this work, we experiment with two variants of sparse modeling, GShard and Base Layer.

<table><caption>Table 16: BLEU of All-to-English multilingual models across FLEURS (FL) and EP/VP domains (for EP/VP column, underlined scores are on EPST data, and others on VoxPopuli data).</caption><thead><tr><th rowspan="2"></th><th colspan="2">Dense (1.2B)</th><th colspan="2">MoE-GShard64 (4.3B)</th><th colspan="2">Base Layer (1.7B)</th></tr><tr><th>EP/VP</th><th>FL</th><th>EP/VP</th><th>FL</th><th>EP/VP</th><th>FL</th></tr></thead><tbody><tr><td>cs</td><td>29.9</td><td>18.7</td><td>30.9</td><td>18.2</td><td>29.9</td><td>17.3</td></tr><tr><td>de</td><td>18.8</td><td>19.0</td><td>19.3</td><td>20.3</td><td>19.4</td><td>19.5</td></tr><tr><td>es</td><td>22.8</td><td>15.2</td><td>23.3</td><td>15.9</td><td>22.9</td><td>14.9</td></tr><tr><td>et</td><td>-</td><td>16.7</td><td>-</td><td>16.7</td><td>-</td><td>16.4</td></tr><tr><td>fi</td><td>26.8</td><td>14.1</td><td>28.2</td><td>14.0</td><td>28.5</td><td>13.9</td></tr><tr><td>fr</td><td>23.5</td><td>18.3</td><td>24.1</td><td>18.9</td><td>23.4</td><td>18.2</td></tr><tr><td>hr</td><td>-</td><td>16.6</td><td>-</td><td>16.8</td><td>-</td><td>16.3</td></tr><tr><td>hu</td><td>20.2</td><td>12.0</td><td>21.3</td><td>12.5</td><td>20.5</td><td>12.1</td></tr><tr><td>it</td><td>36.3</td><td>16.2</td><td>37.8</td><td>14.9</td><td>37.4</td><td>14.0</td></tr><tr><td>lt</td><td>21.9</td><td>9.8</td><td>23.8</td><td>10.3</td><td>23.4</td><td>10.0</td></tr><tr><td>nl</td><td>21.4</td><td>16.4</td><td>22.1</td><td>17.3</td><td>21.5</td><td>16.6</td></tr><tr><td>pl</td><td>21.2</td><td>12.4</td><td>21.3</td><td>13.4</td><td>20.9</td><td>12.5</td></tr><tr><td>pt</td><td>23.8</td><td>21.8</td><td>24.2</td><td>22.3</td><td>23.8</td><td>21.1</td></tr><tr><td>ro</td><td>25.1</td><td>19.7</td><td>25.0</td><td>19.8</td><td>25.3</td><td>19.0</td></tr><tr><td>sk</td><td>30.8</td><td>19.6</td><td>32.2</td><td>18.2</td><td>31.5</td><td>18.4</td></tr><tr><td>sl</td><td>28.3</td><td>13.7</td><td>29.9</td><td>13.7</td><td>28.8</td><td>13.5</td></tr><tr><td>avg</td><td>25.1</td><td>16.3</td><td>26.0</td><td>16.5</td><td>25.5</td><td>15.9</td></tr></tbody></table>

**XM Transformer-GShard.** XM Transformer (1.2B) is initialized with the same XLS-R encoder and unit decoder used in Slavic-to-English experiments. On the decoder side of XM Transformer-GShard, each expert is initialized with the same unit decoder. We set MoE frequency as 2, i.e., every other Transformer layer is an MoE layer.

**XM Transformer-Base Layer.** For our XM Transformer with Base Layer sparsity (1.7B), the encoder is initialized with the same XLS-R encoder, and the dense layers of the decoder is initialized with the same unit decoder as GShard. We

add an additional Base Layer which is randomly initialized as the 7th layer of decoder. There is one expert in each GPU and we used 64 GPUs in our experiments, which means we have 64 Base Layer experts in total.

The sparse variant, Base Layer (1.7B) performs comparably to the dense XM Transformer, with an average of +0.4 BLEU in EP/VP test sets and -0.4 BLEU in FLEURS. The sparsity in Base Layer does not bring obvious gains to all-to-English translation. This is likely because we only add one Base Layer to the decoder with a small expert size. The number of increased model parameters is only 0.5B in Base Layer, while it is 3.1B in GShard. As suggested by (Lewis et al., 2021), the Base Layer performance might improve with more GPUs and a larger expert size.

**Hyperparameters.** For dense XM Transformer, hyperparameters are the same as that for Slavic-to-English. GShard also shares the same set of hyperparameters. As for expert-specific parameters, we use 64 experts with each running on a single GPU with the frequency of 2 so that every other Transformer decoder layer becomes an MoE layer. The capacity token fraction is set as 0.5 so that if more than half of tokens in a sample get routed to one expert, extra tokens would overflow and get dropped.

**Computation.** It took 3 days to train dense XM Transformer for all-to-English with 32 A100 GPUs. It took 5 days to train the GShard counterpart with 64 A100 GPUs.

# D License and Terms of Scientific Artifacts

## D.1 Third-Party Artifacts

**Data.** Common Voice is released under CC0 license, VoxPopuli and CoVoST 2 data are also under CC0 license. As for EuroParl, it is released under a Creative Commons license. The multilingual TEDx corpus is released under a CC BY-NC-ND 4.0 license. FLEURS dataset is under Creative Commons license (CC-BY-4.0). These datasets are publicly accessible and freely downloadable for research purposes.

**Models.** XLS-R model used for the speech encoder initialization is open sourced under Apache-2.0 license. Text LASER used as the teacher model in training is released under BSD license. ASR models available on HuggingFace are released under Apache-2.0 license. These models are publicly
available.

**Code.** The implementations of Textless model, XM Transformer, HuBERT and Vocoder are open sourced under MIT license.

## D.2 SpeechMatrix and translation models

The mined resource, SpeechMatrix, will be released under CC0 license, and the trained speech-to-speech translation models will be released under CC BY-NC 4.0. The data and models are intended for research purposes.