SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations
Year: 2023

Abstract:
We present SpeechMatrix, a large-scale multilingual corpus of speech-to-speech translations (S2ST) mined from real speech of European Parliament recordings. It contains speech alignments in 136 language pairs with a total of 418 thousand hours of speech. To evaluate the quality of this parallel speech, we train bilingual speech-to-speech translation models on mined data only and establish extensive baseline results on Europarl-ST, VoxPopuli and FLEURS test sets. Enabled by the multilinguality of SpeechMatrix, we also explore multilingual speech-to-speech translation, a topic which was addressed by few other works. We also demonstrate that model pre-training and sparse scaling using Mixture-of-Experts bring large gains to translation performance. We are open-sourcing the mined data, speech encoders used for mining, multilingual HuBERT models in four language families for target unit generation, language-specific vocoders for speech synthesis from discrete units, and S2S models trained and presented in this work.1

heading: 1 Introduction
text: Research has progressed in the area of speech-tospeech translation with the goal of seamless communication among people who speak different languages. Direct S2ST models attract increasing research interest, e.g. (Jia et al., 2019). Compared to conventional cascaded models, direct models do not rely on intermediate text representations which make them applicable to the translation of languages without a well-defined writing script. Moreover, direct S2ST have the advantage of higher training and inference efficiency (Lee et al., 2022a).
Despite the benefits of direct approaches, their training is faced with the major issue of data scarcity in parallel speech. Human labeled speech
∗Equal contributions 1https://github.com/facebookresearch/ fairseq/tree/ust/examples/speech_matrix
data is expensive to create, there are very few data resources providing speech alignments, and the data amount is quite limited. To mitigate the data scarcity, some works have leveraged multitask learning (Jia et al., 2019; Lee et al., 2022a), data augmentation with speech variations (Jia et al., 2019), or with synthesized speech (Jia et al., 2022a; Popuri et al., 2022; Nguyen et al., 2022). It is also shown useful to leverage knowledge transferred from pre-trained models (Lee et al., 2022b; Popuri et al., 2022) such as HuBERT (Hsu et al., 2021), wav2vec 2.0 (Baevski et al., 2020) and mBART (Liu et al., 2020).
Recently, Duquenne et al. (2021) is the first work to make speech mining efforts by learning a shared multilingual speech and text embedding space. Speech content is encoded by speech encoders into fixed-size representations which are then used for aligning speech and text across different languages. It demonstrates good empirical gains to train direct speech-to-text and speech-tospeech translation systems with the mined data (Duquenne et al., 2021; Lee et al., 2022b).
In this work, we trained speech encoders for 17 languages2 and mined speech-to-speech alignments for all possible language pairs from VoxPopuli (Wang et al., 2021a), a collection of European Parliament recordings. To the best of our knowledge, SpeechMatrix is by far the largest freely available speech-to-speech translation corpus, with 136 language directions and an average of 1,537 hours of source speech in each direction for a total of 418 thousand hours. We demonstrate that strong S2ST models can be trained with these mined data and validate the good quality of the speech alignments across languages. We are open-sourcing the mined data and the speech encoders used for
2Czech (cs), German (de), English (en), Spanish (es), Estonian (et), Finnish (fi), French (fr), Croatian (hr), Hungarian (hu), Italian (it), Lithuanian (lt), Dutch (nl), Polish (pl), Portuguese (pt), Romanian (ro), Slovak (sk) and Slovenian (sl).
16251
mining, which could pave the way for future research on S2ST. Moreover, for reproducibility, we will release model components including multilingual HuBERT models in four language families for target unit generation, language-specific vocoders for speech synthesis from discrete units, and S2S models trained and presented in this work.

heading: 2 Related Works
text: From bitext mining to speech mining. Bitext mining is to find parallel sentences from monolingual resources, which provides a large amount of training data for machine translation models. Early works on bitext mining used document metainformation (Resnik, 1999), cross-lingual document retrieval (Munteanu and Marcu, 2005) or information retrieval (Abdul-Rauf and Schwenk, 2009; Bouamor and Sajjad, 2018). More recent works use multilingual sentence embeddings (Artetxe and Schwenk, 2018; Yang et al., 2019; Schwenk et al., 2021a). The embedding based approach can be extended to new languages (Reimers and Gurevych, 2020; Heffernan et al., 2022) or the speech modality (Duquenne et al., 2021; Khurana et al., 2022) with knowledge distillation, also called teacher-student approach. These multilingual and multimodal sentence embeddings enabled us to perform large-scale speech-text mining, or speech-speech mining for a small set of languages.
Speech-to-speech translation (S2ST). S2ST started from cascaded systems consisting of automatic speech recognition (ASR), machine translation (MT) and text-to-speech synthesis (TTS) (Nakamura et al., 2006; Do et al., 2015). The reliance on intermediate text outputs poses limitations on cascaded models to support efficient inference and unwritten languages. Given these challenges, there has been a recent surge of research interest in direct approaches to speech translation without the need of texts. Translatotron (Jia et al., 2019) and Translatotron2 (Jia et al., 2022b) train end-to-end S2ST to generate target spectrograms with multitask learning. Another line of research replaces the target spectrograms in S2ST modeling with discrete units which are learned from a large amount of unlabeled speech (Lee et al., 2022a,b). Discrete units have shown to better capture linguistic content than spectrograms. Despite these progress on direct S2ST, it is faced with the challenge of limited parallel speech.
Speech translation corpora. The Fisher dataset,
a collection of approximately 170 hours of telephone conversations in Spanish (Post et al., 2014), is commonly used as training data for SpanishEnglish S2ST. However, it does not provide parallel English speech. Previous works generate synthesized English speech from English text translations provided by Fisher. Another S2S dataset containing synthesized speech is CVSS, which covers parallel S2ST translations from 21 languages into English. It is derived from Common Voice (Ardila et al., 2020) and CoVoST 2 (Wang et al., 2021b), and it synthesizes speech from translated texts. The release of VoxPopuli dataset provides the largest S2S translations in real speech so far (Wang et al., 2021a). It covers pairwise speech-to-speech translations among 15 languages, and each direction has less than 500 hours of speech. In another initiative named FLEURS, the text-to-text evaluation data of the FLoRes-101 benchmark (Goyal et al., 2022) was extended to the speech modality. Supporting 102 languages, FLEURS has a larger language coverage than VoxPopuli, but it only contains around 12 hours of speech per language and it is intended to be used asN -way parallel test data.
In this work, we present SpeechMatrix, a largescale multilingual speech-to-speech corpus mined from VoxPopuli (Wang et al., 2021a). It contains speech alignments in 136 language pairs with an average of 1, 537-hour source speech per direction. The main characteristics of these speech corpora are summarized in Table 1.

heading: 3 Speech-to-Speech Mining
text: The mining approach of this work is built upon the idea of encoding multilingual speech utterances into a shared embedding space. Speech encoders project utterances with similar semantic content to fixed-size representations which are close in the embedding space regardless of their languages. The closeness of embeddings reflects the similarity of speech content, and is used as the alignment score in the mining process. In this section, we discuss speech encoders and speech mining.

heading: 3.1 Speech Encoders
text: We followed the teacher-student approach introduced in (Duquenne et al., 2021) and trained speech encoders with the supervision of the multilingual LASER text encoder (Schwenk et al., 2021b). Transcriptions or written translation of the audio utterances are encoded with LASER text encoder as
2
target vectors for speech encoder training. During training, we minimize the cosine loss between fixed-size representations output by speech encoders, and the outputs of LASER text encoder (whose weights are frozen during training). Speech encoders are initialized with the 2B-parameter XLS-R model (Babu et al., 2021), which was pre-trained on nearly half a million hours of publicly available audios in 128 languages. Following (Duquenne et al., 2022), the fixed-size representation for speech is obtained with max pooling of the encoder outputs which appeared to work better compared to other pooling methods. We summarize the architecture of the speech encoder in Figure 1.
We used various publicly available ASR data sets which cover our languages to train the speech encoders, including CoVoST 2 (Wang et al., 2020, 2021b), Common Voice (Ardila et al., 2020), Europarl (Ardila et al., 2020), mTedx (Salesky et al., 2021), Must-C (Di Gangi et al., 2019) and VoxPopuli (Wang et al., 2021a), as well as speech translation data from the foreign languages into English and from English into German. We removed training samples whose transcription or the written translation consisted of multiple sentences, as LASER has been trained on single sentences only. For better training efficiency, we trained speech encoders for each language family instead of each language. The language grouping is provided in Appendix. To better handle imbalanced training
data, we sample the training data from different languages with the same approach as (Duquenne et al., 2021). For English (en), Slovenian (sl), Lithuanian (lt) and Dutch (nl), we also trained separate monolingual speech encoders that had lower valid cosine loss compared to multilingual encoders, and these four monolingual encoders were used for mining.

heading: 3.2 Evaluation of speech encoders
text: Similarity search is frequently used to evaluate multilingual text encoders (Artetxe and Schwenk, 2018; Feng et al., 2020; Heffernan et al., 2022). We use the following score to measure similarity between the source audio, and the target transcriptions or translations:
sim(x,y) (1)
= cos(x, y) − (
∑ z∈NNk(x) cos(x, z) 2k + ∑ z∈NNk(y) cos(y, z) 2k
)
where x and y are the source and target embeddings, and NNk(x) denotes the k nearest neighbors of x. We used k = 4. We evaluated similarity search of audios against transcriptions on VoxPopuli ASR test set in Table 2, which is our target domain as we plan to mine unlabeled speech from VoxPopuli (see subsection 3.3). We also evaluated similarity search of audio against written translations or transcriptions on CoVoST 2 test set in order to compare with speech encoders in previous work (see detailed analysis in Appendix A). Finally, we report text-to-text similarity search using the LASER text encoder as lower bound for the speech translation similarity search error rate since we use gold transcriptions to search against written translations. We report error rates (in %) that are percentage of audio utterances incorrectly matched with text transcripts from the same test set. We note that error rates are very low for all languages (below 5% and around 1 or 2% for most languages), which is an initial validation of good-quality speech encoders before the large-scale mining.
3

heading: 3.3 Large-scale speech mining
text: We used VoxPopuli as our source of unlabeled unsegmented speech for 17 languages in focus. In principle, performing speech-to-speech or speechto-text mining can be done with exactly the same pipeline as text-to-text mining but with different encoders. We follow the global mining approach as described in Schwenk et al. (2021a) and compare all segments in the source language with all segments in the target language. Similarity scores are calculated in both directions using the margin as described in Equation 1 considering k = 16 neighbors. Segments are considered to be parallel if the margin score exceeds a threshold, we use 1.06 if not specified otherwise. The reader is referred to Schwenk et al. (2021a) for a detailed description of the generic mining pipeline.
There is however one important difference when processing speech: it is not straightforward to segment the audio signal into parts which have the optimal granularity for mining. The VoxPopuli recordings have a rather long duration, e.g. one hour and a half on average for English. We apply Voice Activity Detection (VAD) using Silero-VAD (Silero-Team, 2021) which supports over 100 languages. The resulting segments do not necessarily correspond to complete sentences. On one hand, there may be silence in the middle of an utterance, e.g. a hesitation. On the other hand, two sentences may follow each other without a long silence separating them. We follow the “over segmentation” approach outlined in Duquenne et al. (2021): several possible segments are created and we let the mining algorithm decide which ones match the best. Initial experiments suggest that segments shorter than 1 second or longer than 20 seconds are unlikely to be aligned and therefore were excluded.
After mining, the resulting speech alignments may have overlap as we over-segment the unlabeled speech. A post-processing method Duquenne et al. (2021) is introduced to remove overlaps between mined speech segments on the source speech side. We relax the post-processing of the mined data, allowing for some overlap between mined speech segments: for two audio segments that overlap on
the source side, if the overlap represents more than 20% of the first segment and of the second segment, we discard the alignment with the lowest mining score. We did an ablation study on different thresholds of overlap ratio for one low-resource, one mid-resource and one high-resource direction and found that 20% was the best overlap threshold in all settings.
We report the statistics of the mined speech-tospeech translation pairs in Table 3, with a mining score threshold of 1.06. The mined data totals 418k hours of parallel speech with an average of 1,537 hours of source speech in all translation directions. While some high resource languages like English (en), Spanish (es) or French (fr) can reach up to 5k hours of aligned speech with other spoken languages; lower resource languages such as Estonian (et) and Lithuanian (lt) obtain much fewer alignments, with only a few hours of aligned speech for Lithuanian. We also performed mining of the source speech in sixteen languages against more than twenty billion English sentences from Common Crawl. This yielded speech-text alignments between 827 and 3, 966 hours (c.f. the last column of Table 3). Training and evaluation of speech-totext translation are left for future research.

heading: 3.4 Evaluation Data
text: Besides the speech-to-speech data mined as the train set, we leverage labeled public speech datasets as the evaluation sets.
Test set. In our experiments, we derive test sets in speech translation from three public corpora, evaluating translation models trained on mined data across different domains.
(1) Europarl-ST (EPST) (Iranzo-Sánchez et al., 2020). It is a multilingual speech-to-text translation corpus built on recordings of debates from the European Parliament, containing 72 translation directions in 9 languages.3
(2) VoxPopuli (Wang et al., 2021a). S2S data, as part of VoxPopuli release, provides aligned source and target speech together with source transcriptions. We prepare the speech-to-text data with
3en, fr, de, it, es, pt, pl, ro and nl
4
target speech and source transcription as our test set. To ensure that there is no overlap between the mined data and VoxPopuli test sets, we need to remove speech from mined alignments which are from the same session as test samples. In order to keep as much mined data as possible, we use VoxPopuli test set only when a language direction is not covered by EPST considering their domain similarity. Moreover, similarity scores are provided to indicate the quality of VoxPopuli samples. To choose high-quality data, we sort all sessions in the VoxPopuli S2S data in a decreasing order of the average similarity score of their samples. We keep adding samples from highly ranked sessions to the test set until the test size reaches 1000.
(3) FLEURS (Conneau et al., 2022). Built upon N-way text translations from FLoRes (Goyal et al., 2022), FLEURS provides speech for aligned texts and creates speech-to-speech data covering all mined directions. We take its source speech and target texts as the test data. In the case where multiple utterances correspond to one piece of source text, we generate one test pair for each source utterance respectively. FLEURS texts are from English Wikipedia, which is a different domain from VoxPopuli and EPST.
Valid set. Valid sets are prepared for S2S modeling using VoxPopuli and FLEURS data in a similar way as test sets. For VoxPopuli, we extract a valid set of about 1000 samples by adding data from highly scored sessions which are not in the test set. FLEURS valid set is derived from its valid sam-
ples. We prepare speech-to-unit data from these selected valid samples by transforming the target speech into target units for speech-to-unit modeling, which will be discussed in section 4.

heading: 4 Experiments & Results
text: To evaluate the quality of the mined data, we trained S2ST models on SpeechMatrix data and report the translation performance. We hope that these results will serve as baselines for future studies in speech-to-speech translation.
5

heading: 4.1 Experimental Setup
text: The training and evaluation pipeline of speech-tospeech translation is shown in Figure 2. Recent progress in speech-to-speech modeling suggests to discretize the target speech waveform into a unit sequence, relieving models from the complexity of predicting continuous waveform values. We borrow the idea of training speech-to-unit (S2U) model where units are pre-generated from target speech with a pre-trained HuBERT model (Lee et al., 2022a). During S2U training, models are periodically evaluated on the valid set of speechto-unit samples, and the best checkpoint with the lowest valid loss is saved for model inference.
When it comes to inference, speech could be synthesized from the predicted units with a vocoder, as the output of the S2S pipeline. It is then transcribed into texts by an off-the-shelf ASR model. The BLEU score is calculated by comparing the transcriptions against the ground truth target texts, which serves as the quantitative metric of mined data quality. We note that the ASR BLEU score is not a perfect metric for data quality, as it is unavoidably affected by the quality of ASR models. Next we discuss each module of the pipeline.
Speech-to-Unit. The S2U model takes the source speech and predicts a sequence of target units. It typically has an encoder-decoder architecture, where the encoder consists of convolutional and Transformer encoder layers, and the decoder is a Transformer decoder. We have experimented with different model variants, and discuss bilingual
and multilingual training in section 5 and section 6. HuBERT. HuBERT is used to extract speech features of audio frames, which are then grouped into k-means clusters. The continuous features are thus mapped to corresponding clusters. In this way, speech could be discretized into unit sequence where units are basically indices of clusters. We reuse the same HuBERT model and k-means clusters for English, Spanish and French as in (Lee et al., 2022b) for a fair comparison with existing results. We also train multilingual HuBERT models to cover other languages in SpeechMatrix, and more HuBERT training details can be found in Appendix B.1.
Vocoder. Unit-based HiFi-GAN vocoders are trained to synthesize speech from unit sequence (Polyak et al., 2021). In our experiments, vocoders are separately trained from S2U model. We train vocoders on three datasets:
(1) CSS10 (Park and Mulc, 2019). It is a singlespeaker corpus which we use to train vocoders in German, Finnish, Hungarian and Dutch.
(2) VoxPopuli (Wang et al., 2021a). Given its ASR data with speaker id, we sort speakers based on their speech duration, and keep adding the top speakers until the speech is more than 20 hours.
(3) Common Voice (Ardila et al., 2020). Portuguese and Estonian are not covered by the two corpora above, and thus we turn to Common Voice. Again, we select top speakers and prepare 12-hour and 10-hour speech for the vocoder training in Portuguese and Estonian respectively.
6
Data preprocessing and training are included in Appendix B.3.
ASR. We use off-the-shelf ASR models to transcribe the speech generated by vocoders. Details about the ASR models and their benchmark results of word error rates are provided in Appendix B.2.

heading: 5 Bilingual Speech-to-Speech Baselines
text: In this part, we discuss the bilingual S2S models trained in each of 272 language directions in SpeechMatrix. The architecture of Textless model is used for bilingual translation in our experiments (Lee et al., 2022a). A Textless model consists of a speech encoder, Transformer encoder and decoder.
Training. For a given direction, we extract units for source and target speech with their corresponding HuBERT models (Hsu et al., 2021). Taking source speech, the model is trained to predict target unit sequence with cross-entropy loss as well as source unit reconstruction as an auxiliary task.
For the training efficiency of extensive S2ST experiments, we use a subset of mined data as the train set. Mined samples are selected if their alignment scores are above a preset threshold. We performed an analysis of the threshold selection in subsection B.4.
Comparison with existing results. Since we adopt the same model as the previous work (Lee et al., 2022a) and the only difference lies in the train set, it is straightforward to compare with existing results. Table 4 shows the results of S2ST models which are trained on our SpeechMatrix mined data compared to VoxPopuli S2S data in each of four language directions: es-en, fr-en, en-es and enfr. The threshold of mined data is set as 1.09 to these four directions, yielding an average of 1, 436- hour train set. Compared with 480-hour labeled speech from VoxPopuli, SpeechMatrix achieves an an average improvement of 5.4 BLEU, indicating the good quality and usefulness of the mined data.

heading: 5.1 Large-Scale Bilingual Evaluation
text: A large-scale evaluation is launched covering 272 mined languages directions, and bilingual models are trained for each direction to establish baseline results in speech-to-speech translation.
Table 5 and Table 6 summarize performance of bilingual S2ST models on three test sets. In each direction, Table 5 reports BLEU scores in European Parliament domain, either EPST or VoxPopuli set. EPST BLEU is underlined to be distinguished
from VoxPopuli BLEU. Table 6 reports BLEU in Wikipedia domain, i.e., FLEURS test data.
Bilingual results. Empirically we find that translations into high-resource languages such as en, es and fr outperform those into low-resource languages such as lt and sl based on the speech amount of these languages in Table 3. Another observation is the performance difference across test domains, and BLEU on FLEURS is lower than that on EPST and VoxPopuli data, likely because of the domain mismatch between train and test data.
It is also found that translation results are not symmetric for some language pairs, for example, ro-en has a BLEU of 22.6 while en-ro BLEU is only 7.6 on EPST. Besides different complexity levels of target languages and test sets, such asymmetry also results from the dependency of BLEU scores on the speech synthesis quality of the vocoder and transcription quality of the ASR model. For languages whose vocoder and ASR models are not good, they are likely to receive low BLEU scores. In this case, Romanian vocoder and ASR are not as strong as English models as reflected by its higher word error rate in speech resynthesis as reported in Appendix B.3.

heading: 6 Multilingual Speech-to-Speech Translation
text: Multilingual modeling has been explored in tasks of language understanding and machine translation, demonstrating knowledge transfer among languages. However, to our best knowledge, there are few studies of multilingual S2ST on real speech, partially due to the lack of multilingual speech-tospeech resources. With the massively multilingual data we have mined, we are able to explore multilingual S2ST training.
In this work, we focus on many-to-English translation, studying the translation from 6 Slavic languages to English in subsection 6.1 and the translation from all 16 languages in SpeechMatrix to English in subsection 6.2. English-to-many or manyto-many translation are left to future work. We present here multilingual models used in our experiments (more details can be found in Appendix C:
(1) Textless model. The same model with 70M parameters that we use for bilingual evaluation is reused in the multilingual experiments. Given diverse multilingual data, we increase the model size for larger model capacity, trying multilingual models with 70M and 260M parameters.
7
(2) XM Transformer. Inspired by the recent finding that crossmodal pre-training is beneficial for speech translation (Popuri et al., 2022), we apply XM Transformer to multilingual training, whose encoder is initialized from pre-trained XLSR model with 1B parameters (Babu et al., 2021) and decoder is initialized from a unit decoder pretrained in an mBART style (Popuri et al., 2022). With multilingual speech-to-unit data, the model is further finetuned to minimize the cross-entropy loss in target unit prediction.
(3) XM Transformer with Sparsity. Sparse modeling, in particular Mixture-of-Experts (MoE), has been widely studied in multilingual machine translation. MoE increases the number of parameters without sacrificing computation efficiency.
GShard. GShard is a sparse scaling technique proposed in (Lepikhin et al.). We replace every other Transformer layer with an MoE layer. FFN modules in an MoE transformer layer are shared across experts. A learnable gating function routes input tokens to different experts (NLLB Team et al., 2022). We apply GShard architecture on the decoder of XM Transformer, and expert weights are all initialized with the pretrained unit mBART.

heading: 6.1 Slavic-to-English Translation
text: The six Slavic languages include Czech (cs), Croatian (hr), Lituanian (lt), Polish (pl), Slovak (sk), and Slovenian (sl). In the multilingual setting, all mined data into English are combined from each
Slavic language as the train set.
We summarize ASR BLEU scores of different models averaged over six Slavic-to-English directions in Table 7. For completeness, we report BLEU of each direction in Appendix C. As is shown, Textless model benefits from the parameter increase to 260M, and multilingual training further brings BLEU gains of 5.6 and 4.7 in EP/VP and FLEURS. We tried larger models than 260M but didn’t see more gains.
Comparing against bilingual Textless model (70M), bilingual XM Transformer achieves +3.8 BLEU in EP/VP and +5.0 BLEU in FLEURS. Multilingual training further improves dense XM Transformer by 7.9 and 5.1 BLEU. GShard with 64 experts brings +1.0 BLEU over dense XM Transformer to EP/VP, and +0.3 BLEU to FLEURS. Overall the best Slavic-to-English translation is achieved by XM Transformer with GShard trained in multilingual setting. This demonstrates that multilinguality, pre-training and model sparsity are of help to speech-to-speech translation.
8

heading: 6.2 All-to-English Translation
text: We move forward to a larger-scale multilinguality by extending from Slavic language family to all languages in SpeechMatrix. We adopt the best models in Slavic-to-English translation, i.e., multilingual XM Transformer with both dense and sparse architectures.
Results. Compared with XM Transformer (1.2B) dense model, MoE-GShard64 (4.3B) with the same forward computation time brings gains of +0.9 and +0.2 BLEU to EP/VP and FLEURS respectively. Similar to our findings in Slavic-toEnglish setting, increasing the capacity with sparse modeling benefits in-domain (EP/VP) more than out-of-domain FLEURS test set.
Given sparse architecture of XM Transformer with GShard, all-to-English model shows +0.6 and -0.4 BLEU difference compared with Slavic-toEnglish model on EP/VP and FLEURS respectively, averaged over Slavic languages. Multilingual sparse model benefits from the additional indomain data in other languages when evaluated in EP/VP domain, while sees performance degradation in out-of-domain data.

heading: 7 Limitations and Risks
text: Limitations. The HuBERT model quality is critical to speech-to-speech translation performance, as its extracted units are used by both speech-tounit model and vocoder. We have not explored the optimal strategy of multilingual HuBERT training. One research question is how to choose a group of languages so that a multilingual HuBERT model could be well trained. For example, it is arguable whether Lithuanian (lt) should be included in Slavic or Uralic family. Other questions could be whether a larger HuBERT with more model capacity should be used and how we should deal with language imbalance in multilingual training.
We provide benchmark results of bilingual speech translation with mined data selected by heuristics. One of our future directions is to come up with a better strategy of mined data selection to improve translation performance and training efficiency.
As mentioned in our results analysis, the reported BLEU scores are heavily dependent on the ASR quality, which may not reflect the speech translation performance accurately. Future directions could be improving ASR quality or exploring other evaluation metrics without reliance on ASR models.
Potential Risks. As a technology used for speech generation, the presented speech translation models or the translation models that will be trained with SpeechMatrix dataset might have systemic bias or produce inappropriate outputs.

heading: 8 Conclusion
text: In this paper, we introduce a large-scale multilingual speech-to-speech corpus mined from VoxPopuli. It is the largest resource of speech alignments with a coverage of 17 languages. We perform an extensive evaluation of the mined parallel speech, showing good quality of the speech alignments. Multilingual speech-to-speech models can be efficiently trained on this corpus and we suggest different methods, such as sparse scaling using Mixture-of-Experts, to further boost translation performance in the multilingual setting.

heading: A Speech Encoder
text: A.1 Similarity search on CoVoST
We compared our similarity search results with previous work (Duquenne et al., 2021) in Table 9. We notice that our new speech encoders have lower error rates compared to previous work.
We also provide similarity search of audios against written translations or transcriptions on CoVoST 2 test set for other languages covered by our speech encoders in Table 10, in order to evaluate cross-modal similarity search.

heading: B Bilingual Speech-to-Speech Translation
text: We describe experiment details of bilingual speechto-speech translation.
We train a multilingual HuBERT model for each family on the collection of speech in each component language as shown in Table 11. We collect unlabeled VoxPopuli speech for all languages of the same family as the training data. The HuBERT model consists of 7 convolutional layers and 12 Transformer encoder layers. Each encoder layer has 12 attention heads, the embedding dimension of 768 and the forward dimension of 3072. Models are trained for 3 iterations, and in each iteration pseudo-labels are prepared as the training target for
12
utterances. In the first iteration, the target labels are MFCC features. In the second iteration, we extract speech features from the 6-th layer of the trained HuBERT model and apply k-means clustering to derive a set of 500 labels. In the third iteration, speech features from the 9-th layer are clustered into 500 labels. Lastly after these three iterations, we try feature extraction from different layers including layer 10, 11 and 12 of trained HuBERT. As for feature clustering, we also try different numbers of clusters, 800, 1000 and 1200, to derive multiple sets of target units.
To choose the optimal setup, we launch a resynthesis evaluation to select the HuBERT layer to extract speech features and the number of k-means clusters. We train a vocoder on each set of target units, i.e., vocoder takes the units and synthesizes target speech. The synthesized speech is sent to off-the-shelf ASR models, and Word Error Rate (WER) is reported to measure the speech quality. The resynthesis experiments are discussed in subsection B.3. The optimal HuBERT layer and label size is selected if their corresponding vocoder achieves the lowest WER.
B.2 ASR models
We use ASR models publicly released on HuggingFace to transcribe the generated speech in order to calculate WER or BLEU scores in comparison with ground truth texts. ASR models used in our evaluation are listed in Table 12.
B.3 Vocoder
Data preprocessing. We applied a denoiser4 (Defossez et al., 2020) to the speech of VoxPopuli and Common Voice as the speech preprocessing to increase signal-to-noise ratio (SNR) given that they are noisier than CSS10 audios. Then we prepare vocoder labels with HuBERT models generating k-means cluster labels for each utterance. Single-speaker vocoders are trained in CSS10, and languages from VoxPopuli and Common Voice have multi-speaker vocoders where speaker embeddings are learned. During inference, we select the speaker with the longest speech duration to synthesize speech from predicted unit sequences, who has the most data for the vocoder to learn good speaker embeddings.
Vocoder training and evaluation. Vocoders are trained to synthesize speech from a given sequence of units. The train sets are speech data from CSS10, VoxPopuli and Common Voice. As mentioned before, units are derived from HuBERT models for these speech. Table 13 summarizes WER of ASR models, which reflects the transcription quality in each language. Besides, we report the training data, vocoder WER of synthesized speech from vocoders, and here we include the vocoder results obtained from the optimal HuBERT layer and kmeans cluster size. Layer 11 is the best HuBERT layer for feature extraction in all languages, and most languages have the best k-means size of 1000 except Italian (it) whose best label size is 800.
As shown in Table 13, ASR models are of good quality for high-resource languages such as de, fi
4https://github.com/facebookresearch/ denoiser
13
and pt, while suffering from high error rates in languages such as ro, lt and sl. It is expected to have higher vocoder WER than ASR WER since the former is for synthesized speech. By measuring the gap between the two error rates, we can tell how good a vocoder is and also infer the quality of HuBERT units. For et, pt and lt, the gaps are obviously larger than other languages. It not surprising since we do not have much good-quality vocoder data for these languages. For example, there is only around 10-hour noisy speech from Common Voice for et and pt vocoder training.
B.4 Training
Textless model. A Textless model consists of a speech encoder with 2 convolution layers and 12 Transformer encoder layers. Transformer layer has the embedding dimension of 512 and the forward dimension of 2048. It has two unit decoders with 6 and 2 Transformer decoder layers for target and source unit prediction respectively. The target unit decoder has the embedding dimension of 512 and the forward dimension of 2048, and the source unit decoder’s dimensions are 256 and 2048.
Hyperparameters. We tried learning rates of 0.0003 and 0.0005, and dropout rates of 0.1 and 0.3. The best setup is a learning rate of 0.0005 and a dropout of 0.3 for bilingual Textless model training. Bilingual models are trained with a batch of 20k tokens for 400k steps. A label smoothing weight of 0.2 is applied to the cross-entropy loss.
As for decoding of speech-to-unit models, we set the beam size of 10 in all bilingual and multilingual experiments.
Mined data selection. We performed an analysis of translation performance varying with thresholds from 1.06 to 1.09 on three language pairs: es-en, ro-en and hr-en. Figure 3 shows the thresh-
old, the corresponding speech data size and BLEU score.
For low-resource directions such as hr-en, it is best to include all the mined data. For high- and medium-resource directions, es-en and ro-en, the optimal amount of mined data is around 1k hours and it does not bring further gains to go beyond that data size. Given these observations, we choose the highest threshold that keeps the source speech duration in mined data more than 1k hour for each direction. For example, we use a threshold of 1.09 for es-en and of 1.06 for hr-en.
Computation. Each bilingual model is trained on 16 A100 GPUs for 3 days on average.

heading: C Multilingual Speech-to-Speech Translation
text: We provide details of models and experiment setups in multilingual speech-to-speech translation.
C.1 Slavic-to-English Translation
Textless model. Textless model (260M) has a speech encoder with 4 convolution layers and 12
14
Transformer encoder layers with the embedding dimension of 1024 and the forward dimension of 4096. It has two unit decoders with 6 and 2 Transformer decoder layers for target and source unit prediction respectively. The target unit decoder has the embedding dimension of 1024 and the forward dimension of 4096, and the source unit decoder’s dimensions are 256 and 2048.
For the Textless model (424M), its speech encoder contains 6 convolution layers and 16 Transformer encoder layers with the embedding dimension of 1024 and the forward dimension of 4096. It has two unit decoders with 12 and 2 Transformer decoder layers for target and source unit prediction respectively. The target unit decoder has the embedding dimension of 1024 and the forward dimension of 4096, and the source unit decoder’s dimensions are 256 and 2048.
XM Transformer. XM Transformer (1.2B) is initialized from XLS-R encoder with 7 convolution layers and 48 Transformer encoder layers with the embedding dimension of 1280 and the forward dimension of 5120. Its unit decoder is initialized from a pre-trained mbart-style decoder with 12 layers, embedding dimension of 1024 and forward dimension of 4096.
Hyperparameters. For Textless model, we reuse a learning rate of 0.0005, a dropout of 0.3
and a label smoothing weight of 0.2 for Slavic-toEnglish training. The 70M model has 20k tokens in one batch. The 260M model has batch tokens of 6k and a update frequency of 4. The 424M model has tokens of 4000 and a update frequency of 6. For XM Transformer model, we use a learning rate of 0.0001, a dropout of 0.1 and a label smoothing weight of 0.2. In a batch, token sizes of 1500 and 9000 with update frequency of 15 and 2 are used for V100 and A100 training respectively.
Results. We first extend Textless model from the bilingual to multilingual setting. Translation results are presented for Textless models with different parameter sizes in Table 14. Multilingual Textless model works best with 260M parameters. Compared with its bilingual counterparts, an average gain of 5.6 BLEU is achieved in EP/VP and the gain of 4.7 BLEU in FLEURS.
With the Textless model size fixed as 70M, multilingual training hurts the performance of most languages compared with bilingual training. This is due to the insufficient model capacity, and the language interference is reflected by an average of −2.6 BLEU in FLEURS. We increase model parameters to 260M in both bilingual and multilingual settings. With a larger model capacity, bilingual models achieve gains in high-resource languages including cs, pl and sk, while suffering from performance loss in low-resource directions such as hr, lt and sl.
Given model sizes of 260M, we observe consistent gains of multilingual models over the bilingual models across different language directions and test domains. An average gain of 5.6 BLEU is achieved in EP/VP and the gain of 4.7 BLEU in FLEURS. It demonstrates the positive transfer enabled by multilingual training. As the multilingual model size continues to increase to 424M, we don’t observe further gains likely due to the bottleneck of training data amount.
XM Transformer leveraging pre-trained modules is also trained on Slavic-to-English data. Pre-training is shown to be beneficial, and results are reported in Table 15. Comparing against bilingual Textless model (70M), bilingual XM Transformer outperforms it in all directions except lt-en. The gain in EP/VP is 3.8 BLEU on average, and a larger gain of 5.0 BLEU is achieved in FLEURS. Multilingual training brings further gains to XM Transformer with +7.9 and +5.1 BLEU over bilingual training in EP/VP and FLEURS test set
15
respectively. Comparing against dense XM Transformer, GShard with 64 experts has 1.0 BLEU gains on average over 5 directions on EP/VP, and +0.3 BLEU gains for FLEURS. We believe that it is due to a phenomena mentioned in (Zoph et al., 2022), i.e., MoE specializes in multilingual settings but not by language. GShard in our setting brings larger improvements to in-domain test sets.
Computation. Textless models used 32 A100 GPUs, the 70M model was trained for 3 days, the 260M model was for 5 days, and the 424M model was for 6 days. It took 2 days to train XM Transformer on 32 A100 GPUs for Slavic-toEnglish translation.
C.2 All-to-English Translation
In this work, we experiment with two variants of sparse modeling, GShard and Base Layer.
XM Transformer-GShard. XM Transformer (1.2B) is initialized with the same XLS-R encoder and unit decoder used in Slavic-to-English experiments. On the decoder side of XM TransformerGShard, each expert is initialized with the same unit decoder. We set MoE frequency as 2, i.e., every other Transformer layer is an MoE layer.
XM Transformer-Base Layer. For our XM Transformer with Base Layer sparsity (1.7B), the encoder is initialized with the same XLS-R encoder, and the dense layers of the decoder is initialized with the same unit decoder as GShard. We
add an additional Base Layer which is randomly initialized as the 7th layer of decoder. There is one expert in each GPU and we used 64 GPUs in our experiments, which means we have 64 Base Layer experts in total.
The sparse variant, Base Layer (1.7B) performs comparably to the dense XM Transformer, with an average of +0.4 BLEU in EP/VP test sets and -0.4 BLEU in FLEURS. The sparsity in Base Layer does not bring obvious gains to all-to-English translation. This is likely because we only add one Base Layer to the decoder with a small expert size. The number of increased model parameters is only 0.5B in Base Layer, while it is 3.1B in GShard. As suggested by (Lewis et al., 2021), the Base Layer performance might improve with more GPUs and a larger expert size.
Hyperparameters. For dense XM Transformer, hyperparameters are the same as that for Slavic-toEnglish. GShard also shares the same set of hyperparameters. As for expert-specific parameters, we use 64 experts with each running on a single GPU with the frequency of 2 so that every other Transformer decoder layer becomes an MoE layer. The capacity token fraction is set as 0.5 so that if more than half of tokens in a sample get routed to one expert, extra tokens would overflow and get dropped.
Computation. It took 3 days to train dense XM Transformer for all-to-English with 32 A100 GPUs. It took 5 days to train the GShard counterpart with 64 A100 GPUs.

heading: D License and Terms of Scientific Artifacts
text: D.1 Third-Party Artifacts
Data. Common Voice is released under CC0 license, VoxPopuli and CoVoST 2 data are also under CC0 license. As for EuroParl, it is released under a Creative Commons license. The multilingual TEDx corpus is released under a CC BY-NCND 4.0 license. FLEURS dataset is under Creative Commons license (CC-BY-4.0). These datasets are publicly accessible and freely downloadable for research purposes. Models. XLS-R model used for the speech encoder initialization is open sourced under Apache-2.0 license. Text LASER used as the teacher model in training is released under BSD license. ASR models avaliable on HuggingFace are released under Apache-2.0 license. These models are publicly
16
available. Code. The implementations of Textless model, XM Transformer, HuBERT and Vocoder are open sourced under MIT license.
D.2 SpeechMatrix and translation models The mined resource, SpeechMatrix, will be released under CC0 license, and the trained speechto-speech translation models will be released under CC BY-NC 4.0. The data and models are intended for research purposes.
17
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations of this work is discussed in Section 7.

heading: 3 A2. Did you discuss any potential risks of your work?
text: Potential risks of this work are discussed in Section 7.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: This paper’s main claims are summarized in abstract and introduction (Section 1).
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Sections 2-6.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: SpeechMatrix is mined from VoxPopuli, a raw speech corpus, which is cited in Section 2. We are using public datasets in speech recognition and translation to train speech encoders, and the dataset creators are cited in Section 3.1. Speech translation datasets which are used as testsets for speechto-speech translation models are cited in Section 3.4. As for the model, speech encoder is initialized from XLS-R model, which is cited in Section 3.1. The code and implementations which are used for speech-to-speech translation in our experiments are cited in Section 4, 5 and 6.

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: The license and terms are discussed in Appendix D.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Appendix D discusses the use of existing artifacts and our created artifacts.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The datasets used in this work such as Common Voice and CoVoST have anonymized speakers and require dataset users not to attempt to identify their identities.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? We provide domain coverage and language coverage in Section 3.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. We report the data size of train, valid and test samples in Section 3.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments?

heading: Appendix B.4.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? We report the parameter size and computational costs in Appendix B.4.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? No response.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? We report the experiment results from single runs in Section 4, 5 and 6 of the main paper, as well as Appendix B and C.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? We report the evaluation setup and parameter setting in Appendix B and C.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: One of our future directions is to come up with a better strategy of mined data selection to improve translation performance and training efficiency. As mentioned in our results analysis, the reported BLEU scores are heavily dependent on the ASR quality, which may not reflect the speech translation performance accurately. Future directions could be improving ASR quality or exploring other evaluation metrics without reliance on ASR models. Potential Risks. As a technology used for speech generation, the presented speech translation models or the translation models that will be trained with SpeechMatrix dataset might have systemic bias or produce inappropriate outputs