SlowGuess/NAACL_2022_OCR huggingface dataset:
BLEU (↑)	MOS (↑)
ID	tgt spkemb	tgt SN	tgt text	Es-En	Fr-En	En-Es	En-Fr	Es-En	Fr-En	En-Es	En-Fr
1	S2UT w/ orig-unit	✘	✘	✘	13.1	15.4	16.4	15.8	2.32 ± 0.10	2.43 ± 0.11	2.97 ± 0.14	2.41 ± 0.08
2	S2UT w/ orig-unit	✓	✘	✘	16.1	16.6	19.3	15.6	2.29 ± 0.11	2.25 ± 0.10	3.48 ± 0.11	2.25 ± 0.06
3	S2UT w/ norm-unit	✘	10-min	✘	17.8	18.5	20.4	16.8	2.99 ± 0.07	3.16 ± 0.07	3.92 ± 0.11	2.65 ± 0.08
4	S2UT w/ norm-unit	✘	1-hr	✘	18.8	20.3	21.8	18.7	3.20 ± 0.09	3.26 ± 0.08	4.09 ± 0.11	2.92 ± 0.09
5	S2UT w/ norm-unit	✘	10-hr	✘	18.9	19.9	22.7	18.7	3.26 ± 0.08	3.27 ± 0.08	4.17 ± 0.10	2.84 ± 0.08
6	S2T + tf TTS	✘	✘	ASR	19.2	19.8	21.7	18.5	3.23 ± 0.13	3.22 ± 0.11	4.12 ± 0.11	2.44 ± 0.08
7	S2T + T2U	✘	✘	ASR	19.4	19.7	21.8	18.9	3.16 ± 0.08	3.21 ± 0.07	4.11 ± 0.11	2.87 ± 0.09
8	gt + tf TTS	✘	✘	✘	88.0	87.2	82.0	69.2	-	-	-	-
9	gt + T2U	✘	✘	✘	87.9	87.1	84.6	73.8	-	-	-	-
Table 4: BLEU and MOS (reported with  $95\%$  confidence interval) from systems trained in a single run with VoxPopuli S2ST data (Wang et al., 2021c) and evaluated on Europarl-ST (Iranzo-Sanchez et al., 2020) test sets. The best results from S2UT w/ norm-unit are highlighted in bold. (tgt spkemb: target speaker embedding, SN: speech normalization, gt: ground truth, tf: Transformer)
Es-En	Fr-En	En-Es	En-Fr
ID	data	tgt spkemb	tgt SN	tgt text	EP	CVST	EP	CVST	EP	EP
4	S2UT w/ norm-unit	VP	X	1-hr	X	18.8	9.2	20.3	9.6	21.8	18.7
10	S2UT w/ orig-unit	VP+mined	X	X	X	16.7	12.0	17.2	16.7	19.9	18.2
11	S2UT w/ orig-unit	VP+mined	✓	X	X	18.2	16.3	19.1	16.6	21.6	18.6
12	S2UT w/ norm-unit	VP+mined	X	1-hr	X	21.2	15.1	22.1	15.9	24.1	20.3
13	S2T + tf TTS	VP+mined	X	X	ASR	21.4	14.8	22.4	16.7	24.3	20.9
14	S2T + T2U	VP+mined	X	X	ASR	21.3	14.9	22.3	16.7	24.8	21.6
15	S2T (Wang et al., 2021c) + tf TTS	VP+EP+CVST	X	X	Oracle	26.0	27.3	28.1	27.7	-	-
16	S2T (Wang et al., 2021c) + T2U	VP+EP+CVST	X	X	Oracle	26.0	26.9	28.1	27.3	-	-
8	gt + tf TTS	X	X	X	X	88.0	80.7	87.2	77.3	82.0	68.6
9	gt + T2U	X	X	X	X	87.9	78.8	87.1	75.9	84.6	73.8
Table 5: BLEU scores (↑) from systems trained in a single run with the combination of VoxPopuli S2ST data (VP) (Wang et al., 2021c) and mined S2ST data (Duquenne et al., 2021) and evaluated on Europarl-ST (EP) (Iranzo-Sanchez et al., 2020) and CoVoST 2 (CVST) (Wang et al., 2020b) test sets. The S2T model in Wang et al. (2021c) is trained on more than 500 hrs of S2T data. The best results from S2UT with VP+mined data are highlighted in bold. (tgt spkemb: target speaker embedding, SN: speech normalization, gt: ground truth, tf: Transformer)
However, with only 10 minutes of paired multiple-to-single speaker speech data, we obtain norm-unit that improves S2UT model performance by 1.5 BLEU on average (2 vs. 3). The translation quality improves as we increase the amount of parallel data for training the speech normalizer. In the end, with 10 hours of finetuning data, we obtain an average 4.9 BLEU gain from the four language pairs compared to the basic setup (1 vs. 5).
On the other hand, compared with S2T+TTS systems that uses extra ASR models for converting speech to text for training the translation model (6, 7), our best textless S2ST systems (5) can perform similarly to text-based systems without the need of human annotations for building the ASR models.
We see that the MOS of S2UT systems trained with orig-unit is on average 0.85 lower than that of systems trained with norm-unit (1 vs. 5). We notice that the former often produces stuttering in the output speech, a potential cause to lower MOS. While worse audio quality may affect ASR-based
evaluation and lead to lower BLEU, we verify that this was not the case as the ASR models could still capture the content. We also see that the proposed textless S2ST system can produce audios with similar naturalness as Transformer TTS models (5 vs. 6).
S2ST with supervised data and mined data Next, we add the mined S2ST data for model training, and the results are summarized in Table 5. We apply the speech normalizer trained with 1-hr data, as it provides similar translation performance as a speech normalizer trained with 10-hr data in VoxPopuli-only experiments (4 vs. 5 in Table 4).
On the Europarl-ST test set, we see consistent trend across the S2UT models trained with norm-unit and the two baselines with orig-unit, where the proposed approach gives on average 3.9 BLEU improvement compared to the basic setup (10 vs. 12), indicating that the speech normalizer trained on VoxPopuli and Common Voice data can also be applied to audiios from different domains, e.g. Lib
riVox, where the mined data is collected. The addition of mined data with the proposed speech normalization technique achieves an average of 2.0 BLEU gain over four language directions (4 vs. 12).
We also examine model performance on the CoVoST 2 test set (Wang et al., 2020b) and see even larger improvements brought by mined data (10, 11, 12 vs. 4). One possible reason for this is that LibriVox is more similar to the domain of CoVoST 2 than that of Europarl-ST. With target speaker embedding, mined data improves S2ST by 7.1 BLEU on average (4 vs. 11). S2UT with norm-unit does not perform as well, and one explanation is that we select the best model based on the Europarl-ST dev set during model training.
Compared with S2T+TTS systems trained with text obtained from ASR, there is an average of 0.6 BLEU gap from our proposed system on EuroparlST test sets (12 vs. 14). As the En ASR model was trained on Libripeech (Panayotov et al., 2015), it can decode high quality text output for the mined data. We also list results from the S2T systems from Wang et al. (2021c)<sup>7</sup> (15, 16), which shows the impact of having oracle text and in-domain training data and serves as an upper bound for the textless S2ST system performance.
5.2 Analysis on the speech normalizer
We analyze norm-unit to understand how the speech normalization process helps improve S2UT performance. First, to verify that the process preserves the lexical content, we perform a speech resynthesis study as in Polyak et al. (2021). We use the VoxPopuli ASR test sets, run the unit-based vocoder with different versions of discrete units extracted from the audio as input, and compute word error rate (WER) of the audio output. In addition to comparing between norm-unit and reduced orig-unit, we list the WER from the original audio to demonstrate the quality of the ASR models and the gap caused by the unit-based vocoder.
We see from Table 6 that norm-unit from a speech normalizer finetuned on 1-hr data achieves similar WER as orig-unit, indicating that the normalization process does not change the content of the speech. In addition, we observe that norm-unit sequences are on average  $15\%$  shorter than reduced orig-unit sequences. We find that this is mainly due to the fact that the speech normalizer
WER (↓)	En	Es	Fr
original audio	14.2	15.5	18.5
reduced orig-unit	22.4	22.7	24.1
norm-unit (10-min)	23.5	25.3	31.7
norm-unit (1-hr)	21.2	20.5	24.6
norm-unit (10-hr)	22.0	25.3	24.2
Table 6: Speech resynthesis results on the VoxPopuli ASR test set.
UER (↓)	En	Es	Fr
reduced orig-unit	74.4	70.6	73.5
norm-unit (1-hr)	48.2	31.6	46.4
Table 7: Unit error rate (UER) between units extracted from 400 pairs of audios from the Common Voice dataset.
does not output units for the long silence in the audio, while reduced orig-unit encodes non-speech segments such as silence and background noises. Therefore, norm-unit is a shorter and cleaner target for training S2UT models.
Next, to examine that the speech normalizer reduces variations in speech across speakers, we sample 400 pairs of audios from Common Voice (Ardila et al., 2020) for En, Es and Fr, respectively. Each pair contains two speakers reading the same text prompt. Table 7 shows the unit error rate (UER) between the unit sequences extracted from the paired audios. We see that norm-unit has UER that is on average  $58\%$  of the UER of reduced origin, showing that norm-unit has less variations across speakers.
5.3 Analysis of mined data
Each pair of aligned speech in the mined data has an associated semantic similarity score. In experiments above, we set the score threshold as 1.06, and use all mined data with scores above it. Given
Figure 4: BLEU scores  $(\uparrow)$  on Europarl-ST Es-En test set from models trained with VoxPopuli and mined data filtered at different thresholds (t) for the similarity score.
7Models downloaded from https://github.com/facebookresearch/voxpopuli/.
the trade-off between the quality and quantity of mined data, we analyze how the S2ST performance changes with the threshold set in mined data selection. Figure 4 demonstrates BLEU scores on Europarl-ST Es-En test set from S2UT systems trained with 1-hr norm-unit. The mined data is useful at different thresholds given its gains over the model trained without mined data. As we increase the threshold from 1.06 to 1.07, the performance drops due to less training data.
We present a textless S2ST system that can be trained with real target speech data. The key to the success is a self-supervised unit-based speech normalization process, which reduces variations in the multi-speaker target speech while retaining the lexical content. To achieve this, we take advantage of self-supervised discrete representations of a reference speaker speech and perform CTC finetuning with a pre-trained speech encoder. The speech normalizer can be trained with one hour of parallel speech data without the need of any human annotations and works for speech in different recording conditions and in different languages. We conduct experiments on the VoxPopuli S2ST dataset and the mined speech data to empirically demonstrate its usefulness in improving S2ST system translation quality for the first time. In the future, we plan to investigate more textless approaches to improve model performance such as self-supervised pretraining. All the experiments and ASR evaluation are conducted with public datasets or open-sourced models.
The authors would like to thank Adam Polyak and Felix Kreuk for initial discussions on accent normalization.
Nagaraj Adiga, Yannis Pantazis, Vassilis Tsiaras, and Yannis Stylianou. 2019. Speech enhancement for noise-robust speech synthesis using wasserstein gan. In INTERSPEECH, pages 1821-1825.
Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henrtty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. 2020. Common voice: A massively-multilingual speech corpus. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4218-4222.
Alexei Baevski, Michael Auli, and Abdelrahman Mohamed. 2019. Effectiveness of self-supervised pretraining for speech recognition. arXiv preprint arXiv:1911.03912.
Claudio Bendazzoli, Annalisa Sandrelli, et al. 2005. An approach to corpus-based interpreting studies: developing epic (european parliament interpreting corpus). Proceedings of Challenges of Multidimensional Translation.
Alexandre Bérard, Olivier Pietquin, Christophe Servan, and Laurent Besacier. 2016. Listen and translate: A proof of concept for end-to-end speech-to-text translation. arXiv preprint arXiv:1612.01744.
Cassia Valentini Botinhao, Xin Wang, Shinji Takaki, and Junichi Yamagishi. 2016. Speech enhancement for a noise-robust text-to-speech synthesis system using deep recurrent neural networks. In Interspeech 2016, pages 352-356.
Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. 2018. Voxceleb2: Deep speaker recognition. arXiv preprint arXiv:1806.05622.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Paul-Ambroise Duquenne, Hongyu Gong, and Holger Schwenk. 2021. Multimodal and multilingual embeddings for large-scale speech mining. Advances in Neural Information Processing Systems, 34.
Andrew Gibiansky, Sercan Arik, Gregory Diamos, John Miller, Kainan Peng, Wei Ping, Jonathan Raiman, and Yanqi Zhou. 2017. Deep voice 2: Multi-speaker neural text-to-speech. Advances in neural information processing systems, 30.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pages 369-376.
Tomoki Hayashi, Ryuichi Yamamoto, Katsuki Inoue, Takenori Yoshimura, Shinji Watanabe, Tomoki Toda, Kazuya Takeda, Yu Zhang, and Xu Tan. 2020. Espnet-tts: Unified, reproducible, and integratable open source end-to-end text-to-speech toolkit. In ICASSP 2020-2020 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 7654-7658. IEEE.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. HuBERT: Self-supervised
speech representation learning by masked prediction of hidden units. arXiv preprint arXiv:2106.07447.
Javier Iranzo-Sanchez, Joan Albert Silvestre-Cerdà, Javier Jorge, Nahuel Rosello, Adrià Giménez, Albert Sanchis, Jorge Civera, and Alfons Juan. 2020. Europarl-st: A multilingual corpus for speech translation of parliamentary debates. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8229-8233. IEEE.
Keith Ito and Linda Johnson. 2017. The LJ speech dataset. https://keithito.com/LJ-Speech-Dataset/.
Ye Jia, Michelle Tadmor Ramanovich, Tal Remez, and Roi Pomerantz. 2021. Translatotron 2: Robust direct speech-to-speech translation. arXiv preprint arXiv:2107.08661.
Ye Jia, Ron J Weiss, Fadi Biadsy, Wolfgang Macherey, Melvin Johnson, Zhifeng Chen, and Yonghui Wu. 2019. Direct speech-to-speech translation with a sequence-to-sequence model. Proc. Interspeech 2019, pages 1123-1127.
Takatomo Kano, Sakriani Sakti, and Satoshi Nakamura. 2021. Transformer-based direct speech-to-speech translation with transcoder. In 2021 IEEE Spoken Language Technology Workshop (SLT), pages 958-965. IEEE.
Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu-Anh Nguyen, Morgane Rivière, Abdelrahman Mohamed, Emmanuel Dupoux, et al. 2021. Text-free prosody-aware generative spoken language modeling. arXiv preprint arXiv:2109.03264.
Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. 2020. HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 33.
Felix Kreuk, Adam Polyak, Jade Copet, Eugene Kharitonov, Tu-Anh Nguyen, Morgane Rivière, Wei Ning Hsu, Abdelrahman Mohamed, Emmanuel Dupoux, and Yossi Adi. 2021. Textless speech emotion conversion using decomposed and discrete representations. arXiv preprint arXiv:2111.07402.
Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66-75.
Kushal Lakhotia, Evgeny Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Adelrahman Mohamed, et al. 2021. Generative spoken language modeling from raw audio. arXiv preprint arXiv:2102.01192.
Alon Lavie, Alex Waibel, Lori Levin, Michael Finke, Donna Gates, Marsal Gavalda, Torsten Zeppenfeld, and Puming Zhan. 1997. JANUS-III: Speech-to-speech translation in multiple languages. In 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 1, pages 99-102. IEEE.
Ann Lee, Peng-Jen Chen, Changhan Wang, Jiatao Gu, Xutai Ma, Adam Polyak, Yossi Adi, Qing He, Yun Tang, Juan Pino, et al. 2021. Direct speech-to-speech translation with discrete units. arXiv preprint arXiv:2107.05604.
Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu. 2019. Neural speech synthesis with transformer network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6706-6713.
Satoshi Nakamura, Konstantin Markov, Hiromi Nakaiwa, Gen-ichiro Kikui, Hisashi Kawai, Takatoshi Jitsuhiro, J-S Zhang, Hirofumi Yamamoto, Eiichiro Sumita, and Seiichi Yamamoto. 2006. The ATR multilingual speech-to-speech translation system. IEEE Transactions on Audio, Speech, and Language Processing, 14(2):365-376.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. *fairoseq: A fast, extensible toolkit for sequence modeling*. In *Proceedings of NAACL-HLT* 2019: Demonstrations.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an ASR corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206-5210. IEEE.
Kyubyong Park and Thomas Mulc. 2019. Css10: A collection of single speaker speech datasets for 10 languages. arXiv preprint arXiv:1903.11269.
Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Abdelrahman Mohamed, and Emmanuel Dupoux. 2021. Speech resynthesis from discrete disentangled self-supervised representations. arXiv preprint arXiv:2104.00355.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186-191.
Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. 2020. Fastspeech 2: Fast and high-quality end-to-end text to speech. arXiv preprint arXiv:2006.04558.
Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. 2019. Fastspeech: Fast, robust and controllable text to speech. arXiv preprint arXiv:1905.09263.
Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, Armand Joulin, and Angela Fan. 2021. CCMatrix: Mining billions of high-quality parallel sentences on the web. In ACL, page 6490-6500.
Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. 2018. Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4779-4783. IEEE.
Gabriel Synnaeve, Qiantong Xu, Jacob Kahn, Tatiana Likhomanenko, Edouard Grave, Vineel Pratap, Anuroop Sriram, Vitaliy Liptchinsky, and Ronan Collobert. 2019. End-to-end ASR: from supervised to semi-supervised learning with modern architectures. arXiv preprint arXiv:1911.08460.
Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura. 2019. Speech-to-speech translation between untranscribed unknown languages. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 593-600. IEEE.
Hitomi Tohyama, Shigeki Matsubara, Koichiro Ryu, N Kawaguch, and Yasuyoshi Inagaki. 2004. Ciair simultaneous interpretation corpus. In Proc. Oriental COCOSDA.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. 2017. Neural discrete representation learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 6309-6318.
Ehsan Variani, Xin Lei, Erik McDermott, Ignacio Lopez Moreno, and Javier Gonzalez-Dominguez. 2014. Deep neural networks for small footprint text-dependent speaker verification. In 2014 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 4052-4056. IEEE.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Changhan Wang, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Ann Lee, Peng-Jen Chen, Jiatao Gu, and Juan Pino. 2021a. fairseq  $\hat{\mathbf{s}}^2$ : A scalable and integrable speech synthesis toolkit. arXiv preprint arXiv:2109.06912.
Changhan Wang, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Ann Lee, Peng-Jen Chen, Jiatao Gu, and Juan Pino. 2021b. fairseq s^2: A scalable and integrable speech synthesis toolkit. arXiv preprint arXiv:2109.06912.
Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson,
Juan Pino, and Emmanuel Dupoux. 2021c. VoxPopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 993-1003, Online. Association for Computational Linguistics.
Changhan Wang, Yun Tang, Xutai Ma, Anne Wu, Dmytro Okhonko, and Juan Pino. 2020a. fairseq s2t: Fast speech-to-text modeling with fairseq. In Proceedings of the 2020 Conference of the Asian Chapter of the Association for Computational Linguistics (AACL): System Demonstrations.
Changhan Wang, Anne Wu, and Juan Pino. 2020b. CoV-oST 2: A massively multilingual speech-to-text translation corpus.
Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al. 2017. Tacotron: Towards end-to-end speech synthesis. arXiv preprint arXiv:1703.10135.
Marcely Zanon Boito, William Havard, Mahault Garnerin, Éric Le Ferrand, and Laurent Besacier. 2020. MaSS: A large and clean multilingual corpus of sentence-aligned spoken utterances extracted from the Bible. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 6486-6493, Marseille, France. European Language Resources Association.
Chen Zhang, Xu Tan, Yi Ren, Tao Qin, Kejun Zhang, and Tie-Yan Liu. 2020. UWSpeech: Speech to speech translation for unwritten languages. arXiv preprint arXiv:2006.07926.
A mHuBERT Training details
Table 8 lists the details for the three iterations of mHuBERT training.
iteration	target features	K-means
1	MFCC	100
2	6-th layer from the first iteration	500
3	9-th layer from the second iteration	500
B Unit-based Vocoder
Table 9 shows the resynthesis performance of the unit-based vocoder of each language. The WER on the original audio indicates the quality of the open-sourced ASR model we use for evaluation. The WER difference between original audio and orig-unit shows the quality of the vocoder, and the difference between orig-unit and reduced orig-unit shows the further impact brought by the duration prediction module.
Table 8: Setup for the target labels used in mHuBERT training.
WER (↓)	En	Es	Fr
original audio	2.0	8.4	24.0
orig-unit	2.8	12.0	29.3
reduced orig-unit	3.4	11.9	31.3
C Text-to-Unit (T2U)
Table 10 lists the WER of the audios generated by the T2U model, which is used in generating the reference target units for speech normalizer training. As the T2U model is trained with reduced unit sequences as the target, during synthesis, we apply the unit-based vocoder with duration prediction. We can see that T2U with a unit-based vocoder can produce high quality audio and can serve as another option of TTS.
Table 9: WER on the TTS dev sets (LJSpeech for En, and CSS10 for Es and Fr) of the audiios resynthesized from units.
WER (↓)	En	Es	Fr
original audio	2.0	8.4	24.0
T2U	4.2	9.1	24.4
Table 11 lists the best hyper-parameters for training the speech normalizers for the three languages and three data setups, respectively. All models are trained on 8 GPUs with a batch size of 100-second (maximum total input audio length).
Table 12 lists the best learning rate tuned on the dev set for the S2UT experiments listed in Table 4 and Table 5. All models are trained on 8 GPUs with a total batch size of  $160\mathrm{k}$  tokens and dropout of 0.3, except for Es-En experiment ID 1 which uses 0.1.
Table 10: WER on the TTS dev sets (LJSpeech for En, and CSS10 for Es and Fr).
language	duration	learning rate	mask prob	mask channel prob
En	10-min	0.00003	0.75	0.75
En	1-hr	0.00005	0.5	0.5
En	10-hr	0.0001	0.5	0.75
Es	10-min	0.00003	0.5	0.75
Es	1-hr	0.00003	0.5	0.25
Es	10-hr	0.00005	0.5	0.5
Fr	10-min	0.00003	0.5	0.5
Fr	1-hr	0.00005	0.5	0.25
Fr	10-hr	0.00005	0.5	0.25