Simple and Effective Unsupervised Speech Translation
Year: 2023

Abstract:
The amount of labeled data to train models for speech tasks is limited for most languages, however, the data scarcity is exacerbated for speech translation which requires labeled data covering two different languages. To address this issue, we study a simple and effective approach to build speech translation systems without labeled data by leveraging recent advances in unsupervised speech recognition, machine translation and speech synthesis, either in a pipeline approach, or to generate pseudo-labels for training end-to-end speech translation models. Furthermore, we present an unsupervised domain adaptation technique for pre-trained speech models which improves the performance of downstream unsupervised speech recognition, especially for low-resource settings. Experiments show that unsupervised speech-to-text translation outperforms the previous unsupervised state of the art by 3.2 BLEU on the LibriTrans benchmark, on CoVoST 2, our best systems outperform the best supervised end-to-end models (without pre-training) from only two years ago by an average of 5.0 BLEU over five X-En directions. We also report competitive results on MuST-C and CVSS benchmarks.

heading: 1 Introduction
text: Training supervised speech systems requires large amounts of labeled data which is often not available for all but a small fraction of the over 7,000 languages spoken around the world (Lewis et al., 2022). Despite much recent effort in creating speech translation corpora (Di Gangi et al., 2019a; Wang et al., 2021b), only a few dozen language directions are covered. The lack of labeled training data is even more acute for speech translation because it requires aligned labeled data in two languages which increases the effort to create such datasets. This poses the question of whether speech translation systems can be built using less labeled data or no labeled data at all.
Recent work on unsupervised speech recognition has achieved performance that can enable useful systems using no labeled data (Yeh et al., 2019; Liu et al., 2018; Chen et al., 2019; Baevski et al., 2021; Liu et al., 2022a), enabled in large part by the advances in self-supervised speech representation learning (Schneider et al., 2019; Baevski et al., 2020). These techniques were also used to build unsupervised text-to-speech systems (Liu et al., 2022b). Similarly, unsupervised text-to-text machine translation has shown great promise for certain language directions (Conneau et al., 2018; Lample et al., 2018; Artetxe et al., 2018).
In this paper, we study a method to build endto-end unsupervised speech-to-text and speech-tospeech translation systems trained on synthetic training data obtained by cascading existing unsupervised techniques: we first transcribe speech utterances in the source language using unsupervised speech recognition (Baevski et al., 2021; Liu et al., 2022a), then translate the resulting transcription using unsupervised machine translation (Lample et al., 2018; Artetxe et al., 2018; Liu et al., 2020), and finally synthesize the translation into a target language speech utterance using unsupervised speech synthesis (Liu et al., 2022b). We also consider applying the pipeline directly at inference time. Our approach benefits from the use of self-supervised speech models (Baevski et al., 2020; Liu et al., 2020) and to further improve performance, we present a technique to adapt existing self-supervised models to the target domain.

heading: 2 Background
text: Unsupervised speech recognition. Liu et al. (2018) presents some of the earliest work on unsupervised phoneme recognition and their work applies adversarial training. Wav2vec-U (Baevski et al., 2021) effectively applied self-supervised speech representations, introduced a new evaluation metric and compared to state-of-the-art super-
10771
vised systems trained on large amounts of labeled data. Wav2vec-U 2.0 (Liu et al., 2022a) simplifies audio-side pre-processing and improves accuracy through better architecture as well as better training objective. Lin et al. (2022) shows that out-ofdomain speech pre-training or out-of-domain text data hurts the training robustness of Wav2vec-U models, especially under low-resource settings.
Unsupervised speech synthesis. Recent work has demonstrated unsupervised speech synthesis systems to be able to achieve comparable performance to supervised systems (Liu et al., 2022b; Ni et al., 2022). The systems are trained on data resulting from labeling speech audio data with unsupervised speech recognition models and training text-to-speech models on the resulting models.
Unsupervised machine translation. Lample et al. (2018) and Artetxe et al. (2018) built the first fully unsupervised machine translation (MT) systems by exploiting cross-lingual similarity of representations in multilingual sequence-to-sequence models, as well as back-translation for further refinements of the initial models. mBART (Liu et al., 2020) used a similar model architecture and training process to build unsupervised MT models, but it utilized a larger-scale multilingual text corpus (Conneau et al., 2020) and an updated noising strategy for pre-training with denoising autoen-
coder objective.
End-to-end speech translation. End-to-end sequence-to-sequence modeling has witnessed increased applications in speech-to-text translation (Duong et al., 2016; Bérard et al., 2016; Weiss et al., 2017; Bansal et al., 2017; Vila et al., 2018; Di Gangi et al., 2019b; Ren et al., 2020; Li et al., 2021) and speech-to-speech translation (Jia et al., 2019; Kano et al., 2021; Jia et al., 2022a). Compared to cascaded systems, end-to-end speech translation models have simpler pipeline and lower inference latency. It is shown that recent end-to-end speech-to-text translation (S2TT) models perform comparably to the cascaded counterparts on the well-established MuST-C benchmark (Bentivogli et al., 2021). Given the scarcity of speech translation corpora, there are recent attempts on building end-to-end S2TT models under low-resource settings (Bansal et al., 2018, 2019; Cheng et al., 2021) or unsupervised settings (Chung et al., 2019).

heading: 3 Methods
text: Figure 1 provides an overview of our proposed approach to unsupervised speech-to-text translation (S2TT) and speech-to-speech translation (S2ST). We leverage a cascade of unsupervised models to produce pseudo-labels for end-to-end S2TT and S2ST model training. To mitigate language and
domain mismatch in speech pre-training (wav2vec 2.0), we finetune wav2vec 2.0 models using unlabeled in-domain speech data, and then use the adapted models to build downstream speech recognition models.

heading: 3.1 Unsupervised Cascaded Pseudo-Labeling
text: We cascade unsupervised speech recognition (ASR), unsupervised text de-normalization (TDN) and unsupervised machine translation (MT) models to produce pseudo-labels for S2TT. For S2ST, we additionally apply unsupervised speech synthesis (TTS) models to MT model outputs to obtain synthesized target speech.
Unsupervised ASR. We adopt wav2vec-U 2.0 (Liu et al., 2022a), which learns a mapping from self-supervised speech representations to phonemes via adversarial training and decodes phonemes into words via a weighted finite state transducer (Mohri, 1997). To improve adversarial training stability and suppress overfitting in the low-resource settings, we add Gaussian noise to the frozen input features X
X ′ = X +N (0, σ2) as well as R-Drop regularization (Wu et al., 2021) to the logit outputs of the generator
Lrdp = 1 2 DKL(G1(X ′) || G2(X ′))
+ 1 2 DKL(G2(X ′) || G1(X ′))
where G1 and G2 are two generator instances with different dropout masks, and DKL is the KullbackLeibler (KL) divergence. We add weighted αLrdp to the wav2vec-U 2.0 objective function, where α is a hyper-parameter. After adversarial learning, we follow Baevski et al. (2021) to perform selftraining with a Hidden Markov Model (HMM), and fine-tune the adapted wav2vec 2.0 model again with the CTC objective on the HMM labels. We denote the final ASR model as “w2vu2-CTC”.
Unsupervised MT. We adopt mBART (Liu et al., 2020), which has a Transformer architecture (Vaswani et al., 2017) with model parameters shared across all training languages. It first obtains initial cross-lingual alignments for all languages via a denoising autoencoder objective (Vincent et al., 2010), and then refines the alignments for one specific language pair via bidirectional online back-translation on that pair of languages. We denote this model as “mBART-OBT”.
Unsupervised TDN. ASR models decode normalized spoken-form texts, which have no case or punctuation (except hyphen and apostrophe). MT models, however, encode unnormalized writtenform texts that have case and punctuation. This discrepancy leads to quality degradation when we cascade the two models directly for pseudo-labeling. To mitigate the mismatch, we de-normalize ASR model outputs into their unnormalized written form before feeding them into MT models. The text denormalizer is a mBART model pre-trained with denoising autoencoder objective and fine-tuned with paired data of raw text (output) and its normalized version (input).
Unsupervised TTS. We follow Liu et al. (2022b) to produce phoneme labels for unlabeled speech data with wav2vec-U 2.0, and then train an autoregressive Transformer TTS model (Li et al., 2019) on the pseudo-labeled data. For wav2vec-U 2.0, we perform HMM-based self-training and finetune pre-trained wav2vec 2.0 model with HMM phoneme labels. To alleviate under-generation and over-generation issues in autoregressive models, we add R-Drop style consistency loss
Lc = ||PEOS1 (X)− PEOS2 (X)||1
to the objective function (weighted by a hyperparameter α) for better end-of-sentence (EOS) predictions, where PEOS1 and PEOS2 are two EOS predictions on the same input X with different dropout masks.

heading: 3.2 Unsupervised Adaptation of wav2vec 2.0 Pre-trained Models
text: Next, we present a method to improve performance when the domain of the data used for self-supervised pre-training differs from the downstream task domain which is often the case for lowresource languages. Specifically, we adapt out-ofdomain or out-of-language wav2vec 2.0 models to the domain and language of interest by fine-tuning the entire wav2vec 2.0 models on discrete labels obtained from unlabeled in-domain data using the CTC objective (Graves et al., 2006).
To obtain discrete labels, we first collect all the wav2vec 2.0 speech representations for the training data, and perform k-means clustering to identify K clusters. Then for each utterance, we label each of its T speech representation frames xt by the corresponding cluster ids yt ∈ {1, ...,K}, where t ∈ {1, ..., T}. Finally, we merge identical
consecutive yt to obtain the final labels y′t′ , where t′ ∈ {1, ..., T ′} and T ′ ≤ T .
After unsupervised fine-tuning with discrete labels, we discard the output projection layer used for the CTC objective, and use the resulting wav2vec 2.0 trunk instead of the original wav2vec 2.0 model in the downstream tasks. The adapted models are used to extract speech representations for wav2vecU 2.0 models, as well as pre-train encoders of the CTC models in wav2vec-U self-training.

heading: 3.3 End-to-end Model Training with Pseudo-labels
text: After obtaining pseudo-labels from the cascade of unsupervised models, we train end-to-end S2TT and S2TT models with supervised objectives on these pseudo-labels. For end-to-end S2TT, we adopt the model architecture in Li et al. (2021), which we denote as “w2v2-mBART”. We pretrain its encoder by the unsupervised ASR model, w2vu2-CTC, and pre-train its decoder by the unsupervised MT model, mBART-OBT. For end-to-end S2ST, we adopt a variant of Translatotron 2 (Jia et al., 2022a), Translatotron2+ (Inaguma et al., 2022), which has an additional encoder in between Translatotron 2’s two decoders, and replaces Translatotron 2’s second decoder with an autoregressive Transformer decoder (Li et al., 2019). Similar to w2v2-mBART, we pre-train Translatotron2+’s first encoder and first decoder by w2vu2-CTC and mBART-OBT, respectively.

heading: 4 Experimental Setup
text: We evaluate our translation models on 5 directions into English (Fr-En, Es-En, Ru-En, Et-En and LvEn) and 3 directions out of English (En-Es, En-Ru and En-Fr). The 5 non-English languages are from 4 different Indo-European language family subgroups: Romance (Fr and Es), Slavic (Ru), Uralic (Et) and Baltic (Lv). For the X-En directions, we evaluate S2TT models on CoVoST 2 (Wang et al., 2021b) and evaluate S2ST models on CVSS-C (Jia et al., 2022b), which adds synthetic target speech to CoVoST 2 with a single canonical speaker voice. For the En-X directions, we only evaluate S2TT models. We use MuST-C (Di Gangi et al., 2019a) for En-Es and En-Ru, as well as Libri-Trans (Kocabiyikoglu et al., 2018) for En-Fr. For Libri-Trans, we follow Chung et al. (2019) to combine validation set and test set for evaluation.
Speech pre-training. We use robust wav2vec 2.0 (Hsu et al., 2021) for English speech, which is trained on datasets from multiple domains. For non-English speech, we adapt open-source VoxPopuli1 (Wang et al., 2021a) models by CTC finetuning with 1024 discrete labels (Fr, Es and Ru) or 128 discrete labels (Et and Lv). We use monolingual VoxPopuli models for Fr and Es, and multilingual models of similar languages for Ru, Et and Lv (Slavic, Uralic and Baltic languages, respectively). We extract speech representations from the 15-th layer of the original wav2vec 2.0 models for computing discrete labels.
Speech recognition. For wav2vec-U 2.0 models, we extract speech representations from the 19-th (15-th) layer of the adapted (original) wav2vec 2.0 models. We increase the dropout on the batch normalized input features to 0.2. We set σ = 0.1 for input Gaussian noise and α = 1.0 for R-Drop regularization. For wav2vec-U 2.0 loss weights, we set η = 3 and choose λ, γ and δ from 1.0 / 1.5, 1.5 / 2.5 and 0.3 / 0.5, respectively. For text data, we use open web crawled corpus, CC-100 (Conneau et al., 2020), which is created with little curation and has large language coverage. For supervised baselines, we fine-tune adapted wav2vec 2.0 models with CTC objective on labeled data, which we denote as “w2v2-CTC”.
Machine translation. We use CC-100 (Conneau et al., 2020) to train bilingual mBART large models for each language pair. For bidirectional online back-translation, we use the same CC100 data and follow Liu et al. (2020) to apply 99% vocabulary masking for the first 500 updates. For supervised baselines, we fine-tune mBART models with labeled data, which we denote as “mBART-FT”.
Speech synthesis. We train Transformer models (with Lc weight α = 1.0) on CVSS-C target speech from the It-En direction to avoid content overlaps with the selected 5 directions. For graphemeto-phoneme conversion, we employ g2pE (Park, 2019) for English texts and Phonemizer (Bernard, 2015) with espeak-ng2 backend for texts in other languages. We resample audios to 22,050Hz and extract log-Mel spectrogram with FFT size 1024, window length 1024 and hop length 256.
1https://github.com/facebookresearch/voxpopuli 2https://github.com/espeak-ng/espeak-ng
End-to-end speech translation. For bilingual S2TT, we pre-train its encoder/decoder with w2vu2-CTC/mBART-OBT for unsupervised models, or with w2v2-CTC/mBART-FT for supervised models that leverage pre-training. To alleviate overfitting in low-resource settings (Ru-En, Et-En and Lv-En), we duplicate training examples and equip them with 2 different pseudo-labels from mBARTOBT beam search decoding. For multilingual S2TT and S2ST, we pre-train speech encoder with XLS-R 0.3B (Babu et al., 2021), and pre-train text decoder with mBART-OBT from the En-Fr direction.
Checkpoint selection and averaging. For unsupervised ASR, we adopt the unsupervised metric
in Baevski et al. (2021) and average the best 2 checkpoints in the same run. For unsupervised MT and unsupervised TTS, we average the last 5 checkpoints. For end-to-end S2TT/S2ST, we sort checkpoints by losses on the pseudo-labeled validation set and average the best 5 checkpoints.
Automatic evaluation of speech outputs. Following a common practice, we first transcribe English speech outputs from the TTS or S2ST model with an open-source English ASR model3, and then calculate WER or BLEU on the ASR transcription for automatic evaluation scores.

heading: 5 Results

heading: 5.1 X-En Speech-to-Text Translation
text: For X-En S2TT, we consider models trained for a single language direction (bilingual) and models covering multiple directions (multilingual). Results are reported on five translation directions into English of the CoVoST 2 benchmark and we focus on end-to-end systems but we also consider a cascade of unsupervised models. Supervised models are purely trained on labeled data without pre-training, while as supervised models with pre-training use wav2vec and mBART models, unsupervised models also use pre-trained models but no labeled data.
Table 1 shows that unsupervised end-to-end models outperform the supervised baselines by 5.0
3https://github.com/facebookresearch/fairseq/tree/main/ examples/wav2vec (“Wav2Vec 2.0 Large (LV-60) + Self Training”)
BLEU on average over the five translation directions of the bilingual setup. The supervised models represent the best supervised end-to-end models from two years ago. These improvements are due to advances in unsupervised modeling as well as self-supervised pre-training. The supervised models with pre-training perform generally far above the unsupervised models and shows that there is potential to improve unsupervised speech translation in the future.
The cascaded unsupervised setup performs better than the end-to-end approach for directions with little synthetic training data such as Ru-En, Et-En and Lv-En. This is because end-to-end models are trained on datasets comprising as little as two hours of synthetic speech translation data on which they overfit. Cascaded unsupervised models do not suffer under this issue because they exploit more text for unsupervised machine translation (Table 7).
Supervised learning with pre-training for the bilingual setup performs better than the multilingual setup because only a single translation direction needs to be modeled and because the mBART model was pre-trained on 50 languages while as
only a single language is being used in the X-En setup.

heading: 5.2 En-X Speech-to-Text Translation
text: For bilingual En-X S2TT, we compare our unsupervised models to the previous state of the art (Chung et al., 2019) on Libri-Trans (En-Fr) and we also evaluate them on the MuST-C benchmark for EnEs and En-Ru directions. Table 2 shows the test BLEU of our models and the baselines on both benchmarks. On Libri-Trans, our best system outperforms the previous state of the art, an alignmentbased cascaded system, by 3.2 BLEU (Chung et al., 2019). On MuST-C, our models also achieve competitive results in this high-resource setting of around 500 hours of training data, with 3.4 BLEU and 5.5 BLEU behind the supervised baselines on En-Es and En-Ru, respectively.

heading: 5.3 X-En Speech-to-Speech Translation
text: To train a multilingual X-En speech-to-speech translation model, we combine pseudo-labeled bilingual data for multiple translation directions and use the Translatotron2+ architecture, a variant
of Translatotron 2. We build supervised Translatotron2+ baselines with and without pre-training and evaluate on the CVSS-C benchmark. Table 3 shows that the best unsupervised system is on average only 0.8 BLEU below the supervised baseline. We believe that the unsupervised approach is less effective for speech-to-speech translation compared to speech-to-text translation because of the increased error accumulation in the synthetic data creation process due to the addition of the unsupervised speech synthesis component to which we input unsupervised translation output which in turn is based on unsupervised speech recognition transcriptions. Similarly to speech-to-text translation, the cascaded unsupervised model performs better than the end to end approach and this is most prominent for low-resource directions.

heading: 5.4 Speech Pre-training
text: We evaluate the effectiveness of the unsupervised adaptation technique of wav2vec 2.0 models (§3.1) on the five non-English languages, which have less training data than English. We train wav2vec-U 2.0 models on CoVoST 2 with features extracted from three different wav2vec 2.0 models and their
adapted versions: 1) Out-of-domain models, “VoxPopuli” (Wang et al., 2021a), that are trained with data in the same language (for Fr and Es) or similar languages (for Ru, Et and Lv) from the same language family subgroup; 2) a massively multilingual model for 128 languages, “XLS-R” (Babu et al., 2021), whose training data contains CoVoST 2; 3) a multi-domain English model, “robust wav2vec 2.0” (Hsu et al., 2021), where the target languages are unseen. We report validation PER on Viterbi predictions in Table 4. Speech pre-training on mismatched domains or languages (“VoxPopuli” and “robust wav2vec 2.0”) leads to training convergence failure on three low-resource languages (Ru, Et and Lv). The two languages with the least amount of data, Et and Lv, even fail with in-domain multilingual pre-training. Unsupervised adaptation significantly improves training convergence and model performance for all the 3 scenarios of speech pretraining. In an example worst case scenario, Et-En wav2vec-U 2.0 model is successfully trained with only 3 hours of Et speech data and features from an adapted out-of-language out-of-domain wav2vec 2.0 model (“robust wav2vec 2.0”).

heading: 5.5 Speech Recognition
text: Next, we evaluate the performance of unsupervised speech recognition in our setting. We decode our pre-trained supervised baselines (“w2v2-CTC”) and unsupervised models (“w2vu2-CTC”) with 4- gram language model. They are compared with previous un-pre-trained supervised baselines (Wang et al., 2020) on CoVoST 2 and MuST-C (for En), whose results (test WER) can be found in Table 5. We see that our unsupervised end-to-end models outperform un-pre-trained supervised baselines on average over the six languages, with an average 16.9 WER reduction over the supervised one. Unsupervised ASR works best for languages with little labeled data due to the use of pre-trained features and advances in unsupervised algorithms.

heading: 5.6 Speech Synthesis
text: In our unsupervised setting, the target speech data does not share the same domain as the source one. This realistic setting leads to training-inference time domain mismatch on TTS models. We evaluate the effects of this mismatch by a re-synthesis task on 3 different datasets: CVSS-C (from It-En), Libri-Trans and MuST-C. We synthesize speech using validation texts and report WER on the ASR transcription of the synthesized speech. To quan-
tize domain similarity, we follow Lin et al. (2022) to compute Jensen–Shannon divergence (“JSD”) on 4-gram phoneme distributions, where low JSD suggests high similarity. Table 6 shows the results. We see that both supervised and unsupervised models have higher WER on less similar domains (LibriTrans and MuST-C).

heading: 5.7 Machine Translation
text: We evaluate our unsupervised models (“mBARTOBT”) on the CoVoST 2, MuST-C and LibriTrans benchmarks with test BLEU. For comparison, we also build supervised Transformer baselines (“Transformer”) and supervised mBART baselines (“mBART-FT”). Results are shown in Table 7. We observe that our unsupervised models outperform supervised baselines by 12.1 BLEU on average over the eight considered translation directions. They are behind supervised baselines by only 2.2 BLEU on average. In contrast to supervised baselines that leverage in-domain paired data, the unsupervised models use unpaired CC100 data, which is noisy web data.

heading: 5.8 Text De-normalization
text: We verify the effectiveness of text de-normalization (TDN) by ablating it in the unsupervised cascaded pipeline. In Table 8, we show test BLEU calculated on either raw text (BLEUraw) or normalized text (BLEUnorm) for the ablation. We see that TDN improves BLEUraw greatly by 4.1 on average over all the directions. From the improvements on BLEUnorm, we conclude that TDN not only recovers case and punctuation, but also improves translation of the content.

heading: 6 Conclusion
text: In this paper, we present a simple and effective approach to unsupervised speech-to-text translation (S2TT) and speech-to-speech translation (S2ST). Our S2TT systems outperform the previous state of the art on Libri-Trans by 3.2 BLEU as well as the best supervised end-to-end models (without pretraining) on CoVoST 2 from only two years ago by an average of 5.0 BLEU over five translation directions into English. Our S2TT and S2ST systems also perform competitively on the MuST-C and CVSS-C benchmarks.

heading: 7 Limitations
text: Our unsupervised speech recognition approach requires tools to phonemize text for the language of interest. Phonemizers are not available for all languages and this presents a bottleneck. To address this, future work may develop phonemizers for more languages, explore phonemization approaches that generalize across languages, or wav2vec-U 2.0 model training with graphemic text units such as letters.
We train bilingual unsupervised machine translation models with 2.1B English sentences and at least 46M sentences for the non-English language. For extremely low-resource languages, collecting millions of sentences for model training can be challenging. The feasibility of mBART-based online back-translation approach in this setup remains to be validated.

heading: Acknowledgments
text: We thank Alexei Baevski, Andy Chung, Alexis Conneau, Hongyu Gong, Jiatao Gu and Sravya Popuri for helpful discussions.

heading: A Appendix
text: A.1 Comparison of our CVSS-C supervised baseline to previous work
For evaluation of CVSS-C models, we use an open-source English ASR model4 to transcribe translated speech for BLEU calculation. The previous work (Jia et al., 2022b), however, used transcription from a proprietary ASR model which we do not have access to. As a result, BLEU numbers reported for our model and the previous work are not directly comparable, but the small difference suggests that the two models perform roughly similarly.
A.2 Data Overview for Supervised Learning and Unsupervised Learning
4https://github.com/facebookresearch/fairseq/tree/main/ examples/wav2vec (“Wav2Vec 2.0 Large (LV-60) + Self Training”)
Table 10 provides an overview for the speech and text data used in supervised learning and unsupervised learning.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: A1. Did you describe the limitations of your work?
Left blank.
A2. Did you discuss any potential risks of your work? Left blank.
A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
A4. Have you used AI writing assistants when working on this paper? Left blank.

heading: B Did you use or create scientific artifacts?
text: Left blank.
B1. Did you cite the creators of artifacts you used? Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.

heading: C Did you run computational experiments?
text: Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Left blank.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.

heading: D Did you use human annotators (e.g., crowdworkers) or research with human participants?
text: Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Left blank.

heading: Future Work
text: To address this, future work may develop phonemizers for more languages, explore phonemization approaches that generalize across languages, or wav2vec-U 2. 0 model training with graphemic text units such as letters. We train bilingual unsupervised machine translation models with 2. 1B English sentences and at least 46M sentences for the non-English language. For extremely low-resource languages, collecting millions of sentences for model training can be challenging. The feasibility of mBART-based online back-translation approach in this setup remains to be validated