Speech-to-Speech Translation for a Real-world Unwritten Language
Year: 2023

Abstract:
We study speech-to-speech translation (S2ST) that translates speech from one language into another language and focuses on building systems to support languages without standard text writing systems. We use English↔Taiwanese Hokkien as a case study, and present an endto-end solution from training data collection, modeling choices to benchmark dataset release. First, we present efforts on creating human annotated data, automatically mining data from large unlabeled speech datasets, and adopting pseudo-labeling to produce weakly supervised data. On the modeling, we take advantage of recent advances in applying self-supervised discrete representations as target for prediction in S2ST and show the effectiveness of leveraging additional text supervision from Mandarin, a language similar to Hokkien, in model training. Finally, we release an S2ST benchmark set to facilitate future research in this field. 1 2

heading: 1 Introduction
text: Speech-to-speech translation (S2ST) aims at translating speech from one language into speech in another language. S2ST technology can not only enable communication between people speaking different languages but also help knowledge sharing across the world.
While more than 40% of the languages in the world do not have text written forms3, S2ST for unwritten languages still remains a research area with little exploration mainly due to the lack of training data. The majority of the previous work on this topic conducts experiments on datasets built from applying TTS on S2T corpora to generate synthetic target speech for model training (Tjandra
1The demo can be found at https://huggingface. co/spaces/facebook/Hokkien_Translation.
2We open source our code and model at https://github.com/facebookresearch/ fairseq/tree/ust/examples/hokkien.
3https://www.ethnologue.com
et al., 2019; Zhang et al., 2021). Lee et al. (2022b) presents the first textless S2ST system trained on real S2ST data, while it only investigates translation between high-resource and similar language pairs (English↔Spanish, English↔French).
In this work, we take Taiwanese Hokkien as an example of an unwritten language and study S2ST between English (En) and Taiwanese Hokkien. Taiwanese Hokkien (hereafter Hokkien) is one of the official languages in Taiwan spoken by over 70% of the population (approximately 15.8 million people). Hokkien lacks a unified writing system that is widely adopted by its native speakers, though a few possible writing systems exist, e.g. Chinese characters (Hanji), or romanization systems such as Pehōe-jı̄ (POJ) and Tâi-lô, etc. In addition, Hokkien is a tonal language that has complex tone sandhi rules (Cheng, 1968). Wang et al. (2004) investigates Mandarin-Taiwanese Hokkien S2ST with a cascaded template matching approach. In our work, we focus on En↔Hokkien, a distant language pair, and build one-stage S2ST systems.
We take advantage of the discrete unit-based S2ST approach (Lee et al., 2022a) to translate source speech into target discrete units, where we convert the target speech into a sequence of integers by a self-supervised speech encoder. First, to support En→Hokkien translation, we extend HuBERTbased discrete unit extraction (Hsu et al., 2021) and examine the feasibility of unit-to-waveform generation (Polyak et al., 2021) for tonal languages. Second, we leverage the unit-based speech normalization technique proposed in Lee et al. (2022b) to remove the non-linguistic variations in speech from multiple speakers. The original study takes advantage of synthetic speech generated from TTS as the reference target for normalization, while we build the normalizer with real Hokkien speech data. Last but not least, we study two S2ST model training strategies, speech-to-unit translation (S2UT) with a single decoder (Lee et al., 2022a) or a two-pass
4969
decoding process (Inaguma et al., 2022) that leverages Mandarin (Zh) as a written language similar to Hokkien to provide extra text supervision.
As no En↔Hokkien S2ST dataset is available, we also leverage Mandarin to assist the S2ST data creation process and create a 60-hr human annotated training set and an open benchmark set. Nevertheless, this is still a low-resource problem. To tackle the data scarcity issue, we further apply En↔Zh MT to create weakly supervised data (Popuri et al., 2022; Dong et al., 2022) and learn a joint embedding space for English and Hokkien through Mandarin to support data mining from unlabeled English and Hokkien data (Duquenne et al., 2021).
The contributions of this work are as follows:
• We present empirical studies that consolidate various state-of-the-art techniques for S2ST that were previously studied in a controlled setup with synthetic speech and verify their effectiveness in En↔Hokkien translation, where Hokkien is a language without a widely adopted standard text writing system.
• A benchmark set on En↔Hokkien S2ST 4 and the evaluation model for Hokkien speech 5 will be released to encourage future research in this direction.
• To the best of our knowledge, we are the first to build one-stage S2ST systems for an unwritten language in a real-world scenario.

heading: 2 Related Work
text: Conventionally, S2ST can be achieved via the concatenation of three systems: automatic speech recognition (ASR), machine translation (MT) and text-to-speech synthesis (TTS) (Lavie et al., 1997; Nakamura et al., 2006). In recent years, the advancement from end-to-end speech-to-text translation (S2T) (Bérard et al., 2016) or text-to-speech translation (T2ST) (Zhang et al., 2021; Lee et al., 2022a) have simplified the S2ST pipeline into two stages, which reduces error propagation issues and improves efficiency (Lee et al., 2022a). Most recently, researchers have built one-stage S2ST systems that can be categorized in several aspects. First, systems that model directly from source to
4https://sites.google.com/nycu.edu.tw/ speechlabx/tat_s2st_benchmark
5https://github.com/facebookresearch/ fairseq/tree/ust/examples/hokkien
target speech, with Jia et al. (2019, 2022a,b) predicting spectrogram outputs directly, and Lee et al. (2022a,b); Huang et al. (2022); Popuri et al. (2022); Inaguma et al. (2022) leverage self-supervised speech model such as HuBERT (Hsu et al., 2021) to encode the target speech into a sequence of discrete units and apply knowledge from speech-totext modeling to S2ST. Second, the textless setup, where Jia et al. (2019, 2022b) require extra supervision from target text or phonemes during model training, while Tjandra et al. (2019); Lee et al. (2022b); Popuri et al. (2022) show the possibility of model training with speech data only without going through text. Finally, multiple decoders with multi-pass decoding, where Kano et al. (2021); Inaguma et al. (2022) concatenate multiple decoders learned with additional text targets or speech units with different granularity and perform multi-pass decoding during inference.
While the modeling choices vary, S2ST model training often faces the challenge of data scarcity. Jia et al. (2022c) applies high-quality English TTS and creates an X→En S2ST dataset with synthetic target speech for 21 languages. To create S2ST datasets with real speech, Wang et al. (2021a) aligns ASR transcripts for more than 100 language pairs, and Duquenne et al. (2022a) applies distancebased bitext mining to audio, producing a mined S2ST dataset between 17 European languages. Weakly supervised data created from TTS (Jia et al., 2022a) or a cascaded pipeline with ASR and MT models (Dong et al., 2022; Popuri et al., 2022) is often combined with the S2ST data. In addition, self-supervised pre-training with large-scale unlabeled data also effectively improves S2ST model performance (Jia et al., 2022a; Popuri et al., 2022).

heading: 3 Methodology
text: In this section, we first present two types of backbone architectures for S2ST modeling. Then, we describe our efforts on creating parallel S2ST training data from human annotations as well as leveraging speech data mining (Duquenne et al., 2021) and creating weakly supervised data through pseudolabeling (Popuri et al., 2022; Jia et al., 2022a).

heading: 3.1 Model architectures
text: As illustrated in Fig. 1, we study one model architecture that applies a single-pass decoding process and directly translates source speech to the target, and the second one relies on target text (Mandarin
text in the case of Hokkien speech) to provide extra supervision and performs two-pass decoding. Both architectures predict discrete units as the target, and the speech encoder and text or unit decoders are pre-trained with unlabeled speech or text data.

heading: 3.1.1 Speech-to-unit translation (S2UT)
text: We follow the S2UT approach proposed in Lee et al. (2022a) and adopt HuBERT (Hsu et al., 2021) to convert target speech into discrete units via k-means on intermediate representation. While Hokkien→En systems can be trained on target English speech generated from single-speaker TTS to remove variations in accents from multiple speakers or noises from different recording conditions, when training En→Hokkien systems, we first apply a unit-based speech normalizer (Lee et al., 2022b) on the real Hokkien target speech. The speech normalizer is built by applying Connectionist Temporal Classification (CTC) (Graves et al., 2006) finetuning with the Hokkien HuBERT model using multi-speaker speech as input and the corresponding discrete units extracted from real Hokkien speech from a reference speaker as target.
The resulting S2ST system consists of a sequence-to-sequence S2UT model and a unitbased HiFi-GAN vocoder (Polyak et al., 2021) for unit-to-waveform conversion. For both model architectures, we pre-train the speech encoder with Conformer-based (Gulati et al., 2020) wav2vec 2.0 (Baevski et al., 2020; Popuri et al., 2022) using a large amount of unlabeled speech. To speed up model training, we replace the multilayer convolutional feature encoder with the precomputed 80-dimensional log-mel filterbank features. Preliminary experiments show no performance degradation with filterbank input.

heading: 3.1.2 Single-pass decoding S2UT
text: Lee et al. (2022a) proposes to use a single unit
decoder, which can be trained with standard crossentropy loss. Following Popuri et al. (2022), we apply mBART training (Liu et al., 2020), a denoising autoencoder trained with monolingual text in multiple langauges, using discrete units extracted from unlabeled speech with consecutive duplicate units removed, and use the pre-trained decoder to initialize the unit decoder. During decoding, we perform beam search with the unit decoder.

heading: 3.1.3 Two-pass decoding S2UT: UnitY
text: UnitY model (Inaguma et al., 2022) also performs speech-to-unit translation, while it includes a target text decoder and a target text to target unit encoderdecoder and incorporates an auxiliary target text prediction task during training. All the modules are trained jointly. In En→Hokkien direction, we use Mandarin as the target text due to its proximity to Hokkien and abundance in text data. We follow Inaguma et al. (2022) to apply R-Drop (Wu et al., 2021) regularization during training as well as initializing the target text decoder with a text mBART model (Liu et al., 2020) pre-trained on the combination of En and Zh monolingual text data.

heading: 3.2 Training data
text: In the following sections, we describe three different efforts on creating parallel En↔Hokkien data for model training.

heading: 3.2.1 Supervised human annotated data
text: Since En↔Hokkien bilingual speakers are scarce, we use Mandarin as a pivot language during the data creation process whenever possible. We sample from the following data sources and adopt different strategies to create human annotated parallel data: (1) Hokkien dramas, which include Hokkien speech and aligned Mandarin subtitles
6, (2) Taiwanese Across Taiwan (TAT) (Liao et al., 2020b), a Hokkien read speech dataset containing transcripts in Tâi-lô and Hanji, and (3) MuST-C v1.2 En-Zh S2T data (Cattoni et al., 2021).
We ask Zh-En bilinguals to translate the subtitles of the Hokkien dramas into English to create Hokkien→En S2T data. For the TAT dataset, we leverage a small group of En↔Hokkien bilinguals to translate the Hokkien speech and transcripts directly into English text. For MuST-C, we ask ZhHokkien bilinguals to translate the Mandarin text into a mix of Tâi-lô and Hanji script and then record the Hokkien speech7. The non-standardized script helps to improve the fluency and accuracy of the recorded Hokkien speech, while no Hokkien transcripts are used during S2ST training.
In the end, we build S2ST training sets, where the En→Hokkien set is from MuST-C. For Hokkien→En training, we apply an English textto-unit (T2U) model (Lee et al., 2022b), which is a sequence-to-sequence Transformer model trained on English characters as input and units extracted from the corresponding speech as target, on the English text collected for Hokkien dramas and TAT, as well as the English transcriptions provided in MuST-C, to convert the text into units.

heading: 3.2.2 Mined data
text: To build a shared embedding space for Hokkien and English speech and text data for performing speechto-text or speech-to-speech mining at scale, we again take advantage of Mandarin text as the bridge between the two languages. First, to encode En and Zh text in the same embedding space, we apply the method proposed in Duquenne et al. (2022b) to finetune XLM-R LARGE (Conneau and Lample, 2019) to fit LASER (Artetxe and Schwenk, 2019) English text space using Zh-En parallel MT data. Then, we minimize the mean squared error (MSE) loss between the max-pooled output of the learned text encoder and that of a speech encoder using aligned Hokkien speech and Mandarin or English text8. The text encoder is fixed during speech encoder training, where the latter is initialized with Conformer-based wav2vec 2.0 pre-trained with Hokkien speech, and this process further encodes the Hokkien speech, Mandarin and
6Hokkien drama data is obtained from the collaboration with National Taiwan University.
7The annotators pointed out that it is easier to leverage both systems, which is another evidence of Hokkien lacking a commonly adopted text writing system.
8A subset of the Hokkien dramas data has English subtitles.
English text in the same embedding space. Similarly, we also leverage the fixed text encoder to train an En speech encoder using speech and text pairs from En ASR data. In the end, we create a shared embedding space for En speech and text, Mandarin text, and Hokkien speech, which supports En text and Hokkien speech or En speech and Hokkien speech mining based on cosine similarity.

heading: 3.2.3 Weakly supervised data
text: We take advantage of cascaded systems to create weakly supervised data from ASR and S2T data (Popuri et al., 2022; Dong et al., 2022). For En→Hokkien, we apply En→Zh MT on the En ASR transcriptions, followed by a Zh→Hokkien text-to-unit-translation (T2UT) model, which is a Transformer-based sequence-tosequence model trained with Mandarin characters as input and the corresponding Hokkien normalized units as targets. For Hokkien→En, we apply the Zh→En MT model on the Hokkien drama Mandarin subtitle, followed by En T2U to create pseudo-labeled data.

heading: 4 Experimental Setup
text: In this section, we describe the data, model training details, as well as baseline systems and the evaluation protocol. All experiments are conducted using fairseq (Ott et al., 2019).

heading: 4.1 Data

heading: 4.1.1 Supervised human annotated data
text: We carry out the annotation process in Sec. 3.2.1, and Table 4 summarizes the statistics of the training data. In the end, we create a 61.4-hr human annotated training set for Hokkien→En, and 35-hr for En→Hokkien. We do not combine the synthetic English speech created for Hokkien→En with the real En→Hokkien S2ST dataset during training.

heading: 4.1.2 TAT-S2ST: En↔Hokkien S2ST evaluation dataset
text: As a part of the effort on creating human annotated data, we also create an En↔Hokkien S2ST benchmark set to facilitate future research in the field. The English text translation we collect for the TAT dev and test sets are proofread first, and we recruit native speakers to record the English text translations, producing En↔Hokkien parallel speech data. Table 5 shows the statistics of this benchmark set. While Hokkien does not have a standardized and widely adopted writing system,
TAT provides Tâi-lô transcripts, which is a standardized romanization system for Hokkien, which can be leveraged as reference text in evaluation (Sec. 4.4).

heading: 4.1.3 Mined data
text: We train the En and Zh joint text encoder on CCMatrix (Schwenk et al., 2019), the Hokkien speech encoder on Hokkien dramas, and the English speech encoder on English ASR data from CommonVoice (Ardila et al., 2020), CoVoST-2 (Wang et al., 2021b), Europarl-ST (Iranzo-Sánchez et al., 2020), MuST-C (Di Gangi et al., 2019), Voxpopuli (Wang et al., 2021a) and Librispeech (Panayotov et al., 2015). The learning rate is set to 10−4, with an inverse square root schedule. The maximum number of tokens is set to 640k (equivalent to 40 seconds with 16kHz sampling rate), with a maximum number of sentences set to 32. We train the models with 48 GPUs for 60k steps.
With the trained text and speech encoders, we perform data mining between Hokkien speech from Hokkien dramas and English Common Crawl text, and between the former and Librivox English audio9. We post-process the mined data in order to have a maximum of 20% overlap between any two audio segments. In the end, we obtain 8.1k-hr Hokkien→En S2T mined data and 197-hr En↔Hokkien S2ST mined data. The difference in the volume is mainly due to the domain mismatch in audiobooks from Librivox and Hokkien dramas.

heading: 4.1.4 Weakly supervised data
text: For En→Hokkien, we apply En→Zh MT on the combination of the English transcripts from Librispeech (Panayotov et al., 2015) and TEDLIUM3 (Hernandez et al., 2018), totaling 1.5khr of English speech. The En→Zh MT model is a 12-layer Transformer model trained on CCMatrix (Schwenk et al., 2019) using disjoint BPEs for En and Zh encoded by the sentencepiece toolkit (Kudo and Richardson, 2018), each of size 32768. We use 16 GPUs, a batch size of 14,336 tokens and a learning rate of 10−3 during training.
The Zh→Hokkien T2UT model following the En→Zh translation step is trained on Hokkien dramas and the aligned Mandarin subtitles. We filter out speech containing Mandarin code-switching by applying Mandarin ASR and computing the Levenshtein distance between the ASR output and the subtitles, as well as short sentences with less than
9https://librivox.org/api/
three characters, resulting in 1k-hr Hokkien speech for training.
For Hokkien→En, we apply Zh→En MT on the Mandarin subtitles from 8k-hr Hokkien drama data, followed by an En T2U trained on LJSpeech (Ito and Johnson, 2017). The Zh→En MT is trained with the same setup as En→Zh MT.

heading: 4.2 Model training

heading: 4.2.1 Hokkien HuBERT units
text: To encode En target speech, we use the multilingual HuBERT model, the k-means quantizer and the unit vocoder released from Lee et al. (2022b). Below we focus on how we build Hokkien units and the corresponding unit-based speech normalizer and unit vocoder.
We train a Hokkien HuBERT model using the combination of 10k-hr Mandarin speech from WenetSpeech (Zhang et al., 2022) and 2k-hr Hokkien speech from the combination of Hokkien dramas, TAT and 600-hr of Hokkien speech with various accents in addition to Taiwanese Hokkien, licensed from SpeechOcean10. When modeling Hokkien speech as discrete units, we empirically find that combining Mandarin with Hokkien speech during HuBERT training allows the units to better capture the tones and produce higher-quality speech output in the unit-to-waveform conversion stage.
The HuBERT model is of the BASE architecture and pre-trained for three iterations following Hsu et al. (2021); Lakhotia et al. (2021). In the beginning of each iteration, we randomly sample 300-hr Mandarin and Hokkien speech, respectively, for k-means clustering, and apply temperature sampling to balance the amount of speech from the two languages during training. We use T = 20, and the probability of sampling from a language
l is p̃l = p
1 T l ∑ i p 1 T i , where pi = ni∑ j nj , and ni is
the number of samples from a language. No extra language information is required during pretraining. In each iteration, model weights are randomly initialized and optimized for 400k steps. We use K = 2500 with features from the 12-th layer of the model from the third iteration for extracting Hokkien units.
The Hokkien speech normalizer is trained on 2- hr speech from TAT. We select speaker THF022 as the reference speaker, i.e. the normalization target,
10https://en.speechocean.com/
and create speech pairs by sampling from other speakers reading the same content in TAT. We use mask probability of 0.5, mask channel probability of 0.25 and learning rate of 3 × 10−5 and train for 25k updates. Finally, the Hokkien unit-based HiFi-GAN vocoder is trained on the TTS subset of the TAT dataset, which contains a total of 36 hours of clean speech from two male and two female speakers, following the training procedure in Lee et al. (2022a).

heading: 4.2.2 Wav2vec 2.0 encoder
text: We pre-train the Conformer En wav2vec 2.0 LARGE encoder (Baevski et al., 2020) with the Libri-light corpus (Kahn et al., 2020), which contains around 54k hours of read speech audio. The encoder is trained with a batch size of 2.1-hr for 1M updates, with 32k warmup steps and a peak learning rate of 5× 10−4. For masking, we sample a probability of 0.065 of all time-steps to be starting indices and mask the subsequent 10 time steps. For the Hokkien wav2vec 2.0 encoder, we pre-train it with 30k-hr Hokkien drama data using the same hyper-parameters as the En wav2vec 2.0 encoder.

heading: 4.2.3 Single-pass decoding S2UT
text: The Hokkien unit mBART is trained with 30k-hr Hokkien dramas and 10k-hr Mandarin data from WenetSpeech. The model is trained on 64 GPUs with a batch size of 3072 units, learning rate of 3× 10−4 with Adam and 10k warmup steps. The model is trained with 500k updates with dropout 0.1. We use the En unit mBART released by Popuri et al. (2022) for training Hokkien→En models.
With the pre-trained wav2vec 2.0 encoder and the unit mBART decoder, we follow the best finetuning strategy in Popuri et al. (2022), where the whole encoder and the LayerNorm and both encoder and self attention in the decoder are finetuned with the parallel S2ST data. The models are trained on 32 GPUs with a batch size of 160k tokens. We used 0.1 dropout for all models and 0.2 LayerDrop (Fan et al., 2019). The models are trained using Adam optimizer with 3× 10−4 learning rate, 10k warmup steps an 50k maximum updates.

heading: 4.2.4 Two-pass decoding S2UT: UnitY
text: The text mBART model is pre-trained on the combination of Mandarin and English text data from CC100 (Conneau et al., 2020), Newscrawl (Akhbardeh et al., 2021), Leipzig Corpora (Goldhahn et al., 2012), NewsCommentary (Tiedemann, 2012).
There are 2B English sentences and 230M Mandarin sentences. We learn BPE of size 65536 jointly on both languages and apply temperature sampling with 1T = 0.7 during training.
We combine the pre-trained wav2vec 2.0 encoder, the text mBART decoder, and two randomly initialized Transformer layers for the text encoder and the unit decoder, respectively, to build the UnitY model. We train our two-pass models on 16 GPUs with a batch size of 120k tokens, dropout 0.1 for all models except for the human annotated data only setup where we use dropout 0.3. We use LayerDrop (Fan et al., 2019) 0.1 and label smoothing 0.1, and train the model with a learning rate of 5 × 10−4, 2k warmup steps, and a maximum update of 50k steps. The weight on the auxiliary loss from the text decoder is set to 8.0.

heading: 4.3 Baselines
text: We build two-stage and three-stage cascaded baseline systems for both En↔Hokkien directions. The two-stage cascaded system consists of a source speech (En or Hokkien) to target text (Mandarin or En) end-to-end S2T model and a target text to target speech unit T2U model (T2UT in the case of Zh→Hokkien). The three-stage cascaded system further breaks down the En→Zh S2T model into En ASR followed by En→Zh MT, and the Hokkien→En S2T model is split into a Hokkien→Zh S2T step and a Zh→En MT step.
All the speech encoders for the En ASR and S2T models are initialized with wav2vec 2.0 (Sec. 4.2.2). The text decoders of S2T models are initialized with the text mBART (Sec. 4.2.4). We use the En↔Zh MT models, the En T2U model and the Zh→Hokkien T2UT model described in Sec. 4.1.4 for building the cascaded systems.

heading: 4.4 Evaluation
text: To evaluate the translation quality, we compute ASR-BLEU on the TAT-S2ST evaluation set (Sec. 4.1.2) by applying ASR on the generated speech and computing 4-gram BLEU against the reference text using SACREBLEU (Post, 2018). We use an open-sourced En ASR model11 when evaluating Hokkien→En systems. For En→Hokkien systems, we build an ASR model to transcribe Hokkien speech into Tâilô. The Hokkien ASR is initialized with a w2v-
11https://huggingface.co/facebook/ wav2vec2-large-960h-lv60-self
BERT (Chung et al., 2021) LARGE model pretrained on 10k-hr Mandarin speech from WenetSpeech and 30k-hr Hokkien speech from Hokkien drama, followed by finetuning with CTC loss on 480-hr Hokkien speech and Tâi-lô scripts from TAT (Liao et al., 2020b). Each Tâi-lô syllable is split into initial and final with tone as the target. The resulting Hokkien ASR model achieves 6.8% syllable error rate (SER) on the TAT-Vol1-testlavalier set. To evaluate En→Hokkien translation quality, we compute syllable-level ASR-BLEU.
To evaluate the naturalness of the speech output, we collect mean opinion scores (MOS) ranges from 1 (the worst) to 5 (the best) from human listening tests. Each item is labeled by three annotators.

heading: 5 Results

heading: 5.1 Single-pass vs. two-pass decoding
text: We first study the model architecture choice in both En↔Hokkien directions. Table 1 summarizes the results. We include ASR-BLEU from the target reference speech as an indication of the effect from the unit vocoder and the ASR errors (row 7). We start from training on human annotated data, and it results in very low BLEU score in both directions (row 3, 5), indicating that pre-training, including wav2vec 2.0 and unit or text mBART, is not enough for building a S2ST system under low-resource for distant language pairs. With extra supervision from text, the UnitY model works slightly better than single-pass S2UT by 3.7 BLEU in Hokkien→En (row 3 vs. 5).
We then combine the human annotated data with weakly supervised data. Both systems achieve significant gain (6.2-7.5 BLEU) in both directions, indicating the effectiveness of combining self-supervised pre-training and data augmentation with weakly supervised data in low-resource S2ST for a distant language pair.
In addition, we find that UnitY outperforms single-pass S2UT in Hokkien→En direction (row 4 vs. 6) by 2.9 BLEU. However, in En→Hokkien, UnitY is merely 0.4 BLEU higher than single-pass S2UT. The larger impact from the additional text supervision in Hokkien→En may be due to the fact that the target text and speech are of the same language, or the larger amount of training data available. As the focus of this work is to present a data creation and model training strategy, we leave the investigation to future work.
For the cascaded baselines, the two-stage sys-
tem is worse than the three-stage system in both En↔Hokkien directions (row 1 vs. 2). Our best one-stage system performs similarly to the best cascaded systems (row 2 vs. 6).
For MOS, the cascaded systems and single-stage S2UT systems have similar naturalness in both En→Hokkien and Hokkien→En directions.

heading: 5.2 Mined data
text: In this section, we study how to leverage mined Hokkien→En S2T and En↔Hokkien S2ST data.

heading: 5.2.1 Leveraging mined En↔Hokkien S2ST in En→Hokkien direction
text: In Table 2, we show the results of leveraging the mined En↔Hokkien S2ST data in En→Hokkien direction. In order to train the UnitY model, we apply Hokkien→Zh S2T to generate pseudo-labeled Mandarin text for the mined Hokkien speech as the auxiliary task target.
We first train both one-stage models with mined data and the human annotated data. While the single-pass decoding S2UT model still yields very low BLEU score (row 8), the UnitY model achieves 4.8 BLEU improvement with the extra 197-hr of mined S2ST data (row 5 vs. 10), showing that noisy Mandarin text generated from pseudo-labeling still provides useful signals in model training. We then further combine with weakly supervised data but do not see significant gain with the additional mined data (row 4 vs. 9, 6 vs. 11). Note that the size of mined data is only 13% of the total amount of weakly supervised data we have. As discussed in Sec. 4.1.3, the limited amount of mined data available is mainly due to the domain mismatch issue. In the future, we plan to explore mined data from more similar domains and aim to increase the amount of data for better S2ST performance.
We convert the mined Hokkien→En S2T data to S2ST data with the En T2U model and train UnitY models with the combination of human annotated data and optionally the 8k-hr weakly supervised data to examine the effect of mined data on model performance. Table 3 shows the ASR-BLEU scores on the TAT-S2ST test set with respect to different thresholds on the similarity scores of the mined pairs.
We see that adding 4.7k-hr mined S2T data (t = 1.065) in Hokkien→En is the most helpful and improves the model quality by 3.6 BLEU when only human annotated data is available. With 8.1khr mined data (t = 1.06), the BLEU gain drops
to 0.9 BLEU. In addition, it is 5.3 BLEU lower than the UnitY model trained with human annotated data and 8k-hr of weakly supervised data (Table 1 row 6). As the Hokkien speech for both weakly supervised data and mined data comes from the same Hokkien dramas dataset, the gap implies that pseudo-labeling is a generally effective data augmentation technique for low-resource scenarios, while the quality of the mined data is constrained by the content of the data available for mining. However, combining all three types of data together is still beneficial. We obtain 0.5 BLEU gain by adding 4.7k-hr mined data to the combination of human annotated and weakly supervised data.

heading: 6 Conclusions
text: We present the first En↔Hokkien S2ST systems, where Hokkien is an oral language that does not have standard and widely adopted text writing systems, i.e. an unwritten language. To tackle the challenges of speech translation for unwritten languages and the lack of parallel training data, we present an end-to-end study. First, we explore three options of training data creation including human annotation, weakly supervised data from pseudolabeling and data mining. Second, we investigate two modeling choices including direct speech-tounit translation with a single speech unit decoder and two-pass decoding that leverages extra supervision from target text. Experimental results show that leveraging a similar high-resource written language (Mandarin in the case of Hokkien) is effective in both the data creation process and model training. Finally, we release the benchmark dataset and ASR evaluation model to facilitate research in this field. In the future, we aim to expand study and establish an S2ST model building strategy that works for a diverse set of unwritten languages.

heading: 7 Limitation
text: In our research, we have focused on one language pair, English↔Hokkien, and experimenting in both directions. In the future, we plan to apply the same methodology to additional unwritten languages to evaluate its broad applicability.
Our approach leverages parallel speech-to-text data between the unwritten language and a linguistically similar written language. There remains a question of whether there are unwritten languages without similar written languages.

heading: 8 Acknowledgements
text: We would like to thank Koklioong Loa for consulting on Taiwanese Hokkien; Eric Lam, KaiWei Chang, Iū-thîng Kang and Hung-yi Lee from National Taiwan University for providing Hokkien drama data; Janice Lam for writing and fact-checking the information about Hokkien language; Brian Bui and Carleigh Wood for coordinating the data annotation effort; Ilia Kulikov for setting up the evaluation script; Pengwei Li for the HuggingFace demo integration.

heading: A Dataset stats
text: ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Left blank.
7 A2. Did you discuss any potential risks of your work? The speech to speech translation application has been broadly study in the field. In our work, we expand the application to a new unwritten language, Taiwanese-Hokkien to English, and except that we did not enable new application in our work. Therefore, we did not discuss potential risk of our work.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Left blank.

heading: 3 A4. Have you used AI writing assistants when working on this paper?
text: I use translation tool (Google Translate) for checking if my English sentences are fluent. I used it for the limitation section.
B 3 Did you use or create scientific artifacts? use scientific artifacts - we use FAIRSEQ to development our model (section 4), and open source datasets for training and evaluation (full datasets listed in the appendix). Create scientific artifacts - we release our models and benchmark datasets. The link is in the introduction.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: FAIRSEQ - section 4 open source datasets - appendix
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We list the terms for use in our open source page.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? We did not discuss this in the paper, but we did check and specify the the intended use in our open source page.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We did not discuss the steps in the paper. But we did remove the content about uniquely identifies individual people and offensive content during the data collection process.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? We provide the demographic groups represented stats in the appendix.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. We provide the relevant stats about the splits, including number of examples and durations in the appendix.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? section 5 Results.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Yes, we report the number of parameters in each models and the training parameters including number of updates and number of GPUs in section 4.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Yes, we reported the best-found hyper-parameter values in section 4.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Yes, we provide the error bar for the MOS score. For translation quality (BLEU), we have done multiple rounds of experiments but on the resulting table that we only reported one single run. The description is clear that it is single run result.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? We write our own normalization function and we open source it. The link is in the abstraction.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants?

heading: It is in section 5 for the MOS score. In the 3.2.1 about the S2ST dataset.
text: 7 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? We didn’t report the full text of instructions to the participants but just describe in high level that what the MOS score meant to cover to measure.
7 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No, we did not disclose. Our vendor didn’t expose the information to us.
7 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No we didn’t discuss, but we did have a consent from the user that we collect the data.

heading: 3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?
text: We have internal review to make sure we collect data with the right consent.

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Yes, we report the demographic in the data set table in appendix.

heading: Future Work
text: In the future, we plan to apply the same methodology to additional unwritten languages to evaluate its broad applicability. Our approach leverages parallel speech-to-text data between the unwritten language and a linguistically similar written language. There remains a question of whether there are unwritten languages without similar written languages