Back Translation for Speech-to-text Translation Without Transcripts
Year: 2023

Abstract:
The success of end-to-end speech-to-text translation (ST) is often achieved by utilizing source transcripts, e.g., by pre-training with automatic speech recognition (ASR) and machine translation (MT) tasks, or by introducing additional ASR and MT data. Unfortunately, transcripts are only sometimes available since numerous unwritten languages exist worldwide. In this paper, we aim to utilize large amounts of targetside monolingual data to enhance ST without transcripts. Motivated by the remarkable success of back translation in MT, we develop a back translation algorithm for ST (BT4ST) to synthesize pseudo ST data from monolingual target data. To ease the challenges posed by short-to-long generation and one-to-many mapping, we introduce self-supervised discrete units and achieve back translation by cascading a target-to-unit model and a unit-to-speech model. With our synthetic ST data, we achieve an average boost of 2.3 BLEU on MuST-C En→De, En→Fr, and En→Es datasets. More experiments show that our method is especially effective in low-resource scenarios.12

heading: 1 Introduction
text: End-to-end speech-to-text translation (ST) means directly translating speech in the source language to target text without generating source transcripts (Bérard et al., 2016; Duong et al., 2016). Different from traditional cascading methods which first transcribe the speech with automatic speech recognition (ASR) and then translate the transcripts into the target text with machine translation (MT), end-to-end ST has the potential to reduce latency and avoid error propagation. Hence, it has drawn much attention and achieved great suc-
∗Corresponding author: Yang Feng. 1Code is publicly available at https://github.com/ ictnlp/BT4ST. 2Examples of synthetic ST data are available at https: //bt4st.github.io/ and Appendix B.
cess in recent years (Anastasopoulos et al., 2021, 2022).
However, it is challenging to train an end-to-end ST model with only speech-translation pairs. Traditional cascaded models learn cross-modal mapping with ASR and cross-lingual mapping with MT. In contrast, end-to-end ST requires simultaneous cross-modal and cross-lingual mapping, which is more complicated and usually relies on more training data. However, the amount of ST data is usually limited due to the high cost of data collection, so the ST model trained with only speech-translation pairs is usually unsatisfactory.
To tackle these problems, researchers often utilize source transcripts to assist ST training by introducing auxiliary ASR and MT tasks. With abundant ASR and MT data, the ASR task can help the model learn better cross-modal mapping, while the MT task can help learn better cross-lingual mapping, which can significantly improve ST as shown in recent ST studies (Wang et al., 2020a; Xu et al., 2021; Ye et al., 2021; Fang et al., 2022). Unfortunately, source transcripts are not always available. It is estimated that there are around 3000 unwritten languages in the world which have no orthography for transcription3. For those languages, we can no longer leverage source transcripts to help with training, so many of the latest techniques fail to benefit them.
How to train a stronger ST model without transcripts? In this paper, we address this question from the perspective of data augmentation. Although ASR and MT data are unavailable, a large amount of target-side monolingual data is still easily accessible. Motivated by the success of back translation in MT (Sennrich et al., 2016; Edunov et al., 2018), we aim to develop a back translation algorithm for ST (BT4ST) to synthesize pseudo ST data from monolingual target data. However, compared to text-to-text back translation, gener-
3https://www.ethnologue.com/
4567
ating source speech from the target text without source transcripts is much more challenging4. First, the length of text is usually only tens or hundreds, but the length of speech is about tens of thousands5. Therefore, the model is required to generate an extremely long sequence from a short sequence without the assumption of monotonic alignment, which is a more difficult sequential decision problem. Second, the conversion from text to speech is a one-to-many mapping problem due to the variations in speech (Chen et al., 2021; Ren et al., 2021). For example, the pronunciation of the same content may differ among speakers.
To address these challenges, we introduce selfsupervised discrete units of the source speech as intermediate representations, and achieve back translation by cascading a target-to-unit model and a unit-to-speech model. Since the length of unit sequences is similar to the length of target characters6, and there is a monotonic assumption between the unit sequence and the source speech, the shortto-long generation problem in target-to-unit and unit-to-speech models can be greatly alleviated. Besides, discrete units can disentangle content information from other variation information (e.g., pitch and speaker) (Polyak et al., 2021), which eases the one-to-many mapping problem in the targetto-unit model7. We also introduce a speaker encoder to provide speaker information as input to the unit-to-speech model, which allows us to generate diverse source speeches by coupling different speaker representations. Following this pipeline, we can synthesize large amounts of pseudo ST data from monolingual target data without requiring transcripts. Finally, we train our ST model with both synthetic and real data.
We conduct experiments on MuST-C En→De, En→Fr, and En→Es datasets. By leveraging about 5M additional monolingual target data for each language pair, we achieve an average improvement of 2.3 BLEU compared with the strong baseline. We also observe that our approach is more effective in low-resource scenarios, yielding a boost of
4When transcripts are available, we can decompose it into two sub-tasks: MT for generating transcripts and text-tospeech (TTS) for speech synthesis, which becomes much more manageable.
5For 16kHz audio waveform, 1 second of speech corresponds to a sequence of 16,000 samples.
61 second of speech corresponds to 50 discrete units. 7It should be noted that the one-to-many mapping problem still exist due to the many-to-many mappings between transcripts and translations, but it is not the focus of this work.
5.6 BLEU when only 100 hours of ST data are available. In addition, we generate multiple different pseudo datasets with a simple Diverse BT4ST method, train several models separately with each dataset, and ensemble them together. By ensembling five models, we achieve an average boost of 4.0 BLEU in three translation directions.

heading: 2 Background
text: Our work focuses on developing a back translation algorithm for ST. We first introduce the task definition and model architecture of ST in Section 2.1, and then introduce the concept of back translation in Section 2.2.

heading: 2.1 Speech-to-text Translation
text: Task Definition The goal of speech-to-text translation (ST) is to translate speech in one language into text in another language. We denote the source speech as x = (x1, ..., xI), where I is the length of the audio waveform. The model generates the target sentence y = (y1, ..., yJ), where J is the length of the target text. In this paper, we assume that transcripts of the source speech are not available, which is a more general scenario considering about 3000 unwritten languages in the world.
Model Architecture Our ST model consists of three stacked modules: acoustic encoder, length adaptor, and translation model. The acoustic encoder is a HuBERT (Hsu et al., 2021) model pretrained on unlabelled audio data, which can generate meaningful representations for the source speech. The length adaptor (Li et al., 2021) is a series of convolutional layers to shrink the length of speech representations by a factor of 4. The translation model is a Transformer (Vaswani et al., 2017) with N encoder layers and N decoder layers, which takes the shrunken speech representations as input and outputs the target sentence. We train the ST model by minimizing the cross-entropy loss:
LST = − J∑
j=1
log p(yj |x,y<j). (1)

heading: 2.2 Back Translation
text: Back translation (BT) is a simple and effective method to leverage target-side monolingual data in neural machine translation (NMT). Formally, given a parallel corpus D = {(x(n),y(n))}Nn=1, and a monolingual corpus of the target language
T = {y(m)}Mm=1, we first train a target-tosource model on D. Next, we use this model to generate additional pseudo parallel data D̃ = {(x̃(m),y(m))}Mm=1 from the monolingual corpus T . Finally, D̃ can be used as a complement to D to train a stronger source-to-target model.

heading: 3 Method: BT4ST
text: How to acquire a target-to-source model for ST given a ST parallel corpus? Directly generating source speech from the target text is a challenging problem. Inspired by recent success in self-supervised discrete representation learning for speech (Baevski et al., 2020; Hsu et al., 2021; Lakhotia et al., 2021), we first transform the source speech into a sequence of discrete units with a speech pre-trained model (Section 3.1), which is used as an intermediate representation in the back translation process. With discrete units, we train a target-to-unit model (Section 3.2) and a unit-tospeech model (Section 3.3) on the parallel corpus, where the former predicts the sequence of discrete units corresponding to the source speech, and the latter converts discrete units into waveform. The model architecture is illustrated in Figure 1.

heading: 3.1 Unit-based Speech Representation
text: We use the pre-trained HuBERT (Hsu et al., 2021) model to generate discrete units corresponding to the source speech following Lee et al. (2022a,b). HuBERT generates 50Hz continuous representations for the input speech. We apply the K-means clustering algorithm to the continuous representations of the training data, and then transform the continuous representations into the corresponding cluster indices, i.e., discrete units. Finally, the in-
put speech x = (x1, ..., xI) is converted into a sequence of discrete units z = (z1, ..., zT ), zt ∈ {0, 1, ...,K − 1},∀1 ≤ t ≤ T , where K is the number of clusters, and T is the number of frames where T = ⌊ I320⌋. There are two advantages to use discrete units as intermediate representations rather than predicting audio waveform directly. First, the sequence of discrete units is much shorter than the audio waveform, alleviating the difficulty of short-to-long generation. Second, discrete units can disentangle speech content from the pitch and speaker information (Polyak et al., 2021), which eases the one-to-many mapping problem.

heading: 3.2 Target-to-unit Model
text: Our target-to-unit model is a Transformer-based sequence-to-sequence model, which predicts discrete units of the source speech based on the target text. The target text y is tokenized as characters and fed to the encoder. For the discrete units z, we first merge repeating units into a single one to obtain the reduced discrete units z′ = (z′1, ..., z ′ T ′) following Lee et al. (2022a). For example, (1, 1, 2, 2, 2, 3, 4, 4) will collapse to (1, 2, 3, 4). We then train the model with reduced discrete units as the target, which can accelerate training and inference. The training objective is as follows:
LT2U = − T ′∑
t=1
log p(z′t|y, z′<t). (2)

heading: 3.3 Unit-to-speech Model
text: Our unit-to-speech model is a unit-based HiFiGAN vocoder (Kong et al., 2020) as proposed in Polyak et al. (2021). It takes the reduced discrete units as input and generates the waveform. The
model consists of four modules: duration predictor, speaker encoder, generator, and discriminator.
Duration Predictor As the output of the targetto-unit model is reduced discrete units, we add a duration predictor (Ren et al., 2021) to predict the duration of each unit. It consists of two 1Dconvolutional layers with ReLU activation, each followed by layer normalization and dropout. Finally, a linear layer projects the hidden state into a scalar as duration. We denote the predicted duration vector as d = (d1, ..., dT ′), and the ground truth as d∗ = (d∗1, ..., d ∗ T ′). The duration predictor is optimized with Mean Squared Logarithmic Error (MSLE) between ground truth and predictions as:
LDP = 1
T ′
T ′∑
t=1
(log(1 + dt)− log(1 + d∗t ))2. (3)
Given the duration vector, we expand reduced discrete units by repeating each unit. For example, given z′ = (1, 2, 3, 4) and the corresponding duration vector d = (2, 3, 1, 2), the expanded sequence becomes z = (1, 1, 2, 2, 2, 3, 4, 4). We use ground truth duration during training and predicted one during inference. Finally, the units are converted to continuous representations via a look-up table.
Speaker Encoder Discrete units contain little speaker information, but ST corpus usually contains speech from multiple speakers. To ease the one-to-many mapping problem in speech synthesis, we introduce a speaker encoder to extract the speaker information. The speaker encoder is a pretrained speaker verification network (Wan et al., 2018), which extracts a single 256-dimensional speaker embedding from the speech. The speaker embedding is then concatenated to the representation of each unit. During inference, we randomly select a speaker embedding from the training set for each sample. It allows us to synthesize a pseudo ST dataset containing multiple speakers.
Generator and Discriminator The generator and discriminator are the same as the original HiFiGAN (Kong et al., 2020). The generator consists of several stacked blocks, each containing a transposed convolution layer followed by multiple residual blocks. It takes the concatenated features as input and outputs the waveform x. The discriminator contains a Multi-Period Discriminator (MPD) and a Multi-Scale Discriminator (MSD), which are used to identify the periodic or consecutive patterns in the audio. The generator and discriminator
Algorithm 1: Back Translation for ST Input :ST data D = {(x(n),y(n))}Nn=1,
Target data T = {y(m)}Mm=1, Data selection ratio ρ
Output :ST model Mx→y 1 Procedure BT4ST(D, T , ρ) 2 Get reduced discrete units z′ from x to
create D′ = {(x(n), z′(n),y(n))}Nn=1, for every (x,y) ∈ D
3 Train target-to-unit model Mz′←y and unit-to-target model Mz′→y with paired data (z′,y) in D′
4 Train unit-to-speech model Mx←z′ with paired data (x, z′) in D′
5 Use Mz′←y and Mx←z′ to create D̃ = {(x̃(m), z̃′(m),y(m))}Mm=1, for every y ∈ T 6 Use Mz′→y to translate z̃′ into ỹ, and compute BLEU(ỹ,y), for every (x̃, z̃′,y) ∈ D̃ 7 Select top ρ ·M samples from D̃ based on BLEU scores, denoted as D̃S 8 Training(D, D̃S) 9 End
10 Procedure Training(D, D̃S) 11 Pre-train ST model Mx→y on D̃S 12 Fine-tune ST model Mx→y on D 13 End
are trained adversarially. More details about the HiFi-GAN can be found in Appendix A.

heading: 3.4 Data Selection and Model Training
text: By cascading the target-to-unit model and unitto-speech model, we can synthesize ST data D̃ = {(x̃(m), z̃′(m),y(m))}Mm=1 from the targetside monolingual corpus T = {y(m)}Mm=1, where x̃ is the synthetic speech and z̃′ is the corresponding reduced discrete units. However, the synthetic corpus may contain some low-quality data, which may hurt model training. Therefore, we introduce data selection and 2-stage model training to better utilize the synthetic data.
Data Selection To identify the low-quality data, we train a unit-to-target model to translate the generated units z̃′ back into target text ỹ, and compute the BLEU score (Papineni et al., 2002) between ỹ and the ground truth y, i.e., BLEU(ỹ,y). Finally,
we only keep the top ρ ·M samples according to the BLEU score, where ρ is the selection ratio. The remaining low-quality samples are discarded.
2-stage Model Training After we get the selected synthetic ST data, we train the model in a 2- stage manner following Abdulmumin et al. (2021). We first pre-train the model on synthetic data, and then fine-tune the model on real data. This can prevent the noise-infested synthetic data from overwhelming real data during training. Algorithm 1 describes the whole process of our proposed method.

heading: 4 Experiments

heading: 4.1 Datasets
text: MuST-C MuST-C (Di Gangi et al., 2019) is a multilingual speech translation dataset, which contains about 400 hours of English (En) audio clips and corresponding translations in 8 languages: German (De), French (Fr), Spanish (Es), Italian (It), Portuguese (Pt), Dutch (Nl), Romanian (Ro), and Russian (Ru). We conduct experiments on En→De, En→Fr, and En→Es, because these three languages have larger public monolingual corpora.
Monolingual Target Data We use the target text in WMT (Buck and Koehn, 2016) En→De, En→Fr, and En→Es datasets as monolingual target data. Specifically, we use europarl v78, commoncrawl9, news commentary v1210 subsets for all three languages. The source text is never used in the experiments. The detailed statistics of data we used are shown in Table 1.

heading: 4.2 Model Settings
text: Target-to-unit Model The target-to-unit model is a Transformer with 6 encoder layers and 6 decoder layers. Each layer comprises 512 hidden states, 4 attention heads, and 2048 feed-forward hidden states. The dropout is 0.3, and the label smoothing is 0.1. For target text, we first lowercase the text and segment it into characters using SentencePiece11. For discrete units, we use the pre-trained quantized model12, which learns K = 100 clusters
8http://statmt.org/wmt13/ training-parallel-europarl-v7.tgz
9http://statmt.org/wmt13/ training-parallel-commoncrawl.tgz
10http://data.statmt.org/wmt17/ translation-task/training-parallel-nc-v12.tgz
11https://github.com/google/sentencepiece 12https://dl.fbaipublicfiles.com/textless_nlp/
gslm/hubert/km100/km.bin
from the 6th layer representations of pre-trained HuBERT-Base model13, to convert the speech into discrete units. During training, the batch size is 400, and the maximum learning rate is 5e-4. We train the model up to 100k steps with Adam optimizer (Kingma and Ba, 2015). During inference, we use beam search with a beam size of 8 to generate the reduced discrete units.
The unit-to-target model is the same as the target-to-unit model except for the translation direction, which is used for data selection. During inference, we use greedy search to save time.
Unit-to-speech Model For the unit-to-speech model, the configurations of the generator and discriminator are the same as Polyak et al. (2021). The 1D-convolutional layers in the duration predictor are set to kernel size 3, padding 1, and hidden dimension 256. We use the pre-trained dvector model14 as the speaker encoder to extract the 256-dimensional speaker embedding. We train the model up to 100k steps with a batch size of 128.
ST Model The ST model contains three stacked modules. We use the pre-trained HuBERT-Base model as the acoustic encoder, which takes the raw 16-bit 16kHz audio wave as input. The length adaptor comprises two 1D-convolutional layers with kernel size 5, stride size 2, padding 2, and hidden dimension 1024. The translation model follows Transformer-Base architecture, containing 6 encoder layers and 6 decoder layers. Each layer comprises 512 hidden states, 8 attention heads, and 2048 feed-forward hidden states. The dropout is 0.1, and the label smoothing is 0.1. We refer to this architecture as HU-TRANSFORMER. We learn a vocabulary of size 8k from the target texts in the MuST-C dataset to segment the target texts into subwords for training the ST model.
To augment the ST model with back translation, we first select the top ρ = 75% of synthetic data
13https://dl.fbaipublicfiles.com/hubert/hubert_ base_ls960.pt
14https://github.com/yistLin/dvector
based on BLEU scores. The target texts in the synthetic data are segmented into subwords using the same vocabulary. We use Adam optimizer with 4k warm-up steps to pre-train the model on synthetic data up to 300k steps, and fine-tune the model on MuST-C up to 20 epochs. During both pre-training and fine-tuning, each batch contains at most 16M audio frames, and the maximum learning rate is 1e-4. During inference, we average the checkpoints of the last 10 epochs for evaluation. We use beam search with a beam size of 8. SacreBLEU15 (Post, 2018) is used to compute case-sensitive detokenized BLEU scores and the statistical significance of translation results with paired bootstrap resampling16 (Koehn, 2004). The length penalty is set to 1.2, 1.8, and 0.6 for En→De, En→Fr, and En→Es, respectively. We implement our model with fairseq17 (Ott et al., 2019). All models are trained on 4 Nvidia Tesla V100 GPUs.
15https://github.com/mjpost/sacrebleu 16sacreBLEU signature: nrefs:1 | bs:1000 | seed:12345 |
case:mixed | eff:no | tok:13a | smooth:exp | version:2.0.0 17https://github.com/pytorch/fairseq
Baseline Systems We include two baseline systems: REVISIT-ST (Zhang et al., 2022a) and W-TRANSF. (Ye et al., 2021) for comparison. REVISIT-ST is a carefully designed ST baseline including several techniques like parameterized distance penalty (PDP) and CTC-based regularization. W-TRANSF. is a stronger ST baseline with a pretrained acoustic model, which combines Wav2vec 2.0 (Baevski et al., 2020) and Transformer. Both of them are only trained on speech-translation pairs without using any transcripts. Our HUTRANSFORMER is also a strong baseline model trained on speech-translation pairs from MuST-C, and we examine our proposed BT4ST on top of this by adding synthetic ST data.

heading: 5 Results and Analysis

heading: 5.1 Results on MuST-C Dataset
text: Table 2 shows the results on MuST-C En→De, En→Fr, and En→Es tst-COMMON set. First, we observe that our HU-TRANSFORMER is a strong baseline compared with previous baselines. Second, by synthesizing pseudo ST data with our proposed BT4ST and using it to pre-train the model, we achieve an average boost of 2.3 BLEU in three translation directions. It demonstrates that our approach can effectively utilize external monolingual target data to improve the performance of ST.

heading: 5.2 Impact of Data Selection Ratio
text: Although previous work (Edunov et al., 2018) found that noisy synthetic data can benefit training, we argue that the extremely low-quality data also hurt performance. As described in Section 3.4, we select top ρ · M synthetic data based on BLEU scores. We constrain ρ in [0%, 25%, 50%, 75%, 100%] for experiments, and the results on MuST-C En→De tst-COMMON set are shown in Figure 2. We find that filtering out
the last 25% of samples (i.e., ρ = 75%) gives a 0.4 BLEU improvement (26.2→26.6) and performs best. We use ρ = 75% for all experiments.

heading: 5.3 Impact of Unit Generation Methods
text: The generation method of the target-to-unit model also influences the performance of back translation, as it determines the quality and diversity of synthetic data. We conduct experiments with beam search, greedy search, top-10 sampling18, and sampling. As shown in Table 3, beam search performs best over all methods on MuST-C En→De tst-COMMON set. We consider two reasons for this. First, our ST dataset is actually a low-resource setting, containing only less than 300k parallel data19. Second, target-to-unit generation is a more difficult task compared to text translation. Therefore, beam search is more effective as it generates high probability outputs, while sampling from the model distribution may produce harmful low-quality data. We use beam search for all experiments.

heading: 5.4 Single-speaker vs. Multi-speaker Synthetic Data
text: As described in Section 3.3, we provide speaker embedding as input during speech synthesis, which allows us to synthesize pseudo ST datasets con-
18Top-10 sampling refers to sampling over the ten most likely words.
19Edunov et al. (2018) find that beam search is more effective than sampling in low-resource settings, while the opposite is true for high-resource settings.
taining multiple speakers. To examine whether speaker diversity benefits back translation, we synthesize both multi-speaker pseudo data and singlespeaker pseudo data for experiments. For multispeaker data, we randomly select a speaker embedding from the training set for each sample. For single-speaker data, we use the average speaker embedding on the training set for all samples. As shown in Table 4, we find that the multi-speaker data is slightly better than the single-speaker data, probably because it is closer to the real ST dataset.

heading: 5.5 Results with Different Amounts of Monolingual/Parallel Data
text: In this section, we investigate the results under different amounts of monolingual and parallel data. First, we randomly sample 100 hours of ST data (corresponding to about 58K speech-translation pairs) from MuST-C En→De train set to simulate the low-resource setting. We then vary the amount of monolingual data used for back translation. As shown in Figure 3, we observe that (i) the BLEU score keeps increasing with the number of monolingual data regardless of the size of parallel data, and (ii) our approach is particularly effective in the low-resource setting. With only 12.5% monolingual data (about 0.6M), we observe a 3.4 BLEU improvement compared with the baseline. When all 4.6M monolingual data is used, we achieve a more significant boost of 5.4 BLEU.

heading: 5.6 Diverse BT4ST and Model Ensemble
text: Model ensemble is a widely-used technique in state-of-the-art MT systems (Barrault et al., 2020; Akhbardeh et al., 2021), which can combine differ-
ent single models (e.g., models trained on different data) to achieve stronger performance. To synthesize multiple diverse pseudo datasets from a single monolingual corpus, we introduce a simple Diverse BT4ST method. For the target-to-unit model, we activate the dropout modules during beam search decoding. For the unit-to-speech model, we randomly select the speaker embedding as above. By setting different random seeds, we can generate source speeches with different content and different speakers from the target text. In this way, we obtain multiple different pseudo datasets and train several models individually. We then combine these models by computing the token-level average log probability during decoding. As shown in Table 5, model ensemble can significantly boost performance. We achieve an average boost of 4.0 BLEU in three directions by ensembling five models.

heading: 5.7 Performance of the Target-to-unit Model
text: In this section, we report the performance of our target-to-unit models. We evaluate the performance with two metrics: Unit-BLEU and ASR-BLEU. Unit-BLEU is the BLEU score calculated on the reduced discrete unit sequence. ASR-BLEU is the BLEU score calculated on the transcribed text with the open-source ASR-BLEU toolkit20. As shown in Table 6, our target-to-unit models achieve promising results on MuST-C De→En, Fr→En, and Es→En tst-COMMON set, indicating that our proposed method can synthesize reasonable ST data.

heading: 6 Related Work
text: End-to-end ST End-to-end ST is theoretically attractive due to its advantages in alleviating error propagation and reducing latency, but it also faces
20https://github.com/facebookresearch/fairseq/ tree/ust/examples/speech_to_speech/asr_bleu
many challenges because of data scarcity. Therefore, researchers often leverage source transcripts to help train with auxiliary tasks. Plenty of existing work first pre-train the model with the ASR task (Bansal et al., 2019; Stoian et al., 2020; Wang et al., 2020b), MT task (Han et al., 2021; Fang et al., 2022; Ye et al., 2022; Fang and Feng, 2023; Zhou et al., 2023), or both together (Wang et al., 2020a; Alinejad and Sarkar, 2020; Le et al., 2021; Dong et al., 2021a; Xu et al., 2021), and then fine-tune the model with the ST task, which becomes the de-facto paradigm in recent ST studies. Le et al. (2020); Indurthi et al. (2021); Tang et al. (2021a,b); Dong et al. (2021b); Ye et al. (2021) adopt multitask learning to share knowledge among different tasks to improve ST. Bapna et al. (2021, 2022); Chen et al. (2022); Cheng et al. (2022); Ao et al. (2022); Tang et al. (2022); Zhang et al. (2022b) jointly pre-train the model with speech and text data to learn a unified space for both modalities, which achieve competitive results in ST. Jia et al. (2019); Lam et al. (2022) synthesize ST data with the help of MT model, TTS model and forced alignment tools. However, all of these studies assume that transcripts are available, which does not hold true for large numbers of unwritten languages in the world. Zhang et al. (2022a) first challenge this assumption and propose a set of practices to train a better ST model with only speech-translation pairs. In this paper, we extend this line of research and propose a back translation algorithm to utilize large-scale monolingual target data to improve ST
without transcripts.
Back Translation Back translation for NMT was first proposed by Sennrich et al. (2016) and is widely used in state-of-the-art NMT systems (Akhbardeh et al., 2021). Since then, many techniques have been proposed to improve BT such as Iterative BT (Hoang et al., 2018; Dou et al., 2020), Tagged BT (Caswell et al., 2019; Marie et al., 2020), Tag-less BT (Abdulmumin et al., 2021), MetaBT (Pham et al., 2021), HintedBT (Ramnath et al., 2021), and so on. Edunov et al. (2018) investigates different generation methods of BT in large-scale settings. Huang et al. (2021); Liu et al. (2021) focus on combining back translation with pre-training. Xu et al. (2022) combines synthetic data generated by beam search and sampling to better trade off the importance and quality of synthetic data. Despite the success of BT in MT, BT for ST is still a challenging problem. Nguyen et al. (2022b) introduces a pipeline BT method for speech-to-speech translation which cascades an unsupervised MT model and a TTS model. In contrast, our approach can synthesize pseudo ST data from the target-side monolingual corpus without relying on source transcripts.
Discrete Speech Units Discrete units, as a selfsupervised discrete representation of speech, have proved effective on many tasks, such as spoken language modeling (Lakhotia et al., 2021; Kharitonov et al., 2022; Gat et al., 2022; Borsos et al., 2022), speech-to-speech translation (Lee et al., 2022a,b; Popuri et al., 2022; Inaguma et al., 2022; Chen et al.; Li et al., 2022), speech emotion conversion (Kreuk et al., 2021), speech dialogue (Nguyen et al., 2022a), speech resynthesis (Polyak et al., 2021), speaking style conversion (Maimon and Adi, 2022), and so on. In this paper, we achieve back translation for ST by leveraging discrete units and further prove its effectiveness.

heading: 7 Conclusion and Future Work
text: In this paper, we develop a back translation algorithm for speech translation, which can synthesize pseudo ST data from monolingual target data without relying on transcripts. We utilize selfsupervised discrete units and achieve back translation by cascading a target-to-unit model and a unit-to-speech model. Experimental results on the MuST-C benchmark demonstrate the superiority of our approach, especially in low-resource settings.
This work focuses on enhancing ST when the source transcripts are unavailable, which is an essential but under-explored issue. We hope our work will draw more attention to this issue from researchers, which will benefit more real-world unwritten languages. In the future, we are interested in exploring how to combine advanced BT techniques (e.g., Iterative BT) with our approach.
Limitations
Our work provides an effective solution to augment ST when source transcripts are unavailable, which could benefit many unwritten languages. However, limited by the publicly available ST datasets, we use English as an unwritten language for experiments, which may slightly differ from realworld unwritten languages. Since we never use transcripts in our approach, we believe our work can shed some light on ST for real-world unwritten languages. We are glad to explore this if there are available datasets in the future.
Ethics Statement
Our model is developed and evaluated with publicly available datasets: MuST-C and WMT. The pre-trained models we use, like HuBERT and dvector models, are open and permitted for research purposes. Our use of the above artifacts is consistent with their intended use since they are widely used in the speech research community. Although our method could help the speech translation of unwritten languages like some dialects, the performance of the ST model still heavily relies on the amount of ST training data. Therefore, the output of the model is not always reliable and it would be better to be assisted by professional human translators in real applications.

heading: Acknowledgements
text: We thank all the anonymous reviewers for their insightful and valuable comments. This work was supported by National Key R&D Program of China (NO. 2018AAA0102502)

heading: A Details about the HiFi-GAN
text: The HiFi-GAN vocoder (Kong et al., 2020) consists of one generator and two discriminators. Next, we describe the model architecture and training objectives of HiFi-GAN.
Generator The generator is a convolutional neural network containing several stacked blocks. Each block comprises a transposed convolution layer followed by a multi-receptive field fusion (MRF) module. The transposed convolution layers upsample the input sequence to match the length of output waveforms. The MRF module includes multiple residual blocks with different receptive fields to model different patterns in parallel. The input to the generator is concatenated unit embedding Emb(z) and speaker embedding espkr.
Discriminator The discriminator consists of a Multi-Period Discriminator (MPD) and a MultiScale Discriminator (MSD). MPD is a mixture of sub-discriminators operating on equally spaced samples of the input audio. We adopt 5 subdiscriminators and set the space between samples to [2, 3, 5, 7, 11] respectively. MSD is also a mixture of sub-discriminators operating on different input scales: raw audio, ×2 downsampled audio, and ×4 downsampled audio. MPD and MSD can identify different periodic and consecutive patterns in the audio.
Training Objectives Using G to denote the generator and Dj to denote each sub-discriminator, we define the adversarial loss of G and Dj as:
Ladv(G;Dj) = Ex [ (1−Dj(x̂))2 ] , (4)
Ladv(Dj ;G) = Ex [ (1−Dj(x))2 +Dj(x̂)2 ] ,
(5)
where x denotes the ground truth audio and x̂ = G(Emb(z), espkr) denotes the synthetic audio.
Besides, there are two auxiliary training objectives. The first term measures the L1 distance between mel-spectrogram of the ground truth audio and synthetic audio:
Lmel(G) = Ex [∥ϕ(x)− ϕ(x̂)∥1] , (6)
where ϕ is the function to compute melspectrogram of the audio. The second term is a feature-matching loss which measures the difference in discriminator features between ground truth
audio and synthetic audio:
Lfm(G;Dj) = Ex [ L∑
i=1
1
Ni ∥δi(x)− δi(x̂)∥1
] ,
(7) where L denotes the number of layers in Dj , δi denotes the feature extractor of i-th layer, and Ni denotes the number of features in i-th layer.
Considering all sub-discriminators, the final training objectives are as follows:
LG = J∑
j=1
[Ladv(G;Dj) + λfmLfm(G;Dj)] (8)
+ λmelLmel(G), (9)
LD = J∑
j=1
Ladv(Dj ;G), (10)
where J is the number of sub-discriminators. We set λfm = 2 and λmel = 45.

heading: B Examples of Synthetic ST Data
text: To understand our approach more intuitively, we provide some examples of synthetic ST data in this section. Table 7, 8, and 9 show some examples of synthetic En→De, En→Fr, and En→Es ST data, respectively. For each sample, we give the original target text, generated reduced discrete units, generated source speech, and the corresponding transcript obtained with a state-of-the-art ASR model Whisper-Large21 (Radford et al., 2022). We observe that our method can generate reasonable source speech, though it may contain minor errors or duplicates. This explains why our method can enhance ST successfully.
We also provide an example of our proposed Diverse BT4ST method in Table 10, which generates multiple diverse source speeches from only one target text. We observe that the four outputs of our model are all reasonable and differ from each other in some way, which confirms that Diverse BT4ST is a simple and effective method to generate diverse pseudo data.
We provide the corresponding audio files of the above samples at https://bt4st.github.io/.
21https://github.com/openai/whisper
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations

heading: 3 A2. Did you discuss any potential risks of your work?
text: Ethics Statement

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract, Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? The data and pre-trained model we used are only for research purposes without violating the license.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Ethics Statement
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We use MuST-C and WMT datasets without modification, which are all widely used public datasets. We don’t think this needs much discussion.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 4
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4
C 3 Did you run computational experiments? Section 5

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section4, Section 5
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4, Section 5

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In the future, we are interested in exploring how to combine advanced BT techniques (e. g. , Iterative BT) with our approach. Limitations
Our work provides an effective solution to augment ST when source transcripts are unavailable, which could benefit many unwritten languages. However, limited by the publicly available ST datasets, we use English as an unwritten language for experiments, which may slightly differ from realworld unwritten languages. Since we never use transcripts in our approach, we believe our work can shed some light on ST for real-world unwritten languages. We are glad to explore this if there are available datasets in the future