UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Units
Year: 2023

Abstract:
Direct speech-to-speech translation (S2ST), in which all components can be optimized jointly, is advantageous over cascaded approaches to achieve fast inference with a simplified pipeline. We present a novel two-pass direct S2ST architecture, UnitY, which first generates textual representations and predicts discrete acoustic units subsequently. We enhance the model performance by subword prediction in the first-pass decoder, advanced two-pass decoder architecture design and search strategy, and better training regularization. To leverage large amounts of unlabeled text data, we pre-train the firstpass text decoder based on the self-supervised denoising auto-encoding task. Experimental evaluations on benchmark datasets at various data scales demonstrate that UnitY outperforms a single-pass speech-to-unit translation model by 2.5-4.2 ASR-BLEU with 2.83× decoding speed-up. We show that the proposed methods boost the performance even when predicting spectrogram in the second pass. However, predicting discrete units achieves 2.51× decoding speed-up compared to that case.

heading: 1 Introduction
text: Automatic speech translation to another language is an indispensable technology for international communications, with the spread of social media and virtual communications nowadays. A traditional approach of speech-to-speech translation (S2ST) is to cascade automatic speech recognition (ASR), machine translation (MT), and textto-speech (TTS) components, each of which is optimized separately on different data (Lavie et al., 1997; Nakamura et al., 2006; Wahlster, 2013). With the emergence of sequence-to-sequence models (Sutskever et al., 2014; Cho et al., 2014; Bahdanau et al., 2015), however, it is getting prevailing to adopt a direct approach1. This approach
1Lee et al. (2022a) defines a direct S2ST model as a model that does not use intermediate text representations while Jia
consists in translating input speech into the other language based on a single architecture with fewer components than the cascaded systems (Jia et al., 2019b; Tjandra et al., 2019; Zhang et al., 2021). The direct approach is attractive for building a lowlatency system with a simplified pipeline, thus reducing developing costs. However, direct S2ST models suffer from poor performance due to data scarcity, similar to direct speech-to-text translation (S2TT) models (Bérard et al., 2016). In the field of S2TT, data shortage has been addressed by leveraging pre-training (Bérard et al., 2018; Wang et al., 2021c; Tang et al., 2022), multi-task learning (Weiss et al., 2017; Tang et al., 2021), pseudo labeling (Jia et al., 2019a; Pino et al., 2020), knowledge distillation (Liu et al., 2019; Inaguma et al., 2021b). Consequently, the translation quality of direct S2TT models is approaching that of cascaded S2TT models (Ansari et al., 2020; Anastasopoulos et al., 2021). These techniques have also shown the effectiveness for direct S2ST models and led to a decent performance (Kano et al., 2021; Dong et al., 2022; Jia et al., 2022a; Popuri et al., 2022).
Recent works (Lee et al., 2022a,b) propose to model discrete acoustic units, extracted from HuBERT (Hsu et al., 2021), instead of a continuous speech signal that enables usage of a standard crossentropy loss during training. This speech-to-unit translation (S2UT) model significantly shortens the target sequence length and thus makes training and inference more efficient. The discrete units are directly converted to the waveform with a unit-based neural vocoder (Polyak et al., 2021) bypassing spectrogram representation. On the other hand, Translatotron2 (Jia et al., 2022b) decomposes the target representations into linguistic and acoustic counterparts explicitly. The former predicts a phoneme
et al. (2022b) defines it as a model that directly predicts the target spectrogram. In this paper, we use a more general definition that the entire architecture is optimized jointly and the translation is conducted in a more direct way. We do not include a vocoder in the training pipeline of all direct models.
15655
sequence first, and the latter synthesizes the target spectrogram conditioned on the continuous representation of the linguistic part.
This paper presents a novel two-pass direct S2ST architecture, dubbed UnitY , which takes the best of both worlds of the S2UT model and Translatotron2. Unlike Translatotron2, UnitY models linguistic sequences using subwords (first pass) instead of phonemes, and it models speech as a discrete sequence of acoustic units (second pass). To achieve better translation quality and decoding efficiency, UnitY consists of a deep text decoder and a shallow unit decoder and enables better generalization to the first-pass decoder. We further introduce a textto-unit (T2U) encoder between the two decoders to bridge the gap between textual and acoustic representations. Following the success of large-scale pre-training, we leverage unlabeled text effectively to pre-train the first pass text decoder with multilingual BART (mBART) (Liu et al., 2020) at the subword level.
Extensive experiments show the superiority of the UnitY S2ST system measured by both translation quality and runtime efficiency. First, UnitY achieves 4.2, 3.7, and 2.5 ASR-BLEU improvements over the S2UT model on the Fisher Es→En (Post et al., 2013), CVSS-C (Jia et al., 2022c), and multi-domain En↔Es (Popuri et al., 2022) corpora, respectively. The improvement holds even with high-resource data and pre-training. In addition, our proposed design improves Translatotron2 as well, indicating its versatility for twopass direct S2ST architectures regardless of the choice of the target. Second, UnitY achieves 2.83× and 2.51× decoding speed-ups over the S2UT and improved Translatotron2 models, respectively. A combination of the aforementioned improvements suggests the UnitY design as a starting point for further improvements in direct S2ST. 2

heading: 2 UnitY
text: In this section, we propose UnitY , a two-pass direct S2ST model that generates subwords and discrete acoustic units subsequently. Hereafter, we refer to discrete acoustic units as discrete units for brevity. Let X denote a source speech input, and Y = (y1, . . . , yM ) and U = (u1, . . . , uL) be the corresponding reference text translation and discrete unit sequences in the target language, respectively. Note that there is no duration information
2Code will be available upon the paper acceptance.
for each discrete unit in U , because consecutive units are collapsed (Lee et al., 2022a).

heading: 2.1 Architecture
text: The overall architecture of UnitY is shown in Figure 1. UnitY consists of four modules: speech encoder, first-pass text decoder, text-to-unit (T2U) encoder, and second-pass unit decoder. We build the speech encoder based on Conformer (Gulati et al., 2020), which augments Transformer (Vaswani et al., 2017) with a convolution module, while implementing the rest three modules based on Transformer. UnitY has five major architecture modifications from Translatotron2 (Jia et al., 2022b), (1) generating subwords instead of phonemes in the first pass, (2) generating discrete units instead of spectrograms in the second pass to bypass duration modeling, (3) replacing Long Short-Term Memory (LSTM) (Hochreiter and Schmidhuber, 1997) layers with Transformer layers in both decoders, (4) introducing a T2U encoder between the two decoders, and (5) assigning more model capacities to the first pass.
First-pass text decoder The first-pass text decoder TDec generates a sequence of subwords Y autoregressively by attending the speech encoder output H . The training objective of the first pass is to minimize the direct S2TT loss Ls2t as:
Ls2t(Y |X) = − 1
M
M∑
i=1
logPs2t(yi|X,Y<i)
= − 1 M
M∑
i=1
logPs2t(yi|Dtexti )
Dtexti = TDec(H,Y<i),
where Dtexti ∈ Rdmodel is the i-th continuous decoder state right before projecting it to the logit. We consider that Dtext contains rich acoustic information in addition to contextual information thanks to multiple multi-head cross-attention over H .
There are five advantages of generating subwords instead of phonemes. First, the sequence length is considerably reduced, leading to better training and inference efficiencies (Cherry et al., 2018). Second, using large vocabularies improves the translation quality of the first pass (Gowda and May, 2020). Third, the text output helps the audience understand the translation content while listening to the audio. Fourth, our approach can easily scale to more target languages, as it is unnecessary to prepare separate grapheme-to-phoneme (G2P) models for each target language. Last, readable text can be generated without any complicated post-processing such as WFST (Mohri et al., 2002; Bahdanau et al., 2016).
T2U encoder A bidirectional T2U encoder T2UEnc transforms the continuous states of the first-pass decoder Dtext ∈ RM×dmodel into Z ∈ RM×dmodel as Z = T2UEnc(Dtext). The T2U encoder bridges the gap in representations between text and unit decoders without changing the sequence length.
Second-pass unit decoder The second-pass unit decoder UDec generates a sequence of discrete units U autoregressively by attending to only the T2U encoder output Z. The training objective of the second pass is to minimize Ls2u similar to the S2UT task while being conditioned on Y as:
Ls2u(U |X,Y ) = − 1
L
L∑
i=1
logPs2u(ui|X,Y,U<i)
= − 1 L
L∑
i=1
logPs2u(ui|Duniti )
Duniti = UDec(Z,U<i) = UDec(H,Y,U<i),
where Duniti ∈ Rdmodel is the i-th continuous decoder state right before projecting it to the logit. The unit decoder does not attend to H to synchronize the text and unit outputs, similar to the motivation in (Jia et al., 2022b). In other words, we do not expect that the second-pass decoder corrects translation errors from the first-pass decoder.3 Once the unit generation finishes, a separate unit-based vocoder (Polyak et al., 2021) converts the discrete units to the waveform with duration prediction of each discrete unit (Lee et al., 2022a). The total training objective of UnitY, Ltotal, is formulated
3We also investigate attending to the speech encoder output with an additional cross-attention, but it does not lead to an improvement in ASR-BLEU. We discuss this in §5.1
as follows:
Ltotal = Ls2u(U |X,Y ) +ws2tLs2t(Y |X), (1) where ws2t is a weight for the S2TT loss.

heading: 2.2 Text decoder pre-training
text: Similar to ASR and S2TT studies (Baevski et al., 2020; Li et al., 2021), S2ST models also benefit from self-supervised pre-training (Jia et al., 2022a; Popuri et al., 2022), especially for the speech encoder. In addition to the speech encoder pretraining with wav2vec2.0 (Baevski et al., 2020), Popuri et al. (2022) initializes the unit decoder of the single-pass S2UT model with a unit-based mBART (u-mBART), an encoder-decoder model pre-trained with discrete units converted from a large amount of unlabeled speech data. However, unlabeled text data cannot be leveraged for the single-pass decoder pre-training, although it is more accessible in many written languages.
To fully leverage the unlabeled text data, we initialize the first-pass decoder of UnitY with a text-based mBART (t-mBART) pre-trained with unlabeled text data. Following Li et al. (2021); Popuri et al. (2022), we freeze parameters in the feed-forward network (FFN) of the text decoder during S2ST fine-tuning. We initialize the T2U encoder and second-pass unit decoder randomly.

heading: 2.3 Search algorithm
text: During inference, we perform two-pass beam search decoding. First, we find the most probable text hypothesis Ŷ in the first-pass decoder using beam search with a beam size of B1st. We then feed continuous decoder states Dtext corresponding to Ŷ to the T2U encoder. Next, we generate the most probable discrete unit sequence Û in the second-pass decoder by another beam search with a beam size of B2nd. Finally, Û is taken as input to a separate unit-based vocoder to generate the waveform. We find it more effective to assign a larger beam size to the first pass, i.e., B1st > B2nd, because there is more diversity among beam candidates than the second pass. The computation time is also reduced since the sequence length of text is much shorter than that of discrete units. Therefore, we use B2nd = 1 unless otherwise noted. We present the pseudo algorithm in Appendix A.

heading: 2.4 Deep-shallow two-pass decoders
text: By increasing the number of layers, we assign more model capacities to the first-pass decoder than the
second-pass decoder. We refer to this as deepshallow two-pass decoders. This capacity assignment improves translation quality and inference efficiency simultaneously because of a shorter sequence length in the first pass. A practical capacity assignment for the MT task is studied in Kasai et al. (2021) by trading the number of layers between the encoder and decoder. In this work, we focus on the two-pass decoders for the S2ST task.

heading: 3 Experimental setting

heading: 3.1 Data
text: We use three datasets: Fisher Es→En (Post et al., 2013) (170 hours), CVSS-C (Jia et al., 2022c) (547 hours), and mutli-domain En↔Es (Popuri et al., 2022) (20k hours for En→Es, 14k hours for Es→En) corpora. We combine all 21 language directions to English in the CVSS-C corpus to train a single X-to-En multilingual model. The En→Es part in the multi-domain corpora consists of Europarl-ST (Iranzo-Sánchez et al., 2020), MustC (Di Gangi et al., 2019), TEDLIUM3 (Rousseau et al., 2012), Librispeech (Panayotov et al., 2015), and Common Voice (Ardila et al., 2020). The Es→En part consists of CoVoST2 (Wang et al., 2021b), Europarl-ST, and mTEDx (Elizabeth et al., 2021), Common Voice, and multilingual Librispeech (MLS) (Pratap et al., 2020). More details are described in Appendix D.

heading: 3.2 Pre-processing
text: We follow the same pre-processing as (Lee et al., 2022a,b; Popuri et al., 2022) for acoustic feature extraction, discrete unit extraction, and text normalization. We also discarded over-generated target speech/unit by TTS/T2U models. More details are described in Appendix E.

heading: 3.3 Pre-training
text: We use the same En/Es wav2vec2.0 and En-Es umBART models as Popuri et al. (2022). We train a multilingual w2v-BERT (Chung et al., 2021) model trained on 51 languages with the same setting as Jia et al. (2022a). For text decoder pre-training, we use the same En-Es and 50-language t-mBART models as Wang et al. (2022) and Tang et al. (2020), respectively. We describe the training details and list model URLs in Appendix F.

heading: 3.4 Baseline
text: We build two cascaded S2ST systems and four direct S2ST systems. All speech encoders are based on Conformer. When pre-training the speech encoder of direct S2ST systems with wav2vec2.0/w2v-BERT, we pre-train ASR and S2TT models in the cascaded systems with the same wav2vec2.0/w2v-BERT for a fair comparison. We also pre-train the text decoder of the ASR and S2TT models with t-mBART in that case.
Cascaded (ASR→MT→TTS) We combine a Conformer ASR, a Transformer MT, and a Transformer TTS model. We set the reduction factor of TTS models to 4.
Cascaded (S2TT→TTS) We combine a Conformer direct S2TT model and a Transformer TTS model.
S2SpecT We build a direct S2ST model that predicts spectrogram with a single Transformer decoder, similar to Lee et al. (2022a) (Figure 2a). We refer to it as S2SpecT hereafter. We set the reduction factor of the spectrogram decoder to 3.
S2SpecT2 We train S2SpecT2, an improved version of Translatotron2, by enhancing the architecture and training with the proposed methods
for UnitY. First, we replace phoneme targets with subwords in the first pass (Figure 2b). Second, we replace LSTM decoders with Transformer decoders. Third, we introduce an additional textto-spectrogram (T2S) encoder between text and spectrogram decoders. The second-pass decoder attends to the T2S encoder output only. Fourth, we use an autoregressive Transformer decoder instead of a non-attentive Tacotron (NAT) (Shen et al., 2020) for the second-pass decoder. Last, we apply R-Drop to the first-pass decoder. We use the same reduction factor as S2SpecT.
S2UT We train a direct S2ST model that predicts discrete units with a single Transformer decoder (Lee et al., 2022a) (Figure 2c).

heading: 3.5 Architecture
text: Let N1st, N2nd, and Nt2u be the depth of the first-pass decoder, second-pass decoder, and T2U encoder of UnitY, respectively. We set (N1st,N2nd,Nt2u) to (4,2,2) on Fisher and CVSSC. On the multi-domain corpus, we use (12,2,2) when pre-training the first-pass decoder with tmBART. Otherwise, we use (6,6,2). We describe the other configurations in Appendix G.

heading: 3.6 Training
text: We apply R-Drop (Wu et al., 2021) regularization to all tasks that predict discrete symbols, except the MT task. The training objective of each model with R-Drop is defined in Appendix C. We implement our models based on the Fairseq toolkit (Ott et al., 2019; Wang et al., 2020). The detailed training hyperparameters are described in Appendix H.

heading: 3.7 Decoding
text: We use a beam width of 10 for ASR, S2TT, and S2UT models. For UnitY, we set B1st and B2nd to 10 and 1, respectively. We use a beam width of 10 for the first-pass decoder in S2SpecT2.

heading: 3.8 Vocoder
text: We use a HiFi-GAN vocoder (Kong et al., 2020) to convert spectrograms to the waveform for TTS and direct speech-to-spectrogram models. We use a unit-based HiFi-GAN vocoder (Polyak et al., 2021) to convert discrete units to the waveform for direct speech-to-unit models. Both the vocoders are trained separately.

heading: 3.9 Evaluation
text: Following Lee et al. (2022a), we use a pre-trained ASR model to transcribe the generated target speech and calculate BLEU scores (Papineni et al., 2002), referred to as ASR-BLEU. The ASR model is fine-tuned from a wav2vec2.0 with the connectionist temporal classification (CTC) objective (Graves et al., 2006). We use the sacrebleu toolkit (Post, 2018) to calculate the BLEU scores.

heading: 4 Experimental results
text: In this section, we present the experimental results on three corpora. We study various modeling choices from the perspective of target representation (spectrogram v.s. discrete unit) and decoder architectures (single pass v.s. two pass) in supervised and semi-supervised settings. We also benchmark the decoding efficiency of direct S2ST models.

heading: 4.1 CVSS-C
text: The results on CVSS-C are listed in Table 1. We first compared four direct systems trained from scratch (B6, B15, B18, B21), and UnitY (B21) achieved the best ASR-BLEU. The encoder pretraining with the S2TT model in the cascaded system (B3) improved ASR-BLEU of all the direct S2ST models (B7, B16, B19, B22), similar to Jia et al. (2022c).4 In this case, S2SpecT2 (B16) also achieved similar translation quality to UnitY (B22). Still, UnitY outperformed the S2UT model (B19) by 1.6 ASR-BLEU on average, indicating that the two-pass decoding was the main factor of the improvements. S2SpecT2 (B16) outperformed Translatotron2 (Jia et al., 2022b) (B11) by 1.1 ASRBLEU on average, from which we can confirm that parts of the proposed methods can generalize to the other S2ST architecture.5 Compared to the best cascaded system (B2), the two-pass models (B16, B19) showed better translation quality.
We also pre-trained the speech encoder of all models with multilingual w2v-BERT, the first-pass text decoder of two-pass models with text-based mBART (t-mBART), and the decoder of the S2UT model with unit-based mBART (u-mBART), respectively. Among them (B4, B8, B12, B20, B23), UnitY (B23) showed the best ASR-BLEU.
4Unlike Jia et al. (2022c), we trained the S2TT model with an auxiliary ASR task from scratch instead of pre-training the encoder with that of an ASR model.
5B9 predicts phonemes while B15 predicts subwords in the first pass.
UnitY still outperformed Translatotron2 with a joint speech-text pre-training with mSLAM (Bapna et al., 2022) (B13) and TTS augmentation (B14) by 5.2 and 2.5 ASR-BLEU on average, respectively. The full results in each language direction are presented in Appendix I.

heading: 4.2 Multi-domain En↔Es
text: We present results on the multi-domain corpora (Popuri et al., 2022) in Table 2. C1’, C2’, and C5’ are our improved models of C1, C2, and C5, respectively.6 We observed that UnitY with first-pass decoder pre-training with t-mBART (C7) improved the S2UT model with decoder pretraining with u-mBART (C5’) by 1.3 and 2.5 ASRBLEU on average in En→Es and Es→En, respectively. This confirms the effectiveness of the twopass modeling in the high-resource scenario. Furthermore, UnitY without decoder pre-training (C6) already outperformed C5’ and degraded from C7 only slightly. Comparing UnitY and S2SpecT2, we cannot spot a clear winner. UnitY outperformed S2SpecT2 in Es→En on Europarl-ST and mTEDx, but S2SpecT2 performed better in En→Es. The proposed text decoder pre-training helped S2SpecT2 performance too, especially in En→Es (C4). Finally, we also confirmed that UnitY approached the performance of a strong cascaded system and even outperformed it on Must-C.
6We improved C1 and C2 by R-Drop and better hyperparameters. C5 was also improved by hyperparameter tuning and checkpoint averaging.

heading: 4.3 Decoding efficiency
text: We evaluated the decoding efficiency of direct S2ST models. We measured the runtime and total number of floating point operations (FLOPs) on an Intel® Xeon® Gold 6230 CPU. We randomly sampled 500 utterances from the multi-domain Es→En dev set while keeping the ratio of the number of samples per domain. Note that we also took the vocoder inference into account.
The results in Figure 3 showed that UnitY achieved 2.51× and 2.83× decoding speed-ups over S2SpecT2 and S2UT models, respectively. These confirms the efficiency of discrete unit prediction and two-pass decoding, thanks to reduced output sequence lengths. Deep-shallow two-pass decoders also improved the decoding speed a lot. We found that the translation quality of the twopass models improved by increasing the beam width of the first-pass decoder up to 10. On the other hand, the quality did not degrade significantly by decreasing the beam width of the second-pass decoder down to 1, i.e. greedy decoding. This indicates that the first pass involves more challenges in the modeling pipeline. Therefore, we can obtain better translation quality and decoding speed by assigning more computation time to the first pass.
We also present the results of FLOPs in Appendix I. To summarize, UnitY achieved 1.65× and 3.19× FLOPs reduction over S2SpecT2 and S2UT models, respectively.

heading: 4.4 Fisher
text: We also show the results on Fisher in Appendix I. Although the trend was consistent with CVSS-C, a notable exception was that S2SpecT2 outperformed UnitY when pre-training the speech encoder with wav2vec2.0. However, UnitY has an advantage of decoding efficiency over S2SpecT2.

heading: 5 Analysis
text: In this section, we conduct analyses to shed light on the source of improvements in UnitY. We also study whether the same techniques used for UnitY are helpful for S2SpecT2. We use the multi-domain Es→En corpus, but pseudo-labeled ASR data is excluded for quick exploration, resulting in 196- hour source speech. We report average dev scores over three runs with different random seeds.7

heading: 5.1 Ablation study
text: We first conducted an ablation study for two-pass direct S2ST models in Table 3. We evaluated the translation quality of outputs from both decoders. An additional T2U/T2S encoder was essential for bridging the gap in representations between the first-pass and second-pass decoders, especially for S2SpecT2 (D2, D6). We attribute this to the fact that the gap in representations between text and spectrogram is larger than between text and discrete units. R-Drop was also beneficial for boosting the translation quality of the first-pass decoder, which improved the final performance accordingly (D3, D7). Moreover, we investigated adding another cross-attention over the speech encoder output to the unit decoder, as discussed in §2.1. We expected that the first-pass decoder output lost useful information to generate target speech faithful to source speech. We explored parallel (parallel, D8) and sequential (sequential, D9) cross-attention, similar to (Zhu et al., 2019), but neither showed any improvement. The first-pass decoder already extracted source acoustic information well via multiple cross-attention modules. We also show the results on Fisher in Appendix I.
7We removed three long utterances from the mTEDx dev set to fit the GPU memory.

heading: 5.2 Output unit for first-pass decoder
text: We studied optimal granularity of the output unit for the first-pass decoder in two-pass direct S2ST models. We explored phonemes, characters, and 2k subwords units. The results in Table 4 showed that the subword unit (E6) was the most effective for the first-pass decoder in both UnitY and S2SpecT2 thanks to a better translation quality. Moreover, it gained the largest decoding speed-up. We also show the results on Fisher in Appendix I.

heading: 5.3 Capacity assignment to two-pass decoders
text: We sought to effectively assign the model capacity to the two decoders in UnitY to obtain a better translation quality. The results in Table 5 showed that a 12-layer text decoder with a two-layer unit decoder (G8) was the best in translation quality and decoding speed when initializing the first-pass decoder randomly (G1-G6,G8). Pre-training the first-pass decoder with t-mBART (G9) brought a
large ASR-BLEU gain with a slight speed degradation compared to G8.8 It was sufficient to have a two-layer unit decoder in that case (G9-G11). We also pre-trained the second-pass decoder with u-mBART while initializing the text decoder randomly (G7) or with t-mBART (G12), but neither improved the performance further. Therefore, it is most effective to pre-train the deep text decoder only and keep the unit decoder shallow. Note that G8 is faster than G9 because of the smaller subword vocabulary size (2k v.s. 65k).

heading: 5.4 Data scale
text: Improving the translation quality of S2ST models on low-resource data is crucial since collecting a large amount of training data is challenging. We compared translation quality of direct S2ST models at various training data scales in Figure 4. We observed that UnitY consistently outperformed the S2SpecT2 and S2UT models when the data size was no less than 50 hours. The text decoder pre-training became less effective as the data size increased, consistent with an observation in §4.2, where the improvement in En→Es (+1.3) was smaller than Es→En (+2.5). However, pretraining the text decoder of UnitY was essential for obtaining decent performances in the low-resource settings (≤ 50 hours).

heading: 6 Related works
text: Two-pass sequence generation Two-pass decoding has advantages of maintaining the end-to-end optimization capability while inheriting the benefits of a cascading approach. Xia et al. (2017); Hu et al. (2020) incorporate an additional search process to find a better output. Dalmia et al. (2021)
8We set the depth of the first-pass decoder to 12 because of the availability of the off-the-shelf t-mBART model.
reranks the intermediate hypotheses using an external module such as a language model. Zhao et al. (2019) injects specific information in the intermediate decoder to bias the output toward the desired domain. Sainath et al. (2019) provides an intermediate output to users before generating the final output for streaming applications. The two-pass approach makes the optimization tractable, which has advanced performance of speech translation models (Anastasopoulos and Chiang, 2018; Sperber et al., 2019; Sung et al., 2019; Dalmia et al., 2021; Inaguma et al., 2021a; Yan et al., 2022; Jia et al., 2022b).
Direct speech-to-spectrogram translation Translatotron (Jia et al., 2019b) is the first direct S2ST model but suffered from poor performance even with auxiliary ASR and S2TT tasks. Kano et al. (2021) subsequently pre-trains the components with ASR and S2TT models, which is more effective for distant language pairs. Translatotron2 (Jia et al., 2022b) significantly improves Translatotron by incorporating two-pass decoding. We showed that our methods further improved Translatotron2.
Direct speech-to-unit translation Direct speechto-unit translation models predict discrete units rather than spectrogram. Tjandra et al. (2019) uses vector-quantized variational autoencoder (Van Den Oord et al., 2017) while Lee et al. (2022a) used HuBERT (Hsu et al., 2021) to extract target discrete units. Lee et al. (2022b) normalizes speaker identity of real target speech using a CTC-based speech-to-unit model. Huang et al. (2022) further improves the normalization by considering rhythm, pitch, and energy.

heading: 7 Conclusion
text: We proposed UnitY, a novel efficient two-pass direct S2ST model that subsequently generates both text and discrete unit outputs. We improved the model performance by predicting subwords in the first pass, bridging decoder representations by an additional encoder, deep-shallow two-pass decoders, regularizing the training with R-Drop, and pre-training the first-pass decoder with text-based mBART. Experimental evaluations demonstrated that UnitY outperformed a single-pass S2UT model consistently in translation quality and inference speed. We showed that the proposed methods improve the two-pass direct speech-to-spectrogram model as well, confirming their versatility. Still,
UnitY achieved 2.51× decoding speed-up over the case.

heading: 8 Limitation
text: Since two-pass direct S2ST models require linguistic units as the target for the first-pass decoder, they cannot be used when the target language is unwritten. Compared to cascaded S2ST systems, direct S2ST systems require more data preparation steps, including training a HuBERT model, synthesizing target speech with a TTS model, extracting discrete units with the HuBERT model, and training a unit-based vocoder, etc. Moreover, the target audio quality of direct speech-to-unit systems relies on the quality of discrete units generated by selfsupervised discrete models. It further depends on the availability of speech data to train HuBERT models for the target languages.
Because S2ST systems could generate speech that does not necessarily represent the source speech’s content, there is a potential risk of conveying wrong information.

heading: Acknowledgement
text: We would like to thank Justine Kao and Carleigh Wood for the help on human evaluation.

heading: A Pseudo algorithm for two-pass beam search decoding
text: Algorithm 1 shows the two-pass beam serach decoding algorithm of UnitY as discussed in §2.3. We first encode a source speech X with the speech encoder SpeechEnc and map it to the encoder output H .
The first-pass decoder takes H as input and generates a text sequence. BeamSearch1 is a first-pass beam search function that takes an empty hypothesis set Ω1st and returns the beam candidates. We take the best text hypothesis Ŷ and get the corresponding decoder output Dtext from a cache via the GetHiddenStateFromCache function. Next, the T2U encoder T2UEnc takes Dtext as input and maps it to the output Z.
The second-pass decoder takes H and Z as inputs and generates a discrete unit sequence. BeamSearch2 is a second-pass beam search function that takes an empty hypothesis set Ω2nd and returns the beam candidates. We take the best discrete unit hypothesis Û . Finally, the unit-based vocoder UnitVocoder converts Û to the waveform Ŵ .

heading: B Training with R-Drop
text: UnitY introduces an intermediate S2TT sub-task to make the optimization tractable while maintaining the end-to-end differentiability. However, the easier S2TT task is more likely to overfit than the primary S2UT task. To tackle this problem,
we apply a more effective regularization based on R-Drop (Wu et al., 2021) to the first-pass decoder in addition to standard regularization such as dropout (Srivastava et al., 2014) and label smoothing (Szegedy et al., 2016). Theoretically, R-Drop reduces the inconsistency of model predictions between training and inference by dropout, thus improving the generalization ability. R-Drop duplicates the network input during training and calculates two output probability distributions with different dropout masks. Then, a constraint is introduced by minimizing the Kullback–Leibler (KL) divergence loss between the two probability distributions. We apply R-Drop to both text and unit decoders. The total training objective of UnitY with R-Drop, Ltotal, is modified from Eq. (1) as follows:
Ltotal = 2∑
i=1
Ls2u(U |Xi, Y ) +αLs2ukl (X1,X2)
+ws2t(
2∑
i=1
Ls2t(Y |Xi) + βLs2tkl (X1,X2)), (2)
where Xi is a duplicated input from X , Ls2ukl and Ls2tkl are KL losses for the unit and text decoders, ws2t is a weight for the S2TT loss, and α and β are weights for the KL losses, respectively. We implement R-Drop by duplicating inputs instead of feeding them to the network twice.
Given a set of unique inputs X, the general KL loss Lkl in R-Drop is formulated as follows:
Lkl(X1,X2) = 1
2 (Dkl(P (·|X1)||P (·|X2) +Dkl(P (·|X2))||P (·|X1))),
where Xi is a duplicated input from X, Dkl is a KL divergence, and P is a categorical probability distribution.

heading: C Training objective
text: In this section, we describe training objectives for the baseline S2ST models. In addition to the primary S2ST/S2UT task, we introduce auxiliary S2TT and ASR tasks. We adopted an auxiliary character-level ASR task for the direct S2ST models trained from scratch on Fisher, regardless of the choice of the output unit in the first-pass decoder. We did not use the ASR task in the rest settings.
S2SpecT The architecture of S2SpecT is shown in Figure 2a. Given the target spectrogram S, translation Y , and transcription Ysrc, corresponding to a source speech X , the training objective of S2SpecT is formulated as:
Ltotal = Ls2s(S|X) +ws2tLs2t(Y |X) +wasrLasr(Ysrc|X), (3)
where Ls2s is the primary S2ST loss, Ls2t is the auxiliary S2TT loss, Lasr is the auxiliary ASR loss, ws2t is a weight for the S2TT loss, and wasr is a weight for the ASR loss, respectively. Note that RDrop is not used because the output of the primary S2ST task is continuous.
We adopt the autoregressive decoder of Transformer TTS (Li et al., 2019) as the spectrogram decoder. Therefore, Ls2s is defined as a sum of the L1 loss L1, L2 loss L2, and end-of-sentence (EOS) prediction loss Leos as follows:
Ls2s(S|X) = L1 +L2 +Leos.
S2SpecT2 The architecture of S2SpecT2 is shown in Figure 2b. The training objective of S2SpecT2 is formulated as:
Ltotal = 2∑
i=1
Ls2s(S|Xi, Y )
+ws2t( 2∑
i=1
Ls2t(Y |Xi) + βLs2tkl (X1,X2))
+wasr( 2∑
i=1
Lasr(Ysrc|Xi) + γLasrkl (X1,X2)),
(4)
where Xi is a duplicated input from X , Ls2tkl is the R-Drop’s KL loss for the first-pass decoder, Ls2tkl is the R-Drop’s KL loss for the auxiliary ASR decoder, and β and γ are the corresponding weights for the R-Drop’s KL losses, respectively. Unlike Eq. (3), the primary S2ST task depends on the output from the first-pass decoder. We apply RDrop to the S2TT and ASR tasks only. We also investigated applying R-Drop to the second-pass spectrogram decoder by minimizing the difference of two outputs in the continuous space, but the training was unstable.
S2UT The architecture of S2UT is shown in Figure 2c. In addition to the primary S2UT loss and auxiliary S2TT and ASR losses, we use a CTC
loss on top of the unit decoder following Lee et al. (2022a). The training objective of the S2UT model is formulated as:
Ltotal = 2∑
i=1
Ls2u(U |Xi) +αLs2ukl (X1,X2)
+wctc
2∑
i=1
Lctc(Y |Duniti )
+ws2t( 2∑
i=1
Ls2t(Y |Xi) + βLs2tkl (X1,X2))
+wasr(
2∑
i=1
Lasr(Ysrc|Xi) + γLasrkl (X1,X2)),
(5)
where Ls2u is the primary S2UT loss, Ls2ukl is the R-Drop’s KL loss for the unit decoder, Lctc is the CTC loss, Duniti is the unit decoder output for the i-th forward pass, α is a weight for the R-Drop’s KL loss, and wctc is a weight for the CTC loss, respectively. Unlike Eq. (2), there is no dependency between the primary S2UT task and auxiliary S2TT task except for sharing the same encoder.
S2TT, ASR We also apply R-Drop to S2TT and ASR tasks. The training objective of the S2TT model is formulated as:
Ltotal = 2∑
i=1
Ls2t(Y |Xi) + βLs2tkl (X1,X2). (6)
Similarly, the training objective of the ASR model is formulated as:
Ltotal = 2∑
i=1
Lasr(Ysrc|Xi) + γLasrkl (X1,X2).
(7)

heading: D Data
text: Fisher Es→En (Post et al., 2013) This corpus contains 170-hour Spanish conversational telephone speech with the corresponding transcriptions as well as the English translations. The target speech is synthesized by a high-quality inhouse TTS model trained with a single female speaker (Lee et al., 2022a).
CVSS-C (Jia et al., 2022c) CVSS is a public multilingual S2ST corpus based on CoVoST2 (Wang et al., 2021b). It covers 21 language
directions to English. We use the CVSS-C part of the CVSS corpus, in which a single-speaker female TTS synthesizes the target speech. We combine all language directions to train a single X-to-En multilingual model.
Multi-domain En↔Es (Popuri et al., 2022) Following Popuri et al. (2022), we use all samples from multiple public S2TT corpora in each direction to improve the robustness of model training (Jia et al., 2022b; Chan et al., 2021). We also use all samples from validation sets in all domains for checkpoint selection. We further augment the S2ST training data by pseudo-labeling ASR corpora with MT and T2U/TTS models. We use the TTS model in the cascaded system to synthesize
the target speech for direct speech-to-spectrogram models. For direct speech-to-unit models, we use a T2U model (Lee et al., 2022b) to generate discrete units on the ASR corpora and the TTS+HuBERT pipeline for the S2T corpora. Both T2U and TTS models are based on Transformer. We train En and Es T2U/TTS models on the LJSpeech (Ito and Johnson, 2017) and CSS10 (Park and Mulc, 2019) corpora, respectively.
For En→Es, we use all samples from EuroparlST (Iranzo-Sánchez et al., 2020) and MustC (Di Gangi et al., 2019) and augment the training data by TEDLIUM3 (Rousseau et al., 2012), Librispeech (Panayotov et al., 2015), and Common Voice (Ardila et al., 2020), resulting in 3180-
hour source speech. We removed samples overlapped with mtedx dev/test sets from TEDLIUM3. For Es→En, we use all samples from CoVoST2, Europarl-ST, and mTEDx (Elizabeth et al., 2021), and augment the training data by Common Voice and multilingual Librispeech (MLS) (Pratap et al., 2020), resulting in 1404-hour source speech. In Table 6, we list all the datasets used in each task.

heading: E Pre-processing
text: Speech We convert source audio to 16kHz and generate target speech with 22kHz. When extracting discrete units, we downsample the target speech to 16kHz. For filterbank features, we extract 80- dimensional coefficients on both the source and target sides. We apply utterance-level cepstral meanvariance normalization to both inputs.
Discrete units We extract discrete units with an English HuBERT trained on Librispeech after performing k-means clustering with 100 clusters on Fisher (Lee et al., 2022a). For the rest corpora, we extract discrete units with a multilingual HuBERT (mHuBERT) (Popuri et al., 2022) trained on En, En, and Fr parts of VoxPopuli (Wang et al., 2021a) with the number of k-means clusters of 1000.
Text We lowercase text data and remove all punctuation marks except for apostrophes. When initializing the text decoder in two-pass direct S2ST models randomly, we build vocabularies of 1k, 6k, and 2k unigram subword units (Kudo, 2018) with the SentencePiece toolkit (Kudo and Richardson, 2018) for the Fisher, CVSS-C, and multi-domain corpora, respectively. When pre-training the text decoder with t-mBART, we use the same vocabulary as t-mBART. The reference target translation to calculate ASR-BLEU is normalized with lowercasing, removal of punctuation marks, conversion of digits to spoken forms, and removal of non-verbal words in parentheses like “(Applause)” or “(Music).”
Data filtering For discrete unit generation with a T2U model, we found that target discrete units were over-generated in long-form samples. We filtered out such samples by thresholding with a ratio of the sequence length of the discrete units over the number of corresponding source speech frames. We used a threshold of 0.7 for the multidomain En→Es corpus while using ∞ for the rest. We used the same number of samples for all direct S2ST models for a fair comparison.

heading: F Pre-training
text: In Table 7, we list all the pre-trained self-supervised models and TTS models used in §4.
wav2vec2.0 We use 24-layer Conformer wav2vec2.0 (Baevski et al., 2020) models trained on Libri-Light (Kahn et al., 2020) for En and VoxPopuli for Es, respectively.
w2v-BERT Same as Jia et al. (2019a), we pretrain the w2v-BERT (Chung et al., 2021) on approximately 430k hours of unlabeled speech data in 51 languages spanning from VoxPopuli, Common Voice, MLS, BABEL (Harper et al.; Gales et al., 2014), and VoxLingua107 (Valk and Alumäe, 2021). The w2v-BERT was composed of 24 Conformer layers with 0.6 billions of parameters.
Text-based mBART (t-mBART) We train a tmBART model with En and Es unlabeled text on CC100 (Conneau et al., 2020). We use of a 65k unigram subword unit for the vocabulary. For multilingual experiments on CVSS-C, we use mBART50 (Tang et al., 2020) with multilingual fine-tuning to En. The vocabulary size is a 250k subword unit.
Unit-based mBART (u-mBART) We use a umBART model trained with En and Es unlabeled speech on VoxPopuli. The unit vocabulary is the same as that of the mHuBERT model.

heading: G Architecture details
text: Let dmodel be a model dimension of Transformer, dff be an inner dimension of the FFN layers, and Nhead be the number of attention heads.
Speech encoder We used a 16-layer Conformer encoder stacked on 2-dimensional convolution blocks when training models from scratch. The convolution blocks reduced the input sequence length by a factor of 4. We set (dmodel, dff ,Nhead) to (256,2048,4). We set the kernel size of the depthwise convolution in the convolution module of each Conformer block to 31. When pre-training the encoder with wav2vec2.0 and w2v-BERT, we used a 24-layer Conformer encoder and stacked a one-layer length adaptor (Li et al., 2021) on it. Because an output frame of wav2vec2.0 and w2vBERT corresponds to 20ms and the length adaptor halved the sequence length, the frame rate of every final encoder output corresponds to 40ms in both
cases. In this case, we set (dmodel, dff ,Nhead) to (1024,4096,16).
S2SpecT We used a six-layer Transformer spectrogram decoder. We set (dmodel, dff ,Nhead) to (512,2048,8). When pre-training the speech encoder with wav2vec2.0 or w2v-BERT, we doubled these three values. We set the pre-net dimension and reduction factor of the spectrogram decoder to 32 and 3, respectively.
S2SpecT2 Let Nt2s be the depth of the T2S encoder. We set (N1st,N2nd,Nt2s) to (4,6,2) on Fisher and CVSS-C. On the multi-domain corpus, we set (N1st,N2nd,Nt2s) to (12,6,2) when pretraining the first-pass decoder with t-mBART. Otherwise, we set (N1st,N2nd,Nt2s) to (6,6,2). We used the same dmodel, dff , and Nhead as S2SpecT in all the settings.
S2UT We used a six-layer Transformer unit decoder. When training models from scratch on Fisher, we set (dmodel, dff ,Nhead) to (256,2048,4). We set (dmodel, dff ,Nhead) to (512,2048,8) on CVSS-C. When pre-training the speech encoder with wav2vec2.0 or w2v-BERT, we set (dmodel, dff ,Nhead) to (1024,4096,16).
UnitY We used the same first-pass decoder as S2SpecT2 in all the settings. We set (N2nd,Nt2u) to (2,2). We used the same dmodel, dff , and Nhead as the S2UT model in all the settings.
S2TT We used a six-layer Transformer decoder. When initializing it with t-mBART, we set the depth to 12.
ASR We used the same architecture as the S2TT model except for the vocabulary in all the settings.

heading: H Training details
text: We optimized all models with the mixed precision training using 32GB V100 GPUs (Micikevicius et al., 2018). When fine-tuning the speech encoder from wav2vec2.0 and w2v-BERT, we updated all parameters in the speech encoder. For multilingual training with speech encoder pre-training with w2vBERT on CVSS-C, we over-sampled training data of low-resource directions with an inverse temperature of 0.6, following (Arivazhagan et al., 2019). We list the training hyperparameters in Table 8. The training of A*, B*, and C* models converged within approximately 1, 3, and 5 days, respectively.

heading: I Additional experimental results
text: In this section, we present additional experimental results in §4.
FLOPs In Figure 5, we show the results of FLOPs measured with a subset of the multi-domain Es→En dev set, as discussed in §4.3. UnitY achieved 1.65× and 3.19× FLOPs reduction over S2SpecT2 and S2UT models, respectively.
Fisher Es→En The results on Fisher are shown in Table 9. We report average scores over three runs with different random seeds. Among our four
direct systems trained from scratch (A11, A15, A18, A20), UnitY (A20) achieved the best ASRBLEU. Our S2UT (A18) and S2SpecT2 (A15) outperformed the previous studies (A13, A17) by a large margin.9 Because S2SpecT2 outperformed S2UT, the two-pass decoding was the main factor of the improvements although it was complementary to targeting discrete units. Moreover, the two-pass direct models (A15, A20) outperformed a cascaded system (A6).
Next, we pre-trained the speech encoder with wav2vec2.0 (A12, A16, A19, A21).10 We confirmed that all the models benefited from the pretraining, but the gain was small for S2SpecT. Unlike when training the models from scratch, S2SpecT2 gained the most and achieved the best test ASR-BLEU, 58.3. To the best of our knowledge, this is the new state-of-the-art S2ST result
9A15 predicts phonemes while A16 predicts subwords in the first pass.
10We did not pre-train the text decoder with t-mBART because it was not helpful on this corpus. This is because Fisher is a conversational domain, which is very different from text data used for t-mBART pre-training. We could make the text decoder pre-training effective by including conversational data during t-mBART pre-training, which we leave future work.
on this corpus. However, UnitY has an advantage of decoding efficiency over S2SpecT2 as discussed in §4.3. All direct models (A16, A19, A21) except for S2SpecT outperformed the corresponding cascaded system (A7).
CVSS-C We show the full results of each language direction on CVSS-C in Table 10.
Pre-training first-pass decoder We explored a better pre-training strategy for the first-pass text decoder in UnitY. We investigated pre-training it with an MT model trained with bitext data from scratch (Supervised MT1, Supervised MT2). Supervised MT1 used CCMatrix (Schwenk et al., 2021) while Supervised MT2 is the MT model in the
cascaded system11. Moreover, we fine-tuned the t-mBART model to the MT task in an unsupervised MT way via online back translation (Liu et al., 2020) on CC100 (unsupervised MT). Furthermore, we studied initializing the speech encoder and the text decoder with a separate direct S2TT model. The S2TT model was fine-tuned from wav2vec2.0 and t-mBART models on the same corpus. After the initialization, we fine-tuned the whole parameters of UnitY except FFN layers in the first-pass text decoder (S2TT).
The results in Table 11 showed that pre-training the first-pass decoder with the vanilla t-mBART (F2) or the unsupervised MT model (F3) was the most effective. Pre-training with supervised MT models (F4, F5) did not improve performance, even for the first pass. This is consistent with a finding in Jia et al. (2022a) although they pre-train the first-pass phoneme decoder of Translatotron2 with a phoneme-based supervised MT model. Therefore, leveraging a separate MT system is effective for generating weak supervisions (Popuri et al., 2022) rather than parameter initialization. Pretraining a part of UnitY with an independent S2TT model (F7) was not helpful either. Surprisingly, the BLEU score from the text decoder in UnitY was
11We used OpenSubtitle2018, UNCorpus, EUBookshop v2, Europarl v10, Wikipedia v1.0, and TED2020 v1 for training.
better than that of F7. Therefore, training signals from the unit decoder never affect the text decoder.
Ablation study In Table 12, we show full results of the ablation study presented in §5.1. An auxiliary CTC objective for the unit decoder, as used for the S2UT model (Lee et al., 2022a), was not helpful for UnitY (D10). This was because the introduction of the first-pass decoder already eased for the second-pass decoder to learn monotonic alignments.
Output unit for first-pass decoder In Table 13, we show full results of the comparison of the output units for the first-pass decoder in two-pass direct S2ST models presented in §5.2. The results showed that the subword unit was the best for UnitY regardless of pre-training the speech encoder with wav2vec2.0. In contrast, in the case of S2SpecT2, the best output unit differed according to whether we pre-trained the speech encoder or not. The phoneme unit was best when training the model from scratch (E1) while the subword unit was best when pre-training the encoder (E3’). However, predicting subwords in the first pass led to the best BLEU score for the text output in all the settings.
ASR-chrF Following a finding that ASR-chrF is a more robust evaluation metric than ASR-BLEU in Salesky et al. (2021), we also calculated ASRchrF on Fisher, CVSS-C, and multi-domain corpora in Table 14, Table 15, and Table 16, respectively. Overall, we confirmed the similar trends to ASRBLEU.
I.1 Human evaluation
Finally, we conducted an audio-only human evaluation to assess the translation quality while removing the necessity of ASR systems. We adopted crosslingual semantic textual similarity (XSTS) (Licht et al., 2022) and percent acceptable translations.
Mean translation score We used XSTS, which emphasizes adequacy rather than fluency, as the most appropriate human evaluation protocol. Annotators judged the semantic similarity between the source and the translated sentence. As a result, whether a translation conveys the original meaning is more important than whether it has perfect syntax, wording, and grammar. Annotators assigned each item a score from one to five. A score of no less than three means the meaning is at least “mostly equivalent.” We treat a translation that re-
ceived a score of no less than three as having “acceptable” quality. Annotators need to be bilingual, as they compare the source and translated sentences directly. Since XSTS is an audio-only evaluation metric, it also considers the audio quality.
For each system, we computed the average XSTS score across items. We set a target of over four average XSTS for systems where we expect or desire high-quality translations. We set a target of over three average XSTS for systems where we expect a medium level of quality.
Percent acceptable translations For each system, we also computed the percentage of items that received an XSTS score of three or above. We refer to this as the percent acceptable translations. This metric helps us understand what percentage of translations produced by the system can preserve meaning adequately and what percentage has very low and unacceptable quality. This metric tends to be more stable and less sensitive to annotator agreement than the average XSTS score.
Evaluation setting We used the mTEDx test set (989 samples) and generated the target audio from the S2ST systems. Moreover, we randomly sampled 495 samples and generated the target audio from the reference translation followed by TTS. The reference translations serve as a reference point and a ceiling against which to compare our systems. Three bilingual annotators evaluated each item and assigned it a score from one to five. The median score was taken per item.
Results The results are presented in Figure 6.12 We confirmed that UnitY consistently outperformed the cascaded and S2UT models in both metrics.
12The models used here are early versions and slightly different from the models in Table 2.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 8
3 A2. Did you discuss any potential risks of your work? Section 8
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3.1 and Appendix D
3 B1. Did you cite the creators of artifacts you used? Section 3.1 and Appendix D
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We used public corpora only. All the corpora are open-sourced except Fisher Spanish.The license is described in the corpus-related papers we cited.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? We didn’t modify any contents in the public corpora we used.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? They are described in the corpus-related papers we cited.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section 3.1 and Appendix D
7 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. We followed the same setting as previous works.
C 3 Did you run computational experiments? Section 4.3
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 5.3
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 3, Appendix E, F, G, H
7 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? We reported average scores over 3 runs for Fisher and scores with a single run for the rest large corpora. We didn’t run multiple same jobs for the latter because they are computationally expensive.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 3, Appendix E, F, G, H
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
7 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? We didn’t hire any human annotators in our experiments.
7 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? We didn’t hire any human annotators in our experiments.
7 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? We didn’t hire any human annotators in our experiments.
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? We didn’t hire any human annotators in our experiments.
7 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? We didn’t hire any human annotators in our experiments.

heading: Future Work
text: We could make the text decoder pre-training effective by including conversational data during t-mBART pre-training, which we leave future work. on this corpus. However, UnitY has an advantage of decoding efficiency over S2SpecT2 as discussed in §4. 3. All direct models (A16, A19, A21) except for S2SpecT outperformed the corresponding cascaded system (A7). CVSS-C We show the full results of each language direction on CVSS-C in Table 10. Pre-training first-pass decoder We explored a better pre-training strategy for the first-pass text decoder in UnitY. We investigated pre-training it with an MT model trained with bitext data from scratch (Supervised MT1, Supervised MT2). Supervised MT1 used CCMatrix (Schwenk et al. , 2021) while Supervised MT2 is the MT model in the
cascaded system11. Moreover, we fine-tuned the t-mBART model to the MT task in an unsupervised MT way via online back translation (Liu et al. , 2020) on CC100 (unsupervised MT). Furthermore, we studied initializing the speech encoder and the text decoder with a separate direct S2TT model. The S2TT model was fine-tuned from wav2vec2. 0 and t-mBART models on the same corpus. After the initialization, we fine-tuned the whole parameters of UnitY except FFN layers in the first-pass text decoder (S2TT). The results in Table 11 showed that pre-training the first-pass decoder with the vanilla t-mBART (F2) or the unsupervised MT model (F3) was the most effective. Pre-training with supervised MT models (F4, F5) did not improve performance, even for the first pass. This is consistent with a finding in Jia et al. (2022a) although they pre-train the first-pass phoneme decoder of Translatotron2 with a phoneme-based supervised MT model. Therefore, leveraging a separate MT system is effective for generating weak supervisions (Popuri et al. , 2022) rather than parameter initialization. Pretraining a part of UnitY with an independent S2TT model (F7) was not helpful either. Surprisingly, the BLEU score from the text decoder in UnitY was
11We used OpenSubtitle2018, UNCorpus, EUBookshop v2, Europarl v10, Wikipedia v1. 0, and TED2020 v1 for training. better than that of F7. Therefore, training signals from the unit decoder never affect the text decoder. Ablation study In Table 12, we show full results of the ablation study presented in §5. 1. An auxiliary CTC objective for the unit decoder, as used for the S2UT model (Lee et al. , 2022a), was not helpful for UnitY (D10). This was because the introduction of the first-pass decoder already eased for the second-pass decoder to learn monotonic alignments. Output unit for first-pass decoder In Table 13, we show full results of the comparison of the output units for the first-pass decoder in two-pass direct S2ST models presented in §5. 2. The results showed that the subword unit was the best for UnitY regardless of pre-training the speech encoder with wav2vec2. 0. In contrast, in the case of S2SpecT2, the best output unit differed according to whether we pre-trained the speech encoder or not. The phoneme unit was best when training the model from scratch (E1) while the subword unit was best when pre-training the encoder (E3’). However, predicting subwords in the first pass led to the best BLEU score for the text output in all the settings. ASR-chrF Following a finding that ASR-chrF is a more robust evaluation metric than ASR-BLEU in Salesky et al. (2021), we also calculated ASRchrF on Fisher, CVSS-C, and multi-domain corpora in Table 14, Table 15, and Table 16, respectively. Overall, we confirmed the similar trends to ASRBLEU. I. 1 Human evaluation
Finally, we conducted an audio-only human evaluation to assess the translation quality while removing the necessity of ASR systems. We adopted crosslingual semantic textual similarity (XSTS) (Licht et al. , 2022) and percent acceptable translations. Mean translation score We used XSTS, which emphasizes adequacy rather than fluency, as the most appropriate human evaluation protocol. Annotators judged the semantic similarity between the source and the translated sentence. As a result, whether a translation conveys the original meaning is more important than whether it has perfect syntax, wording, and grammar. Annotators assigned each item a score from one to five. A score of no less than three means the meaning is at least “mostly equivalent. ” We treat a translation that re-
ceived a score of no less than three as having “acceptable” quality. Annotators need to be bilingual, as they compare the source and translated sentences directly. Since XSTS is an audio-only evaluation metric, it also considers the audio quality. For each system, we computed the average XSTS score across items. We set a target of over four average XSTS for systems where we expect or desire high-quality translations. We set a target of over three average XSTS for systems where we expect a medium level of quality. Percent acceptable translations For each system, we also computed the percentage of items that received an XSTS score of three or above. We refer to this as the percent acceptable translations. This metric helps us understand what percentage of translations produced by the system can preserve meaning adequately and what percentage has very low and unacceptable quality. This metric tends to be more stable and less sensitive to annotator agreement than the average XSTS score. Evaluation setting We used the mTEDx test set (989 samples) and generated the target audio from the S2ST systems. Moreover, we randomly sampled 495 samples and generated the target audio from the reference translation followed by TTS. The reference translations serve as a reference point and a ceiling against which to compare our systems. Three bilingual annotators evaluated each item and assigned it a score from one to five. The median score was taken per item. Results The results are presented in Figure 6. 12 We confirmed that UnitY consistently outperformed the cascaded and S2UT models in both metrics. 12The models used here are early versions and slightly different from the models in Table 2. ACL 2023 Responsible NLP Checklist