Can We Achieve High-quality Direct Speech-to-Speech Translation without Parallel Speech Data? (2024)

Abstract: Two-pass direct speech-to-speech translation (S2ST) models have shown promising results which decompose S2ST into speech-to-text translation (S2TT) and text-to-speech (TTS), yet conduct end-to-end training by sharing the target text representation between S2TT and TTS models. However, the training of these models still requires large-scale parallel speech data comprising <source speech, target text, target speech> triplets, which is extremely challenging to collect. On the other hand, S2TT and TTS have accumulated a large amount of data and numerous pretrained models, which can be used to reduce the reliance on parallel speech data. To this end, we propose a composite S2ST model named ComSpeech, which connects pretrained S2TT and TTS models by introducing a vocabulary adaptor based on connectionist temporal classification (CTC). The vocabulary adaptor is employed to adapt the output text sequence of S2TT to the input text sequence of TTS, which are different due to the use of different vocabularies. In this way, ComSpeech can still be trained end-to-end and only needs a small amount of parallel speech data to finetune. We further propose a novel training method ComSpeech-ZS to eliminate the reliance on parallel speech data by aligning the text representation space of S2TT and TTS. Experimental results on the CVSS dataset show that when the parallel speech data is available, ComSpeech surpasses previous two-pass models like UnitY and Translatotron 2 in both translation quality and decoding speed. When there is no parallel speech data, ComSpeech-ZS lags behind ComSpeech by only 0.7 ASR-BLEU and outperforms the cascaded models.1

heading: 1 Introduction
text: Direct speech-to-speech translation (S2ST) refers to translating source language speech directly
*Corresponding author: Yang Feng. 1Project Page: https://ictnlp.github.io/ComSpeech-Site/
into target language speech using an end-to-end model (Jia et al., 2019). In contrast to traditional cascaded S2ST (Wahlster, 2000; Nakamura et al., 2006), which generates target speech through a pipeline of automatic speech recognition (ASR), machine translation (MT), and textto-speech (TTS), direct S2ST not only can reduce error accumulation along modules but is easier to deploy (Lee et al., 2022a), so that it has attracted considerable interest from researchers in recent years (Chen et al., 2022; Fang et al., 2023).
Speech data usually employs smaller semantic units than text data, where a long span of speech units corresponds to a word. As a result, S2ST has to carry out the conversion between longer sequences, which involves implicitly segmenting the speech units into dependent semantic groups and mapping between the source and target group sequences, making S2ST extremely challenging. To reduce the complexity of directly converting speech input to speech output, some two-pass models (Jia et al., 2022b; Inaguma et al., 2023) introduce intermediate target text representations and guide their learning by predicting the ground truth target text during training. In this way, the direct speech-to-speech conversion is decomposed into two passes of speech-to-text conversion and textto-speech conversion. The generation of target text helps organize the long speech units into words at the semantic level, thereby facilitating training. Despite the two-pass architecture, the models are still trained end-to-end by sequentially generating text and speech via multi-task learning. This modeling mechanism reduces learning complexity while retaining the benefits of end-to-end training, surpassing the performance of cascaded S2ST.
Despite the success of two-pass S2ST models, they still face a challenge: the training of these models still requires large-scale parallel speech data composed of <source speech, target text, target speech> triplets, which is difficult to collect.
7264
On the other hand, S2TT and TTS have accumulated a large amount of training data and numerous sophisticated models. Leveraging these existing resources would definitely reduce the cost of constructing an S2ST model. What fits perfectly is that both the output of S2TT and the input of TTS are target text sequences. If S2TT and TTS can share the representations of the text sequence, they can be connected to form a two-pass S2ST model. In this way, the existing resources of S2TT and TTS can be utilized and only a small amount of parallel speech data is needed to finetune the two-pass model. However, the S2TT and TTS models have their own vocabularies. For example, S2TT models commonly use subword vocabularies, whereas TTS models often employ phoneme or character vocabularies. This distinction implies that the target text will be segmented into different vocabulary sequences by S2TT and TTS models. Consequently, S2TT and TTS cannot share the text representations and hence cannot be directly connected.
To solve the above problems, we propose a composite speech-to-speech translation model, named ComSpeech, which connects S2TT and TTS models with a vocabulary adaptor based on connectionist temporal classification (CTC; Graves et al., 2006). The vocabulary adaptor models the conversion from the output vocabulary sequence of S2TT to the input vocabulary sequence of TTS as a non-autoregressive sequence-to-sequence generation problem. With the help of CTC, it can explore all the possible generated sequences that can be organized into the input sequence of TTS. Finally, with only a small set of parallel speech data, ComSpeech can be trained end-to-end with gradient backpropagation through the model, thereby finetuning existing S2TT and TTS models into a twopass S2ST model. Furthermore, we propose a zeroshot training strategy ComSpeech-ZS, with which ComSpeech can be trained solely based on S2TT and TTS training data. It aims to align the text representation space of S2TT and TTS models via contrastive learning, enabling seamless connection between S2TT and TTS models without the need for finetuning with parallel speech data. Experimental results on the CVSS dataset demonstrate that ComSpeech surpasses previous two-pass models like UnitY and Translatotron 2 in both translation quality and decoding speed. In the zero-shot learning scenario, the translation quality of ComSpeech-ZS is only 0.7 ASR-BLEU lower than ComSpeech and surpasses cascaded systems.

heading: 2 Background: Two-pass S2ST
text: The S2ST dataset usually contains three-way parallel data DS2ST = {(S, Y, T )}, where S denotes the source speech, Y is the target text, and T is the target speech. Due to the challenges of directly generating target speech, recent developments have introduced two-pass S2ST models, which enhance translation quality by incorporating the target text into the generation process. Next, we will introduce the model structure of UnitY (Inaguma et al., 2023), as it is one of the most representative twopass S2ST model. Specifically, it consists of four sub-modules: the source speech encoder Fenc, the target text decoder Fdec, the text-to-speech encoder Genc, and the target speech decoder Gdec. Firstly, Fenc and Fdec perform the S2TT task, trained with the cross-entropy loss:
LS2TT = − |Y |∑
i=1
logP (yi|Fdec(Fenc(S), Y<i)).
(1) Using H = Fdec(Fenc(S), Y ) to denote the target text representation from the last layer of Fdec, Genc takes H as its input, and Gdec then predicts the target speech. UnitY uses discrete units (Lee et al., 2022a) of the target speech as the prediction target and is trained using cross-entropy loss:
LS2ST = − |T |∑
i=1
logP (ti|Gdec(Genc(H)), T<i).
(2) In summary, UnitY is trained on the parallel speech dataset DS2ST with the following objective:
LUnitY = LS2TT + γ · LS2ST, (3)
where γ denotes the weight of the second term. However, current two-pass S2ST models like UnitY still have two main issues: (1) First, they implicitly assumes that Fdec and Genc share the same target text vocabulary, making it challenging to simultaneously employ an existing S2TT model F as Fenc and Fdec, while utilizing an existing TTS model G as Genc and Gdec. (2) Second, their training requires parallel speech data, which is extremely difficult to collect. To address these issues, we first propose a composite S2ST model built upon existing S2TT and TTS models (Section 3). Next, to eliminate the reliance on parallel speech data, we introduce a novel training method that uses only S2TT and TTS data to achieve zeroshot S2ST (Section 4).

heading: 3 Proposed Model: ComSpeech
text: In this section, we introduce a more general twopass S2ST model architecture: Composite Speechto-Speech Translation (ComSpeech) model. ComSpeech comprises three modules: F , A, and G, where F denotes arbitrary S2TT model, G denotes arbitrary TTS model for the target language, and A represents the vocabulary adaptor inserted between F and G. The inclusion of the vocabulary adaptor A is necessitated by the fact that S2TT and TTS models typically employ different vocabularies for the target language. It can facilitate the conversion of target text representation sequences between different vocabularies. Figure 1 illustrates the model architecture of ComSpeech.
Formally, we use S to denote the filter bank features of the source speech, and use T to denote the target speech representation, which can be either mel-spectrograms or discrete units, both of which can be synthesized into the waveform using a vocoder. We use W and V to denote the vocabularies of the S2TT model and the TTS model, respectively. The tokenized sequences of the target text Y under these two vocabularies are represented as Y W = (yW1 , ..., y W N ) and Y V = (yV1 , ..., y V M ), respectively. Next, we will introduce the detailed implementations of each module in ComSpeech.
S2TT Model We use the s2t_conformer model in fairseq S2T2 (Wang et al., 2020) as the S2TT model F . Specifically, it comprises a Conformerbased speech encoder (Gulati et al., 2020) and a Transformer-based text decoder (Vaswani et al., 2017). The encoder takes S as its input. The decoder adopts a subword vocabulary W for the target language. Formally, the decoder output hidden states of the S2TT model can be represented as HW = (hW1 , ...,h W N ), where h W i = F(S, Y W<i ). The model is trained with the cross-entropy loss:
LS2TT = − N∑
i=1
logP (yWi |F(S, Y W<i )). (4)
TTS Model We use FastSpeech 2 (Ren et al., 2021) as the TTS model G, consisting of an encoder, a variance adapter, and a decoder. It adopts a phoneme vocabulary V for the target language. For the independent TTS model, the encoder takes the target text embedding EMB(Y V) as input, where
2https://github.com/facebookresearch/fairseq/ tree/main/examples/speech_to_text
EMB(·) denotes the phoneme embedding layer. The variance adaptor and decoder predict variance information and target mel-spectrograms T , respectively. The training objective of FastSpeech 2 can be written as follows:
LTTS = LL1 + Ldur + Lpitch + Lenergy, (5)
where LL1 calculates the L1 distance between the predicted and ground truth mel-spectrograms, Ldur, Lpitch, and Lenergy denote the mean square error (MSE) loss between ground truth and predictions for duration, pitch, and energy, respectively.
It can be observed that the output of F and the input of G are both target text sequences. If the target text representations could be shared between them, they could be combined into a two-pass S2ST model. However, since F and G use different vocabularies, the representation HW output by F cannot be directly understood by the TTS encoder. Therefore, we introduce a vocabulary adaptor to convert the representation sequence corresponding to Y W into one that corresponding to Y V.
Vocabulary Adaptor To achieve the conversion of representation sequences under different vocabularies, we essentially need to solve a sequence-tosequence generation problem from Y W to Y V. To ensure both quality and speed, we adopt a nonautoregressive structure based on connectionist temporal classification (CTC; Graves et al., 2006). Specifically, we first upsample each hidden state in HW by a factor of λ, resulting in an upsampled hidden state sequence ĤW = (ĥW1 , ..., ĥ W λ·N ), where ĥWi = h W ⌊i/λ⌋. Next, we add positional encoding to ĤW and then use an L-layer Transformer encoder E for encoding, obtaining the output hidden state
sequence O = (o1, . . . ,oλ·N ):
O = E(ĤW + POS(ĤW)), (6)
where POS(·) denotes the sinusoid positional encoding (Vaswani et al., 2017). To align O with the target text Y V, CTC extends the output space V with a special blank token ϵ, and maps each element in O into the output space:
P (ai|O) = softmax(Woi+b)[ai] ∀ai ∈ V∪{ϵ}, (7) where W ∈ R(|V|+1)×d and b ∈ R|V|+1 are the weights and biases of the linear layer, and A = (a1, ..., aλ·N ) is referred to as the alignment. CTC introduces a collapsing function β(A) that initially merges all consecutively repeated tokens in A and then removes all blank tokens ϵ. For example: β(aabϵϵbbc) = abbc. During training, CTC marginalizes out all possible alignments as follows:
LCTC = − logP (Y V|O) = − log ∑
A∈β−1(Y V) P (A|O)
= − log ∑
A∈β−1(Y V)
λ·N∏
i=1
P (ai|O),
(8)
where β−1(Y V) denotes all possible alignments of length λ ·N that can be collapsed to Y V.
Through CTC, we can learn the alignment between the output representation sequence O and the target text sequence Y V. However, the length of O is greater than or equal to Y V (λ ·N ≥ M ) due to the presence of repeated and blank tokens in the alignment A. During training, it is essential
to obtain the representation sequence exactly corresponding to the ground truth target Y V, which enables joint training with the subsequent TTS model. Therefore, we compress O to length M following these steps: (1) Firstly, we find the most probable alignment A∗ = argmaxA P (A|O, Y V) via Viterbi algorithm (Viterbi, 1967), which is often referred to as forced alignment. (2) Secondly, each continuous repetition of non-blank tokens in A∗ is divided into a segment. In this way, each token yVi in Y V corresponds to a segment [a∗li , ..., a ∗ ri ], where a∗j = y V i ,∀li ≤ j ≤ ri. (3) Finally, the representations within each segment are merged into a vector according to the prediction confidence (Liu et al., 2020; Gaido et al., 2021):
hVi =
ri∑
j=li
pj · oj , (9)
pj = exp(P (a∗j |O))∑ri
k=li exp(P (a∗k|O))
. (10)
The representations corresponding to blank tokens are discarded. Finally, we obtain the representation sequence HV = (hV1 , ...,h V M ) corresponding to Y V, which is fed into the TTS encoder for subsequent speech synthesis. During inference, we select the most probable path A∗ = (a∗1, ..., a ∗ λ·N ) with argmax decoding: a∗i = argmaxai P (ai|O), and then merge representations in the same manner as during training.
Training ComSpeech can be trained using S2TT, TTS, and S2ST data. Specifically, we can utilize S2TT data to pretrain the S2TT model F and TTS data to pretrain the TTS model G. Subsequently, the entire model is finetuned using S2ST data. The
training objective during finetuning is as follows:
LComSpeech = LS2TT + LCTC + LTTS. (11)
Moreover, as the vocabulary adaptor can establish connections between arbitrary S2TT and TTS models, theoretically, we can leverage any pretrained S2TT and TTS models for S2ST finetuning.

heading: 4 ComSpeech-ZS: Training ComSpeech without Parallel Speech Data
text: As mentioned earlier, collecting S2ST data is extremely challenging. In this section, we explore the possibility of training ComSpeech exclusively on S2TT and TTS data, achieving zero-shot S2ST without the need for parallel speech data. The idea is to train distinct modules using S2TT and TTS data concurrently, while achieving representation alignment in the output space of the TTS encoder. This enables the speech synthesis capabilities learned from the TTS data to generalize to S2ST during inference. Figure 2 illustrates the process of training and inference.
Specifically, we use DS2TT = {(S, Y )} and DTTS = {(Y, T )} to denote the S2TT and TTS datasets, respectively, and train the model concurrently using both datasets. For TTS data batches, we train the TTS model G with LTTS, as illustrated in Figure 2(a). For S2TT data batches, firstly, we train the S2TT model F and vocabulary adaptor A with LS2TT and LCTC. Secondly, since the TTS encoder receives the target text embedding EMB(Y V) during TTS training but vocabulary adaptor’s output HV during inference, we aim for the output representations of the TTS encoder with these two inputs to be identical, thereby achieving zero-shot generalization. Therefore, we introduce a representation alignment loss after the TTS encoder during S2TT training, as illustrated in Figure 2(b).
Representation Alignment Formally, we use CH = (cH1 , ..., c H M ) and C E = (cE1 , ..., c E M ) to denote the output representations of TTS encoder with these two types of input:
CH = Genc(HV), CE = Genc(EMB(Y V)). (12)
Firstly, we align representations with the MSE loss:
LMSE = M∑
i=1
∥cHi − cEi ∥22. (13)
Additionally, we incorporate a contrastive learning objective to align representations. This objective
maximizes the similarity between representations from the same input pairs and minimizes the similarity between representations from different input pairs. The contrastive loss is formulated as:
LCTR =− 1
2
M∑
i=1
log exp(s(cHi , c E i )/τ)∑M
j=1 exp(s(c H i , c E j )/τ)
− 1 2
M∑
i=1
log exp(s(cHi , c E i )/τ)∑M
j=1 exp(s(c H j , c E i )/τ)
,
(14) where we use a similarity function based on L1 distance: s(x,y) = −∥x − y∥1, and τ denotes the temperature hyperparameter. The final training objective for representation alignment is as follows:
LAlign = LMSE + LCTR. (15)
Two-stage Finetuning In the zero-shot learning scenario, we adopt a two-stage finetuning strategy to facilitate the learning process. In the first stage, we only utilize the S2TT dataset DS2TT to train the S2TT Model F and vocabulary adaptor A through the objectives LS2TT and LCTC. In the second stage, we conduct training using both the S2TT and TTS datasets, with the following training objectives:
LComSpeech-ZS = EDTTS [LTTS] + EDS2TT [ LS2TT + LCTC + LAlign
] . (16)
Throughout the entire training process, the S2ST data is not used. During inference, we can achieve zero-shot S2ST as illustrated in Figure 2(c).

heading: 5 Experiments

heading: 5.1 Datasets
text: The S2ST, S2TT, and TTS datasets utilized in our experiments are all sourced from the CVSS dataset (Jia et al., 2022c). CVSS is a largescale S2ST dataset containing <source speech, target text, target speech> triples across 21 source languages to English. We conduct experiments on three language pairs: French→English (Fr→En), German→English (De→En), and Spanish→English (Es→En). Our experiments consist of two scenarios: supervised learning and zeroshot learning scenarios. The former involves training using S2ST data, while the latter involves training using only S2TT and TTS data.
S2ST Dataset In the supervised learning scenario, we utilize triplet data from the CVSS Fr/De/Es→En datasets to train the model.
S2TT Dataset In the zero-shot learning scenario, we employ <source speech, target text> pairs from the CVSS Fr/De/Es→En datasets as the S2TT data, with the target speech being discarded.
TTS Dataset In the zero-shot learning scenario, we combine all <target text, target speech> pairs from CVSS X→En (X/∈{Fr, De, Es}) datasets as the TTS data. We exclude the target speech-text pairs from CVSS Fr/De/Es→En datasets, ensuring that there is no overlap in the target text between S2TT and TTS datasets. This guarantees that in the zero-shot learning scenario, the model has not implicitly utilized parallel <source speech, target speech> pairs for training. See more details about data statistics and processing in Appendix A.

heading: 5.2 Experimental Setup
text: Model Configuration The S2TT model comprises 12 Conformer encoder layers and 4 Transformer decoder layers. The TTS model follows the standard configuration of FastSpeech 2. The vocabulary adaptor contains 4 Transformer encoder layers and the upsample factor λ is set to 5. The detailed hyperparameters can be found in Appendix C. The dropout is set to 0.3. The label smoothing of the S2TT model is set to 0.1. The HiFi-GAN (Kong et al., 2020) vocoder pretrained on the VCTK dataset (Veaux et al., 2017) is used to generate waveform from the mel-spectrogram.
Training During the training process, the batch size for S2TT and S2ST data is set to 320k source
audio frames, while the batch size for TTS data is set to 512. In the supervised learning scenario, the model is first pretrained on S2TT and TTS data3, followed by finetuning using S2ST data. In the zero-shot learning scenario, a two-stage finetuning approach is employed as stated in Section 4. During each training stage, the learning rate warms up to 1e-3 within 4k steps, and the training is halted if the validation loss does not decrease for 10 consecutive validations. We use Adam optimizer (Kingma and Ba, 2015) in all training stages. The temperature τ in contrastive learning is set to 0.1. All models are trained on 4 RTX 3090 GPUs.
Evaluation We average the best 5 checkpoints based on validation loss for evaluation, and employ two evaluation metrics: ASR-BLEU and BLASER 2.0 (Communication et al., 2023). For ASR-BLEU, we first transcribes the translated speech into text using a pretrained ASR model4, and then calculates the BLEU score (Papineni et al., 2002) and the statistical significance of translation results using the SacreBLEU toolkit5 (Post, 2018). For BLASER 2.0, we use the blaser-2.0-ref model6 to evaluate the cross-lingual semantic similarity. The de-
3It should be noted that in the supervised learning scenario, the TTS pretraining data are <target text, target speech> pairs sourced from S2ST data, distinct from the TTS data utilized in the zero-shot learning scenario.
4https://dl.fbaipublicfiles.com/fairseq/ wav2vec/wav2vec_vox_960h_pl.pt
5https://github.com/mjpost/sacrebleu 6https://huggingface.co/facebook/blaser-2.
0-ref
Model BLASER 2.0Fr→En De→En Es→En Supervised Learning
Translatotron 2 (Jia et al., 2022b) 3.1801 2.9034 3.2592 UnitY (Inaguma et al., 2023) 3.1749 2.8278 3.2310 ComSpeech w/ pretrain 3.1890 2.9281 3.2785
Zero-shot Learning
coding speed is measured on the CVSS-C Fr→En test set with a batch size of 1.
Baseline Systems We primarily compare ComSpeech with two state-of-the-art two-pass S2ST models: Translatotron 2 (Jia et al., 2022b) and UnitY (Inaguma et al., 2023). Their S2TT component is identical to the S2TT part of ComSpeech. Upon the S2TT model, a 2-layer Transformer encoder is employed to encode the S2TT decoder hidden states. Finally, UnitY utilizes a 2-layer decoder to generate the target units, while Translatotron 2 employs a 6-layer decoder to generate the target mel-spectrograms. Besides, we include the results of the single-pass model S2UT (Lee et al., 2022a) and the non-autoregressive two-pass model DASpeech (Fang et al., 2023) for comparison. We implement all above models with the open-source fairseq7 (Ott et al., 2019) library.
In the zero-shot learning scenario, we compare our proposed ComSpeech-ZS with a cascaded system: S2TT + G2P + TTS. Here, the S2TT and TTS models correspond to those used for initializing the ComSpeech-ZS model. The G2P is a model that converts English graphemes to phonemes. We achieve this conversion using the g2p_en8 library.
7https://github.com/facebookresearch/fairseq 8https://github.com/Kyubyong/g2p

heading: 5.3 Main Results
text: Results in the Supervised Learning Scenario Table 1 presents the ASR-BLEU scores on the CVSS test set. In the supervised learning scenario, it can be observed that: (1) ComSpeech outperforms Translatotron 2 and UnitY in translation quality across all three language pairs (A1-A2 vs. A5-A6). Since the S2TT parts of these three models are the same, the performance improvement in S2ST is mainly attributed to the adoption of a more powerful TTS model for speech synthesis in ComSpeech. We also report the performance of S2TT and S2ST, along with their gap in Table 2. We observe that ComSpeech narrows the performance gap between S2TT and S2ST compared to previous models, further demonstrating the importance of incorporating a more powerful TTS for speech synthesis. (2) Benefiting from the parallel decoding capability of FastSpeech 2, ComSpeech achieves a 3.40× decoding speedup compared with Translatotron 2. (3) Compared to S2UT, ComSpeech shows significant improvements in translation quality (A3 vs. A5), demonstrating the effectiveness of two-pass modeling in S2ST. (4) Compared to DASpeech, which also employs FastSpeech 2 as the TTS module, ComSpeech shows a 3.1 ASRBLEU improvement in translation quality (A4 vs. A6). We believe the main reason is that DASpeech uses a phoneme vocabulary for the S2TT model to be compatible with FastSpeech 2’s vocabulary, which may lead to a decrease in translation quality. Despite ComSpeech’s slower decoding speed compared to DASpeech, we consider this is not the focus of our work. Theoretically, our proposed vocabulary adaptor can be also used to connect a more powerful non-autoregressive S2TT model with a TTS model. We leave this for future research.
Results in the Zero-shot Learning Scenario In the zero-shot learning scenario, we find that: (1) Despite not using any S2ST data, the translation
quality of ComSpeech-ZS is comparable to that of ComSpeech in the supervised learning scenario, with only a 0.7 ASR-BLEU difference (A6 vs. B2). (2) ComSpeech-ZS also surpasses the performance of Translatotron 2, UnitY, S2UT, and DASpeech (A1-A4 vs. B2), further demonstrating the effectiveness of our proposed model architecture and training approach. (3) The translation quality of ComSpeech-ZS surpasses that of the cascaded system, possibly due to avoiding error accumulation. Meanwhile, ComSpeech-ZS is easier to deploy compared to the cascaded system, requiring only the deployment of a single model without the need to store intermediate results.
Results of BLASER 2.0 Scores Table 3 shows the BLASER 2.0 scores on the CVSS test set. We observe that our ComSpeech and ComSpeech-ZS consistently outperform other baseline systems in most cases. Since BLASER 2.0 evaluates directly based on speech rather than the transcribed text, this further validates that the speech generated by our model not only exhibits accurate translation but also reliable speech quality.

heading: 5.4 Ablation Studies
text: Training Objectives In the zero-shot learning scenario, achieving representation alignment is crucial for the final translation quality. In Table 4, we explore different alignment training objectives. In addition to ASR-BLEU, we follow Wang and Isola (2020) to measure the alignment and uniformity of the representation space, defined as follows:
Alignment = E(cHi ,cEi ) [ ∥cHi − cEi ∥1 ] (17)
Uniformity = logE(cHi ,cEj ) [ e−∥c H i −cEj ∥1 ] (18)
From the results in Table 4, it can be observed that using only MSE loss results in a relatively low ASR-BLEU. Using only contrastive learning also leads to a performance drop of around 2 ASRBLEU, as the alignment score is high in this case.
Hence, both contrastive learning and MSE loss are indispensable. Specifically, our experiments reveal that using L1 distance as the similarity function yields better results than L2 distance and dot product. In this case, even though the uniformity score is higher, the alignment score is the lowest. We speculate that the alignment of representation space plays a more crucial role in the ultimate zero-shot transfer performance.
Pretraining and Finetuning Strategies ComSpeech supports S2TT and TTS pretraining. We explore the impact of pretraining in both supervised and zero-shot learning scenarios. As shown in Table 5, using only TTS pretraining often has a negative effect, especially in the zero-shot learning scenario. On the other hand, S2TT pretraining brings a significant performance improvement, and combining both S2TT and TTS pretraining results in additional performance gains. Overall, pretraining has a greater impact on performance in the zero-shot learning scenario. Additionally, our proposed two-stage fine-tuning strategy leads to a performance improvement of 0.45 ASR-BLEU in the zero-shot learning scenario.

heading: 5.5 Effects of the Size of S2TT and TTS Data
text: In this section, we explore the effects of the size of S2TT and TTS data on ComSpeech-ZS and the cascaded system (S2TT + G2P + TTS).
S2TT Data Size As shown in Figure 3, the size of S2TT data significantly impacts the performance of both the cascaded system and ComSpeech-ZS. With only 10 hours of S2TT data, both systems have poor translation quality. However, as the amount of S2TT data increases, ComSpeech-ZS consistently achieves higher translation quality than the cascaded system, demonstrating the advantages of end-to-end modeling.
TTS Data Size As shown in Figure 4, the performance of the cascaded system remains relatively
stable with the expansion of TTS data size, while ComSpeech-ZS gradually improves with the increase in TTS data. When TTS data is limited, ComSpeech-ZS performs worse than the cascaded system, and we speculate this may be due to the imbalance in the S2TT and TTS data scales during finetuning. As the TTS data exceeds 50 hours, ComSpeech-ZS outperforms the cascaded system, and the gap between them grows as the TTS data size increases. This indicates that ComSpeech-ZS has a higher upper limit than the cascaded system.

heading: 6 Related Work
text: Speech-to-Speech Translation S2ST extends S2TT (Fang et al., 2022; Fang and Feng, 2023a,b; Zhou et al., 2023) which further generates the target speech. Jia et al. (2019) first proposes direct S2ST with a sequence-to-sequence model. Zhang et al. (2021); Lee et al. (2022a,b) propose using the discrete representation of speech as the prediction target and achieve better performance. Huang et al. (2023); Zhu et al. (2023); Wu (2023); Fang et al. (2023, 2024) adopt non-autoregressive models or diffusion models to generate the target speech for faster decoding speed. To make training easier, Jia et al. (2022b); Inaguma et al. (2023) introduce twopass S2ST models that generate target text and target speech successively. To further enhance S2ST, researchers introduce techniques like pretraining (Wei et al., 2023; Zhang et al., 2023; Dong et al., 2024) and data augmentation (Popuri et al., 2022; Jia et al., 2022a; Dong et al., 2022; Nguyen et al., 2022; Communication et al., 2023) to alleviate the data scarcity. Zhang et al. (2024); Ma et al. (2024) achieve simultaneous speech-to-speech translation with multi-task learning and non-autoregressive
streaming Transformer, respectively. Our work extends the research on two-pass S2ST, introducing a more general model architecture and a novel training method that achieves zero-shot S2ST without parallel speech data.
Zero-shot Speech Translation Dinh (2021); Wang et al. (2022); Duquenne et al. (2022, 2023) explore cross-modal alignment between speech and text, achieving zero-shot S2TT using only ASR and MT data. For S2ST, Diwan et al. (2023); Nachmani et al. (2024) focus on achieving zero-shot S2ST using only monolingual speech data in both source and target languages. To the best of our knowledge, we are the first to study achieving zero-shot S2ST using only S2TT and TTS data.

heading: 7 Conclusion
text: In this paper, we first introduce a novel S2ST model architecture, named ComSpeech, which incorporates a CTC-based vocabulary adaptor capable of connecting arbitrary S2TT and TTS models to obtain an S2ST model. Furthermore, we propose a training method using only S2TT and TTS data. By aligning in the TTS encoder’s representation space using contrastive learning, we achieve zeroshot S2ST without relying on parallel speech data. Experimental results on the CVSS dataset demonstrate that ComSpeech surpasses previous S2ST models in both translation quality and decoding efficiency. Moreover, in the zero-shot learning scenario, our model achieves performance close to the supervised learning scenario and surpasses the cascaded system. In the future, we will explore building direct S2ST models based on more powerful S2TT and TTS models.
Limitations
Although our model achieves satisfactory performance in both supervised learning and zero-shot learning scenarios, there are still some limitations: (1) In cases where TTS data is limited, the performance of ComSpeech-ZS still lags behind the cascaded system. In the future, we will explore how to improve performance by balancing the S2TT and TTS training data. (2) Currently, our method cannot preserve the speaker characteristics of the source speech. We will explore this in the future.

heading: Acknowledgement
text: We thank all the anonymous reviewers for their insightful and valuable comments. This paper is supported by National Natural Science Foundation of China (Grant No.62376260).

heading: A Data Statistics and Processing
text: Data Statistics The statistical information for all datasets are presented in Table 6.
Data Processing For the source speech, we convert it to 16000Hz and compute 80-dimensional mel-filterbank features. For the target speech, we convert it to 22050Hz and transform the waveform into mel-spectrograms. Utterance-level and globallevel cepstral mean-variance normalization are applied to the source speech and target speech, respectively. For the target text, the S2TT model employs a subword vocabulary of size 6k which is learned on the target text of CVSS. The TTS model employs a phoneme vocabulary of size 70. We follow Ren et al. (2021) to extract the duration, pitch, and energy information of the target speech.

heading: B Effects of the Hyperparameters of Vocabulary Adaptor
text: The vocabulary adaptor has two important hyperparameters: the number of Transformer encoder layers L and the upsampling factor λ for the input sequence. We investigate the influence of these hyperparameters in the supervised learning scenario without S2TT and TTS pretraining. According to the performance on the CVSS Fr→En dev set as shown in Table 7, we choose L = 4 and λ = 5 in our expriments.

heading: C Hyperparameters
text: We list the hyperparameters of ComSpeech and other baseline models in Table 8.

heading: Future Work
text: In the future, we will explore building direct S2ST models based on more powerful S2TT and TTS models. Limitations
Although our model achieves satisfactory performance in both supervised learning and zero-shot learning scenarios, there are still some limitations: (1) In cases where TTS data is limited, the performance of ComSpeech-ZS still lags behind the cascaded system. In the future, we will explore how to improve performance by balancing the S2TT and TTS training data. (2) Currently, our method cannot preserve the speaker characteristics of the source speech. We will explore this in the future