Textless Acoustic Model with Self-Supervised Distillation for Noise-Robust Expressive Speech-to-Speech Translation (2024)

Abstract: In this paper, we propose a textless acoustic model with a self-supervised distillation strategy for noise-robust expressive speech-tospeech translation (S2ST). Recently proposed expressive S2ST systems have achieved impressive expressivity preservation performances by cascading unit-to-speech (U2S) generator to the speech-to-unit translation model. However, these systems are vulnerable to the presence of noise in input speech, which is an assumption in real-world translation scenarios. To address this limitation, we propose a U2S generator that incorporates a distillation with no label (DINO) self-supervised training strategy into it’s pretraining process. Because the proposed method captures noise-agnostic expressivity representation, it can generate qualified speech even in noisy environment. Objective and subjective evaluation results verified that the proposed method significantly improved the performance of the expressive S2ST system in noisy environments while maintaining competitive performance in clean environments1.

heading: 1 Introduction
text: Speech-to-speech translation (S2ST), which translates speech in one language into speech in other language is indispensable technique for breaking down language barriers in naturalistic communication among international communities (Lavie et al., 1997; Nakamura et al., 2006; Wahlster, 2013). Recently, the direct speech-to-unit translation (S2UT) approach that translates source speech into a discretized semantic unit of target speech has been gaining a lot of attention (Lee et al., 2022a,b; Chen et al., 2023a; Inaguma et al., 2022; Seamless Communication et al., 2023). Thanks to their ability in modeling semantic units using a single network and the success of large-scale pre-training and data augmentation (Popuri et al., 2022), their latest models
1Audio samples are available at https://facebookresearch.github.io/seamless_ communication/demo/dino_pretssel/index.html
can achieve state-of-the-art translation quality (Inaguma et al., 2022; Seamless Communication et al., 2023).
On the other hand, it is also important to preserve source speech’s expressivity features2 such as vocal style, emotion, or tone during translation process to realize natural conversation with speech translator. However, it is non-trivial for a direct S2UT system to preserve the expressivity due to its design. Specifically, since the target discrete units contain linguistic information besides expressivity information (Seamless Communication et al., 2023), the expressivity of source speech is hard to be captured by S2UT model. As a results, the translated speech provides monotonic and robotic sound where the source speech’s expressivity doesn’t exist.
To achieve an expressivity-preserved S2ST system, several studies propose to cascade an additional unit-to-speech (U2S) generator, also known as a textless acoustic model, on top of the S2UT model. This approach proposes the U2S model by replacing the input of the acoustic model used in TTS systems from phonemes to discrete units. For instance, recently proposed PRETSSEL-based S2ST system (Seamless Communication, 2023) generates speech by receiving target language’s discrete units and source language speech’s expressivity embedding. The S2ST system with PRETSSEL can achieve high-quality cross-lingual expressivity transfer and content translation performance because it effectively disentangles the linguistic and paralinguistic information using discrete units and expressivity embedding, respectively.
However, this expressive S2ST framework exhibit issues when applied in the real-world translation scenarios, where the recording environment is noisy. Note that the expressivity encoder embedding space is trained to represent all information except linguistic one; that means a channel informa-
2In this work, we define expressivity as speech’s utterancelevel styles such as vocal style, emotion, or tone.
15524
tion such as background noise also exists in expressivity embedding along with vocal style, emotion, or tone information. Consequently, when the input speech is recorded in the noisy environment, the U2S model tries to transfer background noise to output speech, which critically affects both contents and expressivity preservation performances.
To address aforementioned problem, we propose a textless acoustic model that utilizes the selfdistillation with no label (DINO) strategy to its pretraining (Caron et al., 2021). Following the success of self-supervised speaker representation learning (Chen et al., 2023b), the proposed method introduces two teacher-student encoders and optimizes these encoders using a self-distillation training strategy. Specifically, the student encoder is updated to minimize its output probabilistic distance to the teacher encoder’s output. Then, the teacher encoder weights are iteratively updated by the exponential moving averaged (EMA) weights of the student encoder. In addition, random noise augmentation is applied to the input of both expressivity encoders to learn noise-agnostic expressivity representations.
We applied the proposed training strategy to the PRETSSEL U2S generator, which we refer to as DINO-PRETSSEL. Experimental results verified that the expressive S2ST system with DINOPRETSSEL outperformed conventional S2ST models in noisy recording environments while still achieving compatitive performance in clean recording environments. Specifically, the objective evaluation demonstrated that DINO-PRETSSEL achieved more noise-robust content and prosody preservation performance than other systems based on their ASR-BLEU (Jia et al., 2019) and AutoPCP (Seamless Communication, 2023) scores. Additionally, the subjective evaluation confirmed its superior performance in generating natural speech sound with robust vocal style preservation compared to conventional systems through the mean opinion score (MOS) and speaker-MOS (S-MOS) tests.

heading: 2 Related work

heading: 2.1 Expressive S2ST
text: There have been several studies proposing expressive S2ST model by cascading U2S generator to the S2UT model. For instance, PRETSSEL (Seamless Communication, 2023) and StyleS2ST (Song et al., 2023) adopted FastSpeech (FS)-style non-
autoregressive (NAR) U2S generators (Ren et al., 2019, 2021) for expressive S2ST. It also had been proved that the unit-based VoiceBox is also strong NAR U2S generator (Le et al., 2023; Seamless Communication, 2023). On the other hand, PolyVoice (Dong et al., 2023) proposed a similar cascaded S2ST by cascading two language models as S2UT and U2S generator components. Although previous works have shown impressive performance, they didn’t consider expressivity preservation in noisy environments. To our knowledge, our work is the first work to propose a noise-robust approach to expressive S2ST.

heading: 2.2 Noise-robust expressive TTS
text: As expressive TTS systems are becoming more natural and approaching human-level quality, there is a growing interest in incorporating noise-robustness into these systems in order to use it in the real world scenario. For instance, Hsu et al. (2018) proposed to disentangle noise information from noisy speech during training process of Gaussian mixture variational autoencoder. Swiatkowski et al. (2023) disentangled noise information by using external denoiser (Isik et al., 2020). During inference, they used only clean speech components for the clean speech generation.
Our work was mostly inspired by Pankov et al. (2023). This system achieved a robust voice cloning system by using VITS-based U2S generator with DINO strategy. The main difference of our work is that we focus on the cross-lingual S2ST application, whereas earlier work was applied to monolingual TTS application.

heading: 3 Background: Expressive S2ST with PRETSSEL
text: PRETSSEL is a unit-based textless acoustic model for expressive S2ST system (Seamless Communication, 2023). Specifically, PRETSSEL is pretrained to reconstruct 80-dimensional Mel-spectrogram with 10-ms interval of input speech from the deduplicated (or reduced) XLS-R units (Babu et al., 2022) with 10K K-means clustering and the same Mel-spectrograms.

heading: 3.1 Architecture of PRETSSEL
text: The PRETSSEL is composed of the expressivity encoder and the acoustic model. First, the expressivity encoder extracts a 512-dimensional expressivity embedding vector containing high-level
paralinguistic representations from the input Melspectrograms. Specifically, it adopts the variants of ECAPA-TDNN architecture (Desplanques et al., 2020) that replaces batch normalization (Ioffe and Szegedy, 2015) with layer normalization (Ba et al., 2016).
For a pair of expressivity embedding and discrete XLS-R units, the acoustic model generates Melspectrograms of output speech. The acoustic model architecture is based on FS2 architecture (Ren et al., 2021) consisting of a series of feed-forward Transformer (FFT) blocks, local prosody predictors, variance adaptors, and decoder FFT blocks. Major differences are (1) PRETSSEL uses FiLM conditioning layer (Perez et al., 2018; Oreshkin et al., 2018) to effectively utilize the expressivity embedding, (2) it uses the separately predicted unit duration from external S2UT model, and (3) it individually predicts the binary voiced/unvoiced (VUV) flag and the continuous F0 contour.

heading: 3.2 Pretraining
text: During pretraining, expressivity encoder and acoustic model are jointly trained to minimize three loss terms:
Lpretssel = Lmel + λl · Llocal + λf · Lfilm, (1) where Lmel, Llocal, and Lfilm denote Melspectrogram prediction loss, local prosody prediction loss, and L2 regularization loss at the FiLM layer (Oreshkin et al., 2018), respectively; λv and λv denote weight terms for Llocal and Lfilm, respectively. Specifically, Mel-spectrogram prediction loss is defined by summation of L1 and L2 losses for the predicted Mel-spectrograms before and after PostNet. In addition, local prosody prediction loss is defined by summation of L2 losses for the continuous F0 and energy contours in logarithm scale, and binary cross entropy (BCE) loss for VUV flag.

heading: 3.3 Application in Expressive S2ST
text: For the accurate prediction of translated units and their duration, the work of PRETSSEL proposes a Prosody UnitY2 S2UT model, which is an expressivity variant of the latest SeamlessM4T V2 model (Seamless Communication, 2023). This S2UT model predicts original XLS-R units at 20-ms interval conditioned by PRETSSEL’s expressivity embedding vector. After predicting original units, the reduced units and their duration are obtained by deduplication process. Then,
PRETSSEL’s acoustic model generates output Melspectrograms by taking reduced units, unit durations, expressivity embedding, and target language embedding. Finally, the HiFi-GAN vocoder (Kong et al., 2020) converts Mel-spectrograms into the speech waveform.
Since linguistic information dominates in the discrete units of speech (Seamless Communication et al., 2023), the expressivity embedding learns paralinguistic information such as prosody, vocal, or channel information as mentioned by SkerryRyan et al. (2018). This property enables to efficiently transfer the expressivity of source speech into translated speech, especially when the speech is recorded in clean environment. However, when it comes to the expressive S2ST in real-world situation that assumes noisy condition, the model tries to transfer noise components as well, which critically decrease the quality of translated speech signal.

heading: 4 DINO-PRETSSEL
text: In this section, we propose a DINO-PRETSSEL, where the DINO-based self-supervised training strategy is incorporated into PRETSSEL pretraining as illustrated in Figure 1. The details of this
process are described in the following sections.

heading: 4.1 Learning robust expressivity embedding with self-distillation
text: DINO-PRETSSEL utilizes two expressivity encoders termed teacher and student sharing the same ECAPA-TDNN architectures (Desplanques et al., 2020). Unlike the original PRETSSEL’s expressivity encoder, both teacher and student encoder contain additional projection head layers followed by softmax layer to measure probabilistic distance between student and teacher predictions. The projection heads are multi-layer perceptron (MLP) with linear output layer interleaved by GeLU activation (Hendrycks and Gimpel, 2016). Then, the student and teacher encoders are iteratively updated following the self-distillation framework.
Student training Let qt and qs be the Kdimensional outputs obtained by teacher and student encoders, respectively. Then, we obtain the probability distribution of student output ps by applying softmax function as follows:
pis = exp
( qis/τs ) ∑K
k=1 exp (qks/τs) , (2)
where i is the ith dimension of ps and τs is the temperature parameter that controls the sharpness of the output distribution. We apply similar formula to teacher output pt by using temperature τt. Then, we train the student encoder to match it’s output distribution to the teacher encoder output by minimizing cross-entropy (CE) loss between ps and pt. To freeze teacher encoder weights, we apply stop gradient operator to pt.
Following Chen et al. (2023b), we adopt multicrop strategy (Caron et al., 2020) to DINO loss. Specifically, we randomly sample L long segments and M short segment from single utterance to extract the expressivity embeddings containing longterm and short-term expressivity context. All the L + M segments are fed to student encoder, but only L long segments are fed to teacher encoder. Then, we compute DINO loss as the combination of CE losses between expressivity embeddings obtained from different segments as follows:
Ldino = 1 L · (L+M − 1) L∑
l=1
L+M∑
m=1 m ̸=l
CE ( plt,p m s ) ,
(3) where l and m denote the lth short segment and mth long segment, respectively.
Teacher training After updating the student encoder weights {θs} by one iteration, teacher network weights {θt} are assigned a running average of past student encoder weights by the EMA rule as follows:
θt ← λema · θs + (1− λema) · θs, (4)
where λema controls the extent to which the current student encoder’s weights affect the update of the teacher encoder’s weights. Following original DINO study (Caron et al., 2021) we gradually increases λ from 0.996 to 1.0 until the end of model training using cosine scheduler (Grill et al., 2020).
Avoiding model collapse. Because the teacher encoder is learned from the past weights of student encoder, a trivial solution that the encoders can learn is for the teacher to always present uniformly random values or deterministic values by outputting uniformly distributed or single dimensiondominated pt, respectively. DINO framework prevents this solution by applying centering and sharpening operations to teacher output distribution.
In detail, the centering operation normalizes logits of softmax distribution by the mean statistic c, which is updated by EMA rule as follows:
qt ← qt − c,
c← m · c+ (1−m) · 1 B
B∑
i=1
qt, (5)
where m and B denote the momentum factor for EMA update, and batch size, respectively. The centering operation prevents the situation that one dimension of teacher output dominates other dimensions by normalizing teacher logits to have similar dynamic range.
On the other hand, the sharpening operation makes teacher distribution sharper than the student distribution by setting smaller teacher temperature τt than student temperature τs. Thus, this operation prevents the situation that the teacher distribution to be uniform.
Noise augmentation. To obtain noise-agnostic expressivity representation, we apply random noise augmentation to the input of expressivity encoders. Then, we train DINO-PRETSSEL to predict clean one. At each iteration, we randomly select a noise signal from the noise database and add it to the input speech using a randomly determined signalto-noise ratio (SNR). On the other hand, we always
obtain the discrete units for the acoustic model input from the clean speech signal. By doing so, the model always receives high-quality linguistic input that is not corrupted by noise during training.

heading: 4.2 Pretraining
text: We first start from original PRETSSEL training as described in Equation (1) to initiate DINO training from the stable states of expressivity encoder. After obtaining converged expressivity encoder, we apply the DINO framework as detailed in Section 4. More specifically, we first obtain teacher and student expressivity encoders from the weights of stabilized expressivity encoder. Then, the student encoder is jointly trained along with acoustic model to minimize following loss terms:
Ltotal = Lpretssel + λdino · Ldino, (6)
where λdino denotes weight term of DINO loss. Next, the teacher encoder is updated by following Equation (4). This iteration is repeated until DINO loss is converged.

heading: 5 Experimental setting

heading: 5.1 Dataset
text: We used multi-speaker datasets covering two highresource languages, i.e., English (En) and Spanish (Es) to train PRETSSEL models. We provide a summary of the data statistics, including the number of utterances and duration for each language, in Table 1. For the evaluation, we used dev and test subsets of mExpresso English to Spanish (En→Es) and mDRAL Spanish to English (Es→En) benchmark dataset (Seamless Communication, 2023). More details about evaluation dataset are described in Appendix A.
Note that we could simulate clean environment because those mDRAL Es and mExpresso En speeches were recorded in a professional recording studio with minimal background noise. To simulate noisy environment, we obtained random noise signals from DNS-5 dataset (Dubey et al., 2023), and added those signals to source signals by different levels of SNR.

heading: 5.2 Preprocessing
text: We extracted XLS-R 10K units, 80-dimensional Mel-spectrograms, continuous F0, VUV flags, and energy. More details are described in Appendix B.

heading: 5.3 Architecture
text: As described in Section 3.1, the architecture of DINO-PRETSSEL is similar to the original PRETSSEL with the exception of the teacherstudent expressivity encoders and projection head layers. The projection head for each expressivity encoders has three fully connected layers with 2,048 hidden dimensions followed by L2 normalization and weight normalization layers (Salimans and Kingma, 2016). The output dimension K was set to 65,536. More details are described in Appendix C.

heading: 5.4 Training
text: We first trained DINO-PRETSSEL by 500k iterations using PRETSSEL criteria, and fine-tuned another 300k iterations by using DINO strategy. For noise augmentation, we randomly selected noise segments in each iteration, and mixed them with the source speech at a 50% probability with a random SNR ranging from 6dB to 40dB. We set the length of short and long segments to 4- and 6-seconds, respectively. We set the number of short and long segments to 4 and 2, respectively. More details are described in Appendix D.

heading: 5.5 Expressive S2ST inference
text: Firstly, we used the many-to-many version of Prosody UnitY2 (Seamless Communication, 2023) to translate input speech into target language’s XLS-R 10K units as described in Section 3.3. Then, the DINO-PRETSSEL took XLS-R 10K and last 4 seconds of Mel-spectrograms to synthesize the Mel-spectrograms at target language. Finally, the HiFi-GAN V1 vocoder (Kong et al., 2020) converted Mel-spectrograms into the speech waveform at 24-kHz sampling rate. Unlike original PRETSSEL work, we didn’t include audio watermarking technique to prevent possible distortion from watermarking.

heading: 5.6 Benchmarking systems
text: We included four expressive S2ST systems in the experiments. Note that in case of PRETSSELbased systems, we used the same Prosody UnitY2 and HiFi-GAN models used at original PRETSSEL
0 5 10 15 20Clean SNR (dB)
25 30 35 40 45 50
ASR-BLEU
PRETSSEL Denoiser + PRETSSEL
DINO-PRETSSEL (proposed) S2TT + TTS
S2T-BLEU
0 5 10 15 20Clean SNR (dB)
2.2 2.4 2.6 2.8 3.0 3.2
AutoPCP
0 5 10 15 20Clean SNR (dB)
5 10 15 20 25 30
Estimated SNR (dB)
Figure 2: Objective evaluation results of various expressive S2ST systems under different SNR conditions.
work (Seamless Communication, 2023) for a fair comparison.
S2TT + TTS. We combined a SeamlessM4T V2’s S2TT module (Seamless Communication, 2023) and Coqui-XTTS V2 model3. For the expressivity transfer, the source speech of SeamlessM4T V2 was conditioned to Coqui-XTTS V2.
PRETSSEL. We combined a Prosody UnitY2 and PRETSSEL model as described in Section 3.3.
Denoiser + PRETSSEL. We combined a Prosody UnitY2 and PRETSSEL with high-quality speech enhancement model. Specifically, we applied MetricGAN+ denoiser4 (Fu et al., 2021) to the input of PRETSSEL for removing noise components.
Proposed DINO-PRETSSEL We combined a Prosody UnitY2 and DINO-PRETSSEL as described in Section 5.5.

heading: 6 Objective evaluation

heading: 6.1 Metrics
text: In the objective evaluation, we measured ASRBLEU, AutoPCP, and SNR to measure contents preservation, prosody preservation, and noise suppression performances, respectively. We also included S2T-BLEU scores to specify the upperbound of ASR-BLEU score. We averaged the scores of individual utterances to obtain single representative score. We detailed the evaluation metrics in Appendix E.

heading: 6.2 Results
text: We present the objective evaluation results at Figure 2. In all cases, we confirmed the superior per-
3https://huggingface.co/coqui/XTTS-v2 4https://huggingface.co/speechbrain/
metricgan-plus-voicebank
formance of PRETSSEL-based models compared to the S2TT + TTS baseline. We analyze the trends among the PRETSSEL models as follows.
When the source speech was clean, the noiserobust PRETSSELs (i.e., DINO-PRETSSEL and denoiser-based PRETSSEL) showed a slight drop in prosody preservation performance compared to the original PRETSSEL (AutoPCP), while they demonstrated a slight improvement in content preservation performance (ASR-BLEU). As for the noise suppression performance, we found that DINO-PRETSSEL showed the closest SNR values to the original PRETSSEL. On the other hand, the denoiser-based PRETSSEL provided the highest SNR value, indicating the lowest noise level. We hypothesize that its significantly higher SNR value might indicate that the denoiser removed too much noise information that exists in the source speech, which could potentially degrade the naturalness of the translated speech. We will analyze this at Section 7.
When the source speech was corrupted by noise, the proposed DINO-PRETSSEL demonstrated the best contents preservation (ASR-BLEU), prosody preservation (AutoPCP), and noise suppression (SNR) performance regardless of input noise level. One interesting observation was that the proposed DINO-PRETSSEL model remained robustness to the noise, even when it was much stronger than the noise used during training, i.e., less than 6dB SNR. We presented full results in Appendix F.

heading: 7 Subjective evaluation

heading: 7.1 Metrics
text: For the subjective evaluation, we conducted MOS test to measure the naturalness of translated speeches. We also conducted S-MOS test that measures a vocal style similarity between source and translated speeches. We detailed the evaluation metrics in Appendix G.

heading: 7.2 Results
text: The subjective evaluation results are shown in Table 2. We analyze the trends as follows.
Naturalness. When the source speech was clean, DINO-PRETSSEL provided slightly lower naturalness than the original PRETSSEL (clean source; S4 vs. S2). This is mainly because DINOPRETSSEL’s training objective is broader than PRETSSEL’s objective. More specifically, DINO-
PRETSSEL needs to learn both denoising and expressivity preservation, whereas PRETSSEL only learns expressivity preservation. Because the clean speech translation case doesn’t consider denoising ability, DINO-PRETSSEL’s performance can be worse than PRETSSEL. This could be also interpreted as a trade-off for gaining noise-robustness at the expense of performance in clean environments. However, DINO-PRETSSEL still demonstrated higher naturalness than denoiser-based PRETSSEL (clean source; S4 vs. S3). This supported our hypothesis in Section 6.2 that removing too much noise with the denoiser could have a negative impact on the naturalness of the translated speech.
When the source speech became noisy, in contrast to other models, the naturalness performance of noise-robust PRETSSEL models even increased (noisy sources; S3 and S4). Specifically, DINOPRETSSEL proved to be able to generate the most natural speech compared to other systems in noisy environments (noisy source; S4 vs. others). More specifically, in noisy environments, DINOPRETSSEL achieved 3.54 and 3.64 MOS results, respectively 0.52 and 0.75 scores higher than the original PRETSSEL in the mExpresso (En→Es) and mDRAL (Es→Es) subsets.
Vocal style preservation. When the source speech was clean, the original PRETSSEL showed the highest performance (clean source; S2 vs. others). In contrast, DINO-PRETSSEL showed the lowest vocal style preservation performance among other PRETSSEL-based models (clean source; S4 vs. S2 and S3). This trend was similar to the results observed at MOS test, that means there was a trade-off for gaining noise-robustness.
However, when the source speech became noisy, DINO-PRETSSEL demonstrated outperforming robustness in vocal style preservation by achieving higher S-MOS results compared to other systems
(noisy source; S4 vs. others). More specifically, DINO-PRETSSEL achieved 3.02 and 3.15 S-MOS results in noisy environments, respectively 0.03 and 0.27 scores higher than the original PRETSSEL in the mExpresso (En→Es) and mDRAL (Es→Es) subsets.

heading: 8 Expressive S2ST in-the-wild
text: To test the robustness of DINO-PRETSSEL in realworld noisy recording environments, we conducted a subjective evaluation using noisy samples of the VoxLingua107 dataset (Valk and Alumäe, 2021), which were collected from En videos on YouTube. When choosing source speech, we first measured SNR, and randomly selected 100 noisy speeches having SNR values from 5dB to 15dB. We then applied Silero voice activity detection (Silero, 2021) to remove leading and trailing silence. Then, we conducted MOS and S-MOS tests for the En→Es translation as shown the results in Table 3. The evaluation results verified the proposed DINOPRETSSEL presented significantly higher performance to the noisy samples, especially compared to both PRETSSEL and denoiser-based PRETSSEL. Specifically, it achieved 3.59 MOS and 3.11 S-MOS results, which were 0.58 and 0.48 scores higher than those of original PRETSSEL, respectively.

heading: 9 Ablation study

heading: 9.1 Visualizing expressivity embeddings
text: The expressivity encoder of proposed DINOPRETSSEL can effectively extract consistent vocal style and prosody information from source speech even in noisy environment. To show this, we extracted expressivity embeddings from clean and noisy speeches, and drew the t-distributed stochastic neighbor embedding (t-SNE) plots (van der Maaten and Hinton, 2008). In particular, we chose confused, happy and sad speech samples from Expresso En benchmarking data (Seamless Communication, 2023), which showed clear differences in their speaking style. To simulate noisy environment, we randomly obtained noise from DNS5 dataset (Dubey et al., 2023), and mixed those with the speeches by 10dB SNR. When drawing the t-SNE plot, we marked clusters by following {speaker ID, style ID} labels.
As illustrated in Figure 3, both expressivity embeddings from PRETSSEL and DINO-PRETSSEL could be distinguishable by following the speaker and style labels when the source speech was clean. However, as illustrated in Figure 4, the PRETSSEL lost its ability to distinguish speaker and style, whereas the DINO-PRETSSEL still preserved the clusters when the input speech was corrupted by noise. This verified the robustness of proposed DINO-PRETSSEL’s expressivity encoder in the noisy environment.

heading: 9.2 Objective evaluation using ground-truth units
text: By using a ground-truth unit as input, we can simulate the best performance of the U2S models when there is no translation error from the preceding S2UT model. To evaluate the system in this situation, we generated speeches using ground-truth XLS-R 10K units at target language, and measured ASR-BLEU, AutoPCP and SNR metrics.
The evaluation results are presented in Figure 5. In overall, we could observe that the trends were similar to those reported in Section 6.2. However, there was a noticeable difference in the ASRBLEU results that DINO-PRETSSEL were not affected by the presence of noise in the source speech. This means that the drop in ASR-BLEU scores for DINO-PRETSSEL in the S2ST pipeline was primarily due to translation errors from the S2UT model. That means, it is possible further improve translation quality of S2ST system by enhancing the robustness of the S2UT model as well, and this could be a potential direction for future work. Full results are detailed in Appendix F.

heading: 10 Conclusion
text: In this paper, we proposed DINO-PRETSSEL, which incorporated DINO training strategy into the PRETSSEL-based U2S generator for the noiserobust S2ST system. As our method employed strong self-distillation method in learning expressivity representation, the U2S generator could robustly transfer source speech’s expressivity at S2ST in noisy recording environment. The objective and subjective evaluations conducted on noisy source speech consistently verified that the proposed DINO-PRETSSEL outperformed other systems through its high ASR-BLEU, AutoPCP, MOS, and S-MOS results. We also verified that DINO-PRETSSEL has robustness in real-world
noisy speech. Future works include improving the robustness of Prosody UnitY2 by utilzing this robust expressivity embedding. Additionally, we plan to explore additional data augmentation methods such as reverberation for the training of DINOPRETSSEL.

heading: 11 Limitations
text: The DINO strategy significantly reduces the pretraining speed because it requires extracting expressivity embeddings by multiple times, and the head layers also increases computational complexity. In our experiments, DINO-PRETSSEL took 12.7 days for pretraining, which is 5 days longer than PRETSSEL’s 7.9 days.
In addition, because the proposed system transfers source speech’s expressivity, it has potential risk of abusing biometric data of source speech. However, this risk can be mitigated by adopting audio watermarking technique to the translated speech waveforms (Roman et al., 2024).

heading: A Benchmarking data
text: The statistics of benchmarking data are shown in Table 4. The mExpresso En→Es dataset is distributed under the CC-BY-NC 4.0 license. The DNS-5 dataset is distributed under CC-BY-NC 0.0 and 4.0 licenses.

heading: B Pre-processing
text: All preprocessings were performed after adjusting sampling rate of input speech to 16-kHz.
Discrete units. Following Seamless Communication et al. (2023), we extracted continuous speech representations from 35th layer of XLS-R-1B model (Babu et al., 2022) at 20-ms frame interval. Then, we applied 10K K-means clustering algorithm on these representations to obtain discretized units. Finally, we deduplicated it to have unique value at each unit sequence their duration.
Mel-spectrogram. We extracted 80-dimensional Mel-spectrograms with frame size and hop size of 400 (25-ms) and 160 (10-ms). We applied zeromean and unit-variance normalization to input and output Mel-filterbank features to stabilize model training.
Pitch. To extract F0 and VUV flag, we first extracted F0 in every 5-ms by using DIO algorithm (Morise et al., 2016). Then, we obtained VUV flag specifying non-zero values of F0, while obtaining continuous F0 contour by linearly interpolating zero values. Finally, we converted linear F0 and energy values into log-scale. Using the duration of unit, F0 and VUV flag features were averaged to have a reduced unit-scale.
Energy. To extract energy, we extracted energy contour every 5 ms using a 35-ms Hanning window. Using the duration of unit, energy feature was averaged to have a reduced unit-scale.

heading: C Architecture
text: The details of hyperparameter used for DINOPRETSSEL architecture are described in Table 5. The hyperparameters were selected based on the ones that performed the best in our experiments. Note that the DINO training strategy requires an additional expressivity encoder with projection layers, which increases the required number of network parameters. However, these additional components are not necessary for inference, allowing for a reduction in model size.

heading: D Training
text: For the other PRETSSEL models, we trained the model by 800k iterations. The loss coefficients for local prosody prediction λlocal, FiLM regularization λfilm, and DINO loss λdino were set to 1.0, 10−4, and 0.5, respectively. We used the Adam optimizer with β1 = 0.9 and β2 = 0.98 with fixed learning rate of 10−4. We used gradient accumulation with frequency of four (Hermans et al., 2017).
Total 16 V100 GPUs were used to train a models. The DINO-PRETSSEL and PRETSSEL spent total 12.7 and 7.9 days for their pretraining, respectively. We implement our models based on the Fairseq toolkit (Ott et al., 2019).
In addition to the noise augmentation, we applied SpecAugment (Park et al., 2019) with frequency mask with a maximum width of 8 and time mask with a maximum width of 10 during training. Note that SpecAugment was only applied at student encoder. To alleviate language imbalance in training data, we applied temperature-based resampling (Holtzman et al., 2019) with the temperature set to 5.
For the DINO-related training setting,we adopted the commonly used DINO hyperparameters, which has proven to be robust in the works of image representation (Caron et al., 2021) speaker verification (Chen et al., 2023b), and text-tospeech (Pankov et al., 2023). Specifically, we set the student temperature τs to 0.1, whereas we linearly increased the teacher temperature τt from 0.04 to 0.07 during first 20,000 iterations. The momentum factor m for EMA rule of mean statistic was set to 0.9. EMA coefficients λema for teacher network update was initially set to 0.996, and gradually increased to 1.0 using cosine scheduler (Grill et al., 2020).

heading: E Objective evaluation metrics
text: All metrics were measured after adjusting speech’s sampling rate to 16-kHz.
ASR-BLEU. We obtained normalized transcription of translated speech using Whisper-Large ASR model5 (Radford et al., 2022). Then, we measured BLEU scores between reference and transcriptions (Papineni et al., 2002).
S2T-BLEU. We obtained translated text from S2TT components of Prosody UnitY2. Then, we measured BLEU scores between reference and transcriptions (Papineni et al., 2002).
AutoPCP. We computed utterance-level prosody preservation score by using AutoPCP-multilingualv2 model on source language and translated target language speeches (Seamless Communication, 2023).
SNR. Following the work of Seamless Communication (2023), we computed SNR from the energy
5https://github.com/openai/whisper
ratio between denoised speech and residual speech. Specifically, we applied DEMUCs denoiser6 (Defossez et al., 2020) to the original speech s for obtaining denoised speech ŝ. Then, we computed SNR as follows:
SNR = 10log10 ( ||ŝ||22 ||s− ŝ||22 ) , (7)
where || · ||2 denotes L2 norm.

heading: F Detailed objective evaluation scores
text: Evaluation of S2ST system with predicted units. The detailed results of S2ST systems for mExpresso (En→Es) and mDRAL (Es→En) are shown in Table 6 and Table 7, respectively.
Evaluation of U2S models with ground-truth units. The detailed results of U2S systems with ground-truth units for mExpresso (En→Es) and
6https://github.com/facebookresearch/denoiser
mDRAL (Es→En) are shown in Table 8 and Table 9, respectively.

heading: G Subjective evaluation metrics
text: For subjective evaluation, we randomly sampled total 100 and 90 utterances from the combined dev/test sets of mExpresso and mDRAL dataset, respectively. In case of VoxLingua107 dataset, we randomly sampled total 100 utterances from its training set. Each item was evaluated by three annotators. Before conducting evaluation, they were informed on the purpose of the human evaluation studies. A more detailed protocol explanation can be found in Seamless Communication et al. (2023).
MOS. We adopted the 5-point Likert scale MOS protocol (ITU-T Recommendation P.808, 2018) to evaluate the speech quality. The target language’s native speakers were asked to rate speech’s naturalness on the scores ranging from 1. Extremely
unnatural to 5. Extremely natural.
S-MOS. We adopted an S-MOS protocol to measure the similarity of source- and target-voices. Monolingual English listeners were asked to listen to both source and target audio and rate the similarity of the voices (disregarding the content and manner of the utterances) on a 5-point Likert scale ranging from 1. Not at all similar to 5. Extremely similar.

heading: Future Work
text: Future works include improving the robustness of Prosody UnitY2 by utilzing this robust expressivity embedding. Additionally, we plan to explore additional data augmentation methods such as reverberation for the training of DINOPRETSSEL