Title: DiffNorm: Self-Supervised Normalization for Non-autoregressive Speech-to-speech Translation
Venue: NeurIPS 2024
Accepted: yes
Keywords: speech-to-speech translation

non-autoregressive modeling

diffusion models
Abstract: Non-autoregressive Transformers (NATs) are recently applied in direct speech-to-speech translation systems, which convert speech across different languages without intermediate text data. Although NATs generate high-quality outputs and offer faster inference than autoregressive models, they tend to produce incoherent and repetitive results due to complex data distribution (e.g., acoustic and linguistic variations in speech). In this work, we introduce DiffNorm, a diffusion-based normalization strategy that simplifies data distributions for training NAT models. After training with a self-supervised noise estimation objective, DiffNorm constructs normalized target data by denoising synthetically corrupted speech features. Additionally, we propose to regularize NATs with classifier-free guidance, improving model robustness and translation quality by randomly dropping out source information during training. Our strategies result in a notable improvement of about $+7$ ASR-BLEU for English-Spanish (En-Es) translation and $+2$ ASR-BLEU for English-French (En-Fr) on the CVSS benchmark, while attaining over $14\times$ speedup for En-Es and $5 \times$ speedup for En-Fr translations compared to autoregressive baselines.

DIFFNORM: Self-Supervised Normalization for Non-autoregressive Speech-to-speech Translation

Abstract

Non-autoregressive Transformers (NATs) are recently applied in direct speech-tospeech translation systems, which convert speech across different languages without intermediate text data. Although NATs generate high-quality outputs and offer faster inference than autoregressive models, they tend to produce incoherent and repetitive results due to complex data distribution (e.g., acoustic and linguistic variations in speech). In this work, we introduce DIFFNORM, a diffusion-based normalization strategy that simplifies data distributions for training NAT models. After training with a self-supervised noise estimation objective, DIFFNORM constructs normalized target data by denoising synthetically corrupted speech features. Additionally, we propose to regularize NATs with classifier-free guidance, improving model robustness and translation quality by randomly dropping out source information during training. Our strategies result in a notable improvement of about +7 ASRBLEU for English-Spanish (En-Es) and +2 ASR-BLEU for English-French (En-Fr) translations on the CVSS benchmark, while attaining over 14 × speedup for En-Es and 5 × speedup for En-Fr translations compared to autoregressive baselines. 1

1 Introduction

Speech-to-speech translation (S2ST) systems are essential to bridge communication gaps and have wide application potential. We focus on non-autoregressive modeling for direct speech-to-speech translation, converting source speech to the target without intermediate text data. Such direct S2ST systems [25, 23, 22, 32, 31, 21, 20] can preserve non-linguistic information, avoid error propagation from cascaded systems [29, 34] (e.g., a combination of speech recognition and machine translation systems), and achieve faster inference speed.

Non-autoregressive Transformers (NAT) [21, 20] has played a central role in current S2ST work [31, 32, 21]. NAT translates source waveforms into target speech units via parallel decoding, achieving performance comparable to or better than autoregressive models while greatly reducing inference time. This process is often referred to as speech-to-unit (S2UT) translation [31]. The predicted speech units are then converted to target waveforms via a unit vocoder in the unit-to-speech synthesis stage [32, 38]. However, NATs suffer from incoherent and repetitive generations, referred to as the multi-modality problem [12]. This issue stems from NAT's assumption of conditional independence during parallel decoding, worsened by the complex and multi-modal nature of training data distribution.

In this work, we propose DIFFNORM, a self-supervised speech normalization strategy that alleviates the multi-modality problem of NAT models by simplifying the target distribution. Instead of distilling training data from an autoregressive model [10] or utilizing perturbed speech to train a normalizer [32, 21], we rely on the denoising objective of Denoising Diffusion Probabilistic Models [15, DDPM] to normalize target speech units. DIFFNORM inject synthetic noise to speech features and use diffusion

model to gradually recover the feature, obtaining a simplified and more consistent data distribution that obscures non-crucial details. As the denoising objective is learned in a self-supervised manner over latent speech representations, it eliminates the necessity for transcription data [32] or manually crafted perturbation functions [21]. As illustrated in Fig. 1, applying DIFFNORM to the target data obtains normalized speech units that lead to better NAT training for speech-to-unit translation.

Besides using DIFFNORM as a data-centric strategy to mitigate the multi-modality problem, we also propose a regularization strategy to enhance the NAT model's robustness and generalizability when facing complex data distribution (e.g., linguistic diversity and acoustic variation [21, 20]). Inspired by classifier-free guidance [17, CG], during training, we occasionally drop out source information and replace it with a "null" representation, compelling the models to generate coherent units without conditioning on the source data. During iterative parallel decoding of the NAT model, we obtain higher-quality translation by mixing conditional and unconditional generation. Ultimately, combining DIFFNORM and CG results in our top-performing state-of-the-art system, achieving approximately +7 and +2 ASR-BLEU increment for En-Es and En-Fr translation compared to previous non-autoregressive systems on the CVSS [24] benchmark.

In conclusion, we alleviate the multi-modality problem by proposing (1) diffusion-based normalization and (2) regularization with classifier-free guidance. To the best of our knowledge, we are the first to adapt diffusion models and classifier-free guidance into speech-to-speech translation and NAT modeling. Our methods obtain notable improvement compared to previous systems and maintain fast inference speed inherent in non-autoregressive modeling, achieving speedups of 14 × for En-Es and 5 × for En-Fr compared to autoregressive baselines.

2 Problem formulation and overview

We aim to develop a direct (textless) speech-to-speech translation system that transduces a source speech x = ( x 1 , · · · , x N ) into target speech. We follow Lee et al. [31] to reduce speech-to-speech translation into two sub-tasks: speech-to-unit translation and unit-to-speech synthesis. In this work, we focus on speech-to-unit translation and follow prior work [31, 32, 21] to use the same unit-to-speech component for a fair comparison. To generate speech units for the target language, we first extract speech feature h = ( h 1 , · · · , h M ) ∈ R M × H , where each feature has H dimensions. Subsequently, a K-means clustering model is trained on extracted features and used to generate speech units y =

speech fea ture

h

DiffNo rm:

speech normalization

target s peech

mHuBERT

S2UT model

source s peech

x

predicted units

y?

Unit-Vocoder

synthes ized ta rget s peech

normalized gold units

ynorm

LM head

z0

z1

zT

.. .

( y 1 , · · · , y M ) ∈ R M × 1 . Once the source speech x and target speech unit y are prepared, we train sequence-to-sequence models to translate from source speech into target units. In practice, we follow Lee et al. [32] to use a multilingual-HuBERT (mHuBERT) model to extract H = 768 dimensional speech features h . Then we use a 1000-cluster K-means model to predict speech units given the encoded features. 2 For speech-to-unit translation, we follow Huang et al. [21] to adopt Conditional Masked Language Modeling [10, CMLM], a kind of non-autoregressive transformer (NAT).

To mitigate NAT models' multi-modality problem, we propose DIFFNORM (section §3), which denoises synthetically corrupted speech features to construct normalized speech units y norm . As shown in Fig. 1, such normalized units are then used to train the S2UT (CMLM) model, which generates better-translated units for the unit vocoder to synthesize target speech.

Additionally, we propose to incorporate classifier-free guidance [17, CG], a widely adopted strategy for diffusion-based image generation, as a regularization method to improve the non-autoregressive speech-to-unit system (section §4). As shown in Fig. 3, by forcing NAT models to unmask target speech units without conditioning on source information, the model becomes more robust, generating coherent speech units that result in higher-quality translations. During inference, we follow classifierfree guidance to mix conditional and unconditional generation for the NAT model's iterative decoding, further enhancing its translation quality.

3 DIFFNORM: denoising diffusion models for speech normalization

Previous normalization strategy [32, 21] on speech-to-unit translation relies on connectionist temporal classification (CTC) fine-tuning [4] using the HuBERT [19] model. For example, Lee et al. [32] rely on single-speaker data from VoxPopuli [58] to produce normalized (speaker-invariant) speech units for HuBERT-CTC fine-tuning. On the other hand, Huang et al. [21] generate acoustic-agnostic units by perturbing rhythm, pitch, and energy information with (manually) pre-defined transformation functions. We propose to normalize speech units using self-supervised denoising objectives from Denoising Diffusion Probabilistic Models [15, DDPM]. We believe DDPM is more suitable for speech normalization because: (1) It only requires monolingual speech data, without the need of transcriptions to create a text-to-unit model as in [32]. (2) It learns to denoise features in a high-dimensional space rather than relying on hand-designed perturbation as in [21]. DIFFNORM consists of a variational auto-encoder (VAE) and a diffusion model. Since vanilla VAE [27] and DDPM [15] are applied to image generation, we modify them to support token generation and incorporate multitasking objectives. The V AE model reduces the dimension of the speech feature, mapping feature h into lower dimensional latent z . The diffusion model (visualized in Fig. 2) is then trained to denoise on the latent representation space, aiming to recover the original latent z 0 from the standard Gaussian noise z T . 3 In the subsequent sections, we begin by outlining the architecture of our V AE model (§3.1). Then, we delve into how the diffusion model is trained using the latent variables encoded by the V AE (§3.2).

3.1 Variational auto-encoders for latent speech representation

Since speech features encoded by mHuBERT have a high dimension ( H = 768 ), we compress the feature into lower-dimension latents for the diffusion model. The VAE model consists of an encoder, a decoder, and a language modeling head. Following our problem formulation (§2), we first prepare a sequence of target speech features h = ( h 1 , · · · , h M ) ∈ R M × H and their corresponding speech units y = ( y 1 , · · · , y M ) . Our VAE model's encoder will map the feature into lower dimension latent z = f ( h ; θ enc ) ∈ R M × Z where Z < H is the pre-defined latent dimension. Then VAE model's decoder reconstructs the speech feature ˆ h = f ( z ; θ dec ) and we apply the language modeling head to convert the reconstructed feature into a distribution over speech units' vocabulary v = f ( ˆ h ; θ lm ) ∈ R M × V . Following prior work [21, 32], we cluster speech features into V = 1000 units. The training objective is a weighted combination of reconstruction loss ( L recon), negative log-likelihood (NLL) loss ( L nll), and a Kullback-Leibler (KL) divergence term resulted from the Gaussian constraint [27] to regularize the representation space:

where the reconstruction loss is defined as L recon ( ˆ h , h ) = || h -ˆ h || 2 , and the NLL loss is computed as the cross-entropy between ground-truth units y and the predicted vocabulary distribution v : L nll ( v , y ) = -∑ M i =1 y i log v i , where y i is the one-hot version of y i . Lastly, L kl can be solved analytically when we assume Gaussian distribution for both prior and posterior approximation of latent z (more details in [27] and Appendix B). Though the Gaussian constraint has a small weight λ 3 , we show in the ablation study (§6.2) that regularizing latent distribution is critical for the diffusion model. For details on the architecture of our V AE model, we direct readers to Appendix B.

3.2 Diffusion model for denoising latent speech representation

Training Once the VAE model is trained, we encode speech feature as the first step feature z 0 = f ( h ; θ enc ) for the diffusion model. Diffusion models consists of a (1) forward process that

?dec

?enc

z0

zT

nois e injection

zT-1

z0

.. .

recons tructed

fea ture

?

LM-head

normalized

units

ynorm

speech

fea ture

h

gradually transforms z 0 into a standard Gaussian distribution, and a (2) reverse process that denoise and recovers the original feature z 0 . Following DDPM [15], with a pre-defined noise scheduler (let β t ∈ (0 , 1) be the scaling of noise variance, define α t = 1 -β t and denote ¯ α t = ∏ t s =1 α t as the noise level for time t ), the forward and reverse process can be written as:

where mean µ θ ( z t , t ) and variance σ 2 are parameterized by trainable models, typically based on U-Net [44] or Transformer [57]. We follow DDPM to train models using the re-weighted noise estimation objective:

L noise ( θ, t ) = ❊ x 0 , ϵ ,t || ϵ -ϵ θ ( z 0 , t ) || 2 (3) where the network learns to predict the injected Gaussian noise ϵ from the forward process. Besides noise estimation, we also train the model with auxiliary reconstruction and NLL loss using VAE model's decoder and language modeling head. Our training procedure is summarized in Alg. 1.

As shown in Alg. 1, we randomly sample the current timestep t ∈ [1 , T ] and compute corresponding scheduling parameters β t , α t , ¯ α t 4 . The training process involves the regular DDPM objective that injects Gaussian noise for the model ϵ θ to perform noise estimation (Alg. 1, line 6-8). Additionally, we generate the pseudo latent ˆ z 0 by reversing the noise injection process (line 9), which is then decoded by the VAE into speech features and units for reconstruction loss (line 10) and NLL loss (line 11). Finally, the objective is a weighted sum of noise estimation loss, reconstruction loss, and NLL loss:

In our analysis (§6.2), we show that adding NLL and reconstruction loss is indeed helpful in improving the diffusion model's reconstruction quality. Lastly, to parameterize our diffusion model for noise estimation, we modify Diffusion Transformer [37, DiT] to suit our task. For details of our architecture and hyper-parameters, we refer readers to Appendix B.

Inference For speech normalization, we choose a start time T ∈ [0 , 200] that decides the amount of noise to inject for the diffusion model to recover. Given the start time T , we follow Denoising Diffusion Implicit Models [48, DDIM] sampler to reverse noised latent z T back to z 0 . Then, our VAE model converts z 0 back to speech units with its decoder and language modeling head. Our inference procedure is summarized in Alg. 2 and visualized in Fig. 2.

Conformer blocks

encoder

repres enta tion

g

null repres enta tion

g?

repla ce with

p

drop

unmask prediction

target s peech

ynorm

Rand o m  Mas king

unit embedding

Decoder blocks

z0

z1

zT

.. .

DiffNorm

Note that since our diffusion model is used to normalize speech units as a data preprocessing strategy, inference speed is not a concern. Therefore we use step size ∆ t = 1 for DDIM sampling throughout our experiments. However, the inference speed could be easily improved by setting a larger step size.

4 Classifier-free guidance for non-autoregressive transformer

In §3, we proposed DIFFNORM to normalize speech units and obtained speech-unit pairs ( x , y norm ) that benefit NAT training. In this section, we propose to adapt classifier-free guidance [17] to regularize NAT models (visualized in Fig. 3), further enhancing NAT-based S2UT model's translation quality.

Training We largely adhere to standard CMLM training [21] but introduce a small dropout probability p drop = 0 . 15 for the encoder representations. Specifically, we parameterize the encoder and decoder of the CMLM as ϕ enc and ϕ dec , respectively. The encoder processes the source speech into a representation g = f ( x ; ϕ enc ) . The decoder then predicts the vocabulary distribution v = f (ˆ y | g ; ϕ dec ) from g and the randomly masked target speech units ˆ y . The total amount of masked token is uniformed sampled: n ∼ U [1 , M ] ; subsequently, n of M tokens from the target units y are randomly masked to form noisy target units ˆ y . CMLM also trains a length predictor that estimates the output length given input sequences and we refer readers to [21] for more details.

Using the predicted distribution v , we compute the NLL loss against the ground truth units y at the masked positions to train the model. If dropout is applied, the decoder receives a "null" representation g ∅ -a randomly-initialized learnable vector-forcing it to rely solely on the target information to unmask units. For more details, we refer readers to Appendix C and previous paper [10, 21].

Inference CMLM generates tokens through iterative unmasking. Given the input sequence x , CMLM first predicts the length of output sequence M and initialize a sequence of M masked tokens ˆ y 0 = ([ mask ] 1 , · · · , [ mask ] M ) . Given the total number of iterations T 5 and current iteration t ∈ [1 , T ] , CMLMdecodes all tokens y i , i ∈ [1 , M ] in parallel using their probabilities:

From the generated tokens ˆ y t = ( y t 1 , · · · , y t M ) , we replace k = M · T -t T tokens with the lowest probabilities p t i with mask tokens. This re-masked sequence ˆ y t is then inputted into the CMLM for further decoding iterations. This process continues until the final step t = T , at which point no tokens are re-masked. With our proposed regularization from classifier-free guidance, we compute two vocabulary distribution in each iteration step t :

where p t orig is the same as vanilla CMLM and p t uncond is the unconditional probability obtained with the "null" representation. Then we select and re-mask tokens using the adjusted probability

where the hyper-parameter ω is used to control the degree to push away from the unconditional distribution. In the special case of ω = 0 , only conditional distribution is used during iterative decoding, resulting in the same generation process as standard CMLM. Through our intensive analysis

(Appendix F), we determine that a relatively small ω (e.g., ω = 0 . 5 ) gives the best result, especially when the number of decoding iterations is large (i.e., T > 10 ). Notably, even when ω = 0 -which corresponds to the traditional CMLM decoding method - CMLMs regularized with classifier-free guidance during training can consistently outperform their standard counterparts. This observation further validates the effectiveness of our proposed regularization.

5 Experiments

5.1 Experimental setup

Dataset We perform experiments using the established CVSS-C datasets [24], which are created from COVOST2 by employing advanced text-tospeech models to synthesize translation texts into speech [59]. CVSS-C comprises aligned speech in multiple languages along with their respective transcriptions. Our methods are evaluated on two language pairs: English-Spanish (En-Es) and English-French (En-Fr), with detailed data statistics provided in Table 1. As our focus is on direct

speech-to-speech translation, transcriptions are solely utilized for evaluation purposes. Utilizing the speech data from CVSS, we preprocess the target speech to generate speech units using the mHuBERT and K-means model, as described in our problem formulation (§2).

Evaluation To evaluate the performance of various speech-to-speech translation systems, we adopt the standard methodology established in previous studies [32, 31, 21] to compute the ASR-BLEU score. Firstly, our speech-to-unit translation system generates speech units based on the input speech, which are then transformed into speech waveforms using a unit-vocoder. The unit-vocoder is built upon the HifiGAN architecture [28] with customized objectives, detailed in Appendix D. Once we have the waveforms, we employ an ASR model to transcribe them and calculate the BLEU score against the reference transcriptions. This ASR model is fine-tuned based on the WAV2VEC2.0 [5] model with the CTC objective. We direct readers to the original paper [7] for more details. Both the unit-vocoder and ASR models are sourced from an off-the-shelf repository, ensuring consistency in evaluation methodology with previous studies [32, 21]. 6

Normalized dataset construction We adhere to the procedure outlined in Alg. 2 to generate normalized speech units. By manipulating start time T , we control the level of noise in the latent z T = √ ¯ α t z 0 + √ 1 -¯ α t ϵ for the diffusion model, balancing between the reconstruction quality and normalization effect. We explore different levels of noise injection and choose T = 100 for En-Es and T = 120 for En-Fr (further details in §6.1). Hence, we adopt these settings to construct our normalized dataset, CVSS-NORM.

Compared systems In this section, we provide brief descriptions of evaluated speech-to-unit models. For autoregressive models, we evaluate the Transformer model trained following Lee et al. [31]. We also assess the Conformer model trained similar to the Transformer, but with its encoder replaced by a Conformer-Encoder [14]. Lastly, the Norm Transformer shares the same architecture as the Transformer, but it is trained on normalized speech units that are speaker-invariant. The normalized dataset is constructed following the strategy proposed by Lee et al. [32].

For non-autoregressive systems, we train the Conditional Masked Language Model (CMLM) following Huang et al. [21], using a Conformer-based encoder and a Transformer decoder. The CMLM + Bilateral Perturbation (BiP) system retains the same architecture as CMLM but is trained on normalized speech units constructed with BiP [21].

For our improved systems, we train CMLM + DIFFNORM, which shares the same architecture as CMLM but is trained on CVSS-NORM, the normalized dataset obtained through DIFFNORM. Additionally, we train the CMLM + CG model that incorporates classifier-free guidance introduced in §4. Finally, the CMLM + DIFFNORM + CG system uses the architecture of CMLM + CG and is trained on our normalized dataset CVSS-NORM.

5.2 Results

Table 2 summarizes the (ASR-BLEU) performances of various S2UT systems. Note that, for nonautoregressive methods, Table 2 shows their results obtained with 15 decoding iterations. For more details on the inference speedup achieved through non-autoregressive modeling, we plot Fig. 4 to provide details on the 'quality vs. latency' trade-off. Observations from Table 2 are:

DIFFNORM greatly enhances translation quality compared to systems using original speech units (model 6 vs. 4; 8 vs. 7). For example, CMLM with DIFFNORM achieves about +7 BLEU score on En-Es compared to CMLM trained on original units. DIFFNORM also outperforms previous normalization strategies (model 6 v.s. 2, 5), validating the superior quality of normalized speech units obtained through our methods. Besides normalization, classifier-free guidance effectively improves speech-to-unit quality as a regularization strategy, leading to better translation quality (model 7 v.s. 4; 8 v.s. 6). Finally, combining both classifier-free guidance and DIFFNORM (model 8) results in the best overall system, outperforming both autoregressive and non-autoregressive baselines.

Decoding speed In Fig. 4, we illustrate the 'quality-latency' trade-off of various non-autoregressive speech-to-unit systems. Quality is measured using ASR-BLEU, while latency is determined by the relative speedup over the autoregressive system, calculated as 'generated units/second'. For instance, the first marker on the line plot represents 15 decoding iterations, resulting in a speedup of 5.34 × compared to the autoregressive baseline. Additionally, we include the performance of the Conformer-based autoregressive model as a horizontal dashed line. We observe that our improved system consistently outperforms the baseline CMLM model [21] and achieves better performance than the autoregressive model with more than 14 × speedup for En-Es and 5 × speedup for En-Fr.

6 Analysis

6.1 Effect of synthetic noise injection

In this section, we explore the effects of varying degrees of noise injection on DIFFNORM. Recall (from Alg. 2) that synthetic noise is injected into the latent representation as z T = √ ¯ α T z 0 + √ 1 -¯ α T ϵ . Hence, adjusting the start time T allows us to control the degree of noise injection.

Setup and Evaluation We perturb the degree of noise injection by varying T , and assess the reconstruction quality of normalized speech units and downstream performance of CMLM models trained with different normalized units. For reconstruction quality, we measure the accuracy ( AccRec ) of reconstructed units and ASR-BLEU ( BL-Rec ) of synthesized speech when using original

6x

9x

12x

16x

10

15

20

Performance (BLEU)

AR=13.75

En-Es

6x

9x

12x

16x

7.5

10.0

12.5

15.0

17.5

AR=17.07

En-Fr

CMLM

CMLM + CG

CMLM + DiffNorm

CMLM + DiffNorm + CG

Relative Speedup

units/speech as the reference. For downstream performance, we report the downstream ASR-BLEU of translated speech produced by CMLM models trained with the particular noise setup ( BL-Dn ). 7

Results Shown in Table 5, increasing T leads to more noise and a decline in reconstruction quality, which aligns with expectations that higher noise levels pose greater challenges for the diffusion model in restoring original features accurately. Interestingly, we find from Table 3 that when T is too small or large (e.g., T = 50 or T = 150 ), the normalized units do not result in an ideal downstream system. We hypothesize that there is barely any normalization effect when T is too small; and when T is too large, the reconstructed units are no longer semantically correct. This phenomenon is visualized in Fig. 5, where we plot the log-mel spectrograms of the reconstructed speech. As more noise is introduced (larger T ), the reconstructed speech becomes smoother (e.g., portions of blank speech are filled in by diffusion), exhibiting a more pronounced deviation from the original speech.

From the Table 3, we observed a substantial enhancement in En-Es translation performance with T = 50 . Consequently, we conducted additional experiments for En-Es translation with start times T = 10 and T = 30 , detailed in Table 4. The results show that at a minimal start time of T = 10 , the reconstruction accuracy reaches 93.8%, yielding a downstream ASR-BLEU score of 15.98. This

score significantly surpasses the baseline CMLM result of 12.58. The high accuracy indicates that about 6% of tokens vary post-reconstruction, likely due to the V AE model's regularization effect since the diffusion model does not cause notable reconstruction deviations at such a small T . This suggests that the Spanish speech dataset might have considerable acoustic variations and noise, which the V AE model can partially mitigate. As T increases, the diffusion model further refines the representation, leading to improved downstream results, as evidenced by the rise in ASR-BLEU score from from

Mel frequency

Oiriginal

T = 100

T = 150

T = 200

T = 10 to T = 100 . This ablation study underscores that the optimal choice of T is affected by the dataset quality .

6.2 Ablation on training objectives for DIFFNORM

DIFFNORM requires an auto-encoder to map speech features into lower-dimension latents and then train denoising diffusion models using the encoded latents. In this section, we perform ablation experiments to investigate the effect of (1) latent dimension (2) Gaussian constraints (3) multitask objective for diffusion models. Firstly, to investigate the effect of latent dimension, we train auto-encoders that encode speech features into 16, 32, and 128 dimensions. As shown in Table 5, it comes as no surprise that a larger latent dimension yields superior reconstruction results, evidenced by higher accuracy across systems trained with varying objectives. Next, we turn our attention to the importance of applying Gaussian constraints to the auto-encoder's latent representation space. Our results (comparing

with and without KL ) reveal that incorporating Gaussian constraints is crucial. Latent spaces not regularized by L kl lead to significantly poorer performance. Finally, we explore the effectiveness of our proposed multitasking objective for the diffusion model by training another vanilla DDPM solely with L noise (no Multitask ). We find that the diffusion model's reconstruction capability indeed improves when employing the multitasking objective, particularly when denoising from a more noisy latent representation (e.g., T = 150 ). Through our ablation analysis, we identify the optimal setup for our DiffNorm model, which involves (1) mapping speech features to 128 dimensions, (2) regularizing latent space by Gaussian constraints, and (3) utilizing the multitask objective for diffusion training.

7 Related work

Direct speech-to-speech translation (S2ST) Direct S2ST aims to directly translate speech into another language without cascaded systems that rely on transcriptions or translations. Translatotron [25] and Translatotron 2 [23] are among the first systems for direct S2ST, which uses sequence-tosequence model to map source speech into spectrograms. Then, a spectrogram decoder is used to synthesize the target language's speech. Instead of transducing speech into spectrogram, Tjandra et al. [53], Zhang et al. [62] utilize Vector-Quantized Variational Auto-Encoder (VQ-V AE) [56] to discretize target speech and convert speech-to-speech translation into a speech-to-unit task. More recently, Lee et al. [31] improved the speech-to-unit models by obtaining such units with a k-means clustering model trained on self-supervised representation from HuBERT [19]. To convert units back to speech, Lee et al. [31] follows Polyak et al. [38] to train a unit-vocoder based on HifiGAN [2].

Speech normalization Speech representation are typically extracted from pre-trained encoders [19, 3, 40, 5] and can be compressed or adapted in different speech-to-speech/text tasks [61, 63, 50, 52, 51]. Inspired by previous work on speech enhancement [54, 1], Lee et al. [32] propose to

normalize speech by synthesizing speaker-invariant waveforms through text-to-speech (TTS) systems [60, 46, 42, 41]. Huang et al. [21] propose to normalize speech with Bilateral Perturbation that focus on the rhythm, pitch, and energy information. Different from previous normalization methods that requires transcription [32] or manually designed perturbation [21], our DIFFNORM strategy leverages diffusion models. Diffusion models [15, 48, 49] have achieved remarkable generative ability to produce high-quality images [8, 16, 43, 37, 45] and audio [33, 39, 47]. Using self-supervised denoising objectives [15], we train effective denoising models capable of normalizing speech for training better speech-to-unit systems.

Non-autoregressive speech-to-speech translation For sequence-to-sequence modeling, autoregressive [18, 6, 57] and non-autoregressive [12, 13, 10, 20] models have been widely explored. Lee et al. [31] reduce speech-to-speech into speech-to-unit task and follow the widely-used modeling strategy, Transformer [57], to predict speech units. Later, non-autoregressive models [21, 9] have been explored and Huang et al. [21] is the first to use a non-autoregressive transformer model, Conditional Masked Language Model [10, CMLM], for speech-to-unit task. Fang et al. [9], on the other hand, adopts Directed Acyclic Transformer [20] for speech-to-unit translation.

8 Conclusion

We improve speech-to-unit translation system through (1) corpus distillation by constructing normalized speech units with DIFFNORM and (2) regularization with classifier-free guidance. Our improved non-autoregressive (NAR) models, greatly outperform previous NAR models, yielding an increase of approximately +7 and +2 BLEU points for En-Es and En-Fr translation, respectively. Notably, DIFFNORM and classifier-free guidance maintain the inference speed advantages inherent in NAR models. Consequently, our approach obtains better performance to autoregressive baselines while achieving over 14 × speedup for En-Es and 5 × speedup for En-Fr translations.

References

Adiga, N., Pantazis, Y., Tsiaras, V., and Stylianou, Y. (2019). Speech Enhancement for NoiseRobust Speech Synthesis Using Wasserstein GAN. In Proc. Interspeech 2019 , pages 1821-1825.

Andreev, P., Alanov, A., Ivanov, O., and Vetrov, D. (2023). Hifi++: A unified framework for bandwidth extension and speech enhancement. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 1-5.

Babu, A., Wang, C., Tjandra, A., Lakhotia, K., Xu, Q., Goyal, N., Singh, K., von Platen, P., Saraf, Y., Pino, J. M., Baevski, A., Conneau, A., and Auli, M. (2021). Xls-r: Self-supervised cross-lingual speech representation learning at scale. In Interspeech .

Baevski, A., Auli, M., and rahman Mohamed, A. (2019). Effectiveness of self-supervised pre-training for speech recognition. ArXiv , abs/1911.03912.

Baevski, A., Zhou, H., Mohamed, A., and Auli, M. (2020). wav2vec 2.0: A framework for self-supervised learning of speech representations.

Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. CoRR , abs/1409.0473.

Chen, P.-J., Tran, K., Yang, Y., Du, J., Kao, J., Chung, Y.-A., Tomasello, P., Duquenne, P.-A., Schwenk, H., Gong, H., Inaguma, H., Popuri, S., Wang, C., Pino, J., Hsu, W.-N., and Lee, A. (2023). Speech-to-speech translation for a real-world unwritten language. In Rogers, A., Boyd-Graber, J., and Okazaki, N., editors, Findings of the Association for Computational Linguistics: ACL 2023 , pages 4969-4983, Toronto, Canada. Association for Computational Linguistics.

Dhariwal, P. and Nichol, A. (2021). Diffusion models beat gans on image synthesis.

Fang, Q., Zhou, Y., and Feng, Y. (2023). DASpeech: Directed acyclic transformer for fast and high-quality speech-to-speech translation. In Advances in Neural Information Processing Systems .

Ghazvininejad, M., Levy, O., Liu, Y., and Zettlemoyer, L. (2019). Mask-predict: Parallel decoding of conditional masked language models. In Inui, K., Jiang, J., Ng, V ., and Wan, X., editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 6112-6121, Hong Kong, China. Association for Computational Linguistics.

Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative adversarial networks.

Gu, J., Bradbury, J., Xiong, C., Li, V. O., and Socher, R. (2018). Non-autoregressive neural machine translation. In International Conference on Learning Representations .

Gu, J., Wang, C., and Zhao, J. (2019). Levenshtein transformer. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., and Garnett, R., editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc.

Gulati, A., Qin, J., Chiu, C., Parmar, N., Zhang, Y ., Yu, J., Han, W., Wang, S., Zhang, Z., Wu, Y ., and Pang, R. (2020). Conformer: Convolution-augmented transformer for speech recognition. CoRR , abs/2005.08100.

Ho, J., Jain, A., and Abbeel, P. (2020). Denoising diffusion probabilistic models. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Systems , volume 33, pages 6840-6851. Curran Associates, Inc.

Ho, J., Saharia, C., Chan, W., Fleet, D. J., Norouzi, M., and Salimans, T. (2021). Cascaded diffusion models for high fidelity image generation.

Ho, J. and Salimans, T. (2022). Classifier-free diffusion guidance.

Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural computation , 9(8):1735-1780.

Hsu, W.-N., Bolte, B., Tsai, Y .-H. H., Lakhotia, K., Salakhutdinov, R., and Mohamed, A. (2021). Hubert: Self-supervised speech representation learning by masked prediction of hidden units.

Huang, F., Zhou, H., Liu, Y., Li, H., and Huang, M. (2022). Directed acyclic transformer for non-autoregressive machine translation. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S., editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 9410-9428. PMLR.

Huang, R., Liu, J., Liu, H., Ren, Y ., Zhang, L., He, J., and Zhao, Z. (2023). Transpeech: Speechto-speech translation with bilateral perturbation. In The Eleventh International Conference on Learning Representations .

Inaguma, H., Popuri, S., Kulikov, I., Chen, P.-J., Wang, C., Chung, Y.-A., Tang, Y., Lee, A., Watanabe, S., and Pino, J. (2023). UnitY: Two-pass direct speech-to-speech translation with discrete units. In Rogers, A., Boyd-Graber, J., and Okazaki, N., editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 15655-15680, Toronto, Canada. Association for Computational Linguistics.

Jia, Y ., Ramanovich, M. T., Remez, T., and Pomerantz, R. (2022a). Translatotron 2: High-quality direct speech-to-speech translation with voice preservation.

Jia, Y., Ramanovich, M. T., Wang, Q., and Zen, H. (2022b). Cvss corpus and massively multilingual speech-to-speech translation.

Jia, Y ., Weiss, R. J., Biadsy, F., Macherey, W., Johnson, M., Chen, Z., and Wu, Y . (2019). Direct speech-to-speech translation with a sequence-to-sequence model. ArXiv , abs/1904.06037.

Kingma, D. P. and Ba, J. (2017). Adam: A method for stochastic optimization.

Kingma, D. P. and Welling, M. (2022). Auto-encoding variational bayes.

Kong, J., Kim, J., and Bae, J. (2020). Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis.

Lavie, A., Waibel, A., Levin, L., Finke, M., Gates, D., Gavalda, M., Zeppenfeld, T., and Zhan, P. (1997). Janus-iii: speech-to-speech translation in multiple languages. In 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing , volume 1, pages 99-102 vol.1.

Lecun, Y., Bottou, L., Bengio, Y., and Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324.

Lee, A., Chen, P.-J., Wang, C., Gu, J., Popuri, S., Ma, X., Polyak, A., Adi, Y., He, Q., Tang, Y., Pino, J., and Hsu, W.-N. (2022a). Direct speech-to-speech translation with discrete units. In Muresan, S., Nakov, P., and Villavicencio, A., editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 3327-3339, Dublin, Ireland. Association for Computational Linguistics.

Lee, A., Gong, H., Duquenne, P.-A., Schwenk, H., Chen, P.-J., Wang, C., Popuri, S., Adi, Y ., Pino, J., Gu, J., and Hsu, W.-N. (2022b). Textless speech-to-speech translation on real data. In Carpuat, M., de Marneffe, M.-C., and Meza Ruiz, I. V., editors, Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 860-872, Seattle, United States. Association for Computational Linguistics.

Liu, H., Chen, Z., Yuan, Y., Mei, X., Liu, X., Mandic, D., Wang, W., and Plumbley, M. D. (2023). AudioLDM: Text-to-audio generation with latent diffusion models. Proceedings of the International Conference on Machine Learning .

Nakamura, S., Markov, K., Nakaiwa, H., Kikui, G., Kawai, H., Jitsuhiro, T., Zhang, J.-S., Yamamoto, H., Sumita, E., and Yamamoto, S. (2006). The atr multilingual speech-to-speech translation system. IEEE Transactions on Audio, Speech, and Language Processing , 14(2):365-376.

Nichol, A. Q. and Dhariwal, P. (2021). Improved denoising diffusion probabilistic models. In Meila, M. and Zhang, T., editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 8162-8171. PMLR.

Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., and Auli, M. (2019). fairseq: A fast, extensible toolkit for sequence modeling. In Ammar, W., Louis, A., and Mostafazadeh, N., editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations) , pages 48-53, Minneapolis, Minnesota. Association for Computational Linguistics.

Peebles, W. and Xie, S. (2023). Scalable diffusion models with transformers.

Polyak, A., Adi, Y., Copet, J., Kharitonov, E., Lakhotia, K., Hsu, W.-N., Mohamed, A., and Dupoux, E. (2021). Speech resynthesis from discrete disentangled self-supervised representations.

Popov, V., Vovk, I., Gogoryan, V., Sadekova, T., and Kudinov, M. (2021). Grad-tts: A diffusion probabilistic model for text-to-speech. In Meila, M. and Zhang, T., editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 8599-8608. PMLR.

Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. (2022). Robust speech recognition via large-scale weak supervision.

Ren, Y., Hu, C., Tan, X., Qin, T., Zhao, S., Zhao, Z., and Liu, T.-Y. (2022). Fastspeech 2: Fast and high-quality end-to-end text to speech.

Ren, Y., Ruan, Y., Tan, X., Qin, T., Zhao, S., Zhao, Z., and Liu, T.-Y . (2019). Fastspeech: Fast, robust and controllable text to speech.

Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2022). High-resolution image synthesis with latent diffusion models.

Ronneberger, O., Fischer, P., and Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation.

Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S. K. S., Ayan, B. K., Mahdavi, S. S., Lopes, R. G., Salimans, T., Ho, J., Fleet, D. J., and Norouzi, M. (2022). Photorealistic text-to-image diffusion models with deep language understanding.

Shen, J., Pang, R., Weiss, R. J., Schuster, M., Jaitly, N., Yang, Z., Chen, Z., Zhang, Y ., Wang, Y ., Skerry-Ryan, R., Saurous, R. A., Agiomyrgiannakis, Y., and Wu, Y. (2018). Natural tts synthesis by conditioning wavenet on mel spectrogram predictions.

Shen, K., Ju, Z., Tan, X., Liu, Y., Leng, Y., He, L., Qin, T., Zhao, S., and Bian, J. (2023). Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers.

Song, J., Meng, C., and Ermon, S. (2021a). Denoising diffusion implicit models. In International Conference on Learning Representations .

Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. (2021b). Score-based generative modeling through stochastic differential equations.

Tan, W., Chen, Y., Chen, T., Qin, G., Xu, H., Zhang, H. C., Durme, B. V., and Koehn, P. (2024a). Streaming sequence transduction through dynamic compression.

Tan, W., Inaguma, H., Dong, N., Tomasello, P., and Ma, X. (2024b). Ssr: Alignment-aware modality connector for speech language models.

Tang, C., Yu, W., Sun, G., Chen, X., Tan, T., Li, W., Lu, L., Ma, Z., and Zhang, C. (2024). Salmonn: Towards generic hearing abilities for large language models.

Tjandra, A., Sakti, S., and Nakamura, S. (2019). Speech-to-speech translation between untranscribed unknown languages. 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) , pages 593-600.

Valentini-Botinhao, C., Wang, X., Takaki, S., and Yamagishi, J. (2016). Speech Enhancement for a Noise-Robust Text-to-Speech Synthesis System Using Deep Recurrent Neural Networks. In Proc. Interspeech 2016 , pages 352-356.

van den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., and Kavukcuoglu, K. (2016). Wavenet: A generative model for raw audio.

van den Oord, A., Vinyals, O., and kavukcuoglu, k. (2017). Neural discrete representation learning. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R., editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. (2017). Attention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R., editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc.

Wang, C., Riviere, M., Lee, A., Wu, A., Talnikar, C., Haziza, D., Williamson, M., Pino, J., and Dupoux, E. (2021). VoxPopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. In Zong, C., Xia, F., Li, W., and Navigli, R., editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 993-1003, Online. Association for Computational Linguistics.

Wang, C., Wu, A., and Pino, J. (2020). Covost 2 and massively multilingual speech-to-text translation.

Wang, Y., Skerry-Ryan, R. J., Stanton, D., Wu, Y., Weiss, R. J., Jaitly, N., Yang, Z., Xiao, Y., Chen, Z., Bengio, S., Le, Q. V., Agiomyrgiannakis, Y., Clark, R. A. J., and Saurous, R. A. (2017). Tacotron: Towards end-to-end speech synthesis. In Interspeech .

Yu, W., Tang, C., Sun, G., Chen, X., Tan, T., Li, W., Lu, L., Ma, Z., and Zhang, C. (2023). Connecting speech encoder and large language model for asr.

Zhang, C., Tan, X., Ren, Y., Qin, T., Zhang, K., and Liu, T.-Y. (2020). Uwspeech: Speech to speech translation for unwritten languages.

Zhang, D., Li, S., Zhang, X., Zhan, J., Wang, P., Zhou, Y., and Qiu, X. (2023). Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities.

Supplementary Material

A Qualitative Examples

From §5.2 we demonstrate the effectiveness of DiffNorm and classifier-free guidance by evaluating ASR-BLEU scores. In this section, we provide example transcriptions of generated Spanish and French speech in Table 6 and Table 7. Compared to the vanilla CMLM model that has erroneous or incomplete translation, our method results in higher-quality translation.

B Diffusion Model Hyperparameters

B.1 Variational Auto-Encoder

Architecture : Our Auto-encoder consists of an encoder, a decoder, and a language modeling head. The encoder architecture follows WaveNet [55] that combines Convolutional Neural Networks [30] with residual connections. In practice, we use 2 stacks of WaveNet Residual Blocks, where each block has 3 layers. Our decoder uses the same architecture as the encoder to revert the latent dimension back

to the original dimension. Additionally, we apply a Transformer [57] encoder on top of the decoded feature to further enhance model capacity. The Transformer-encoder has the same configuration as the vanilla Transformer-base model (6 layers, 8 attention heads, etc.). Lastly, to decode speech units from the feature, we use a language modeling head which is parameterized by a feedforward network that converts feature dimension ( H = 768 ) into vocabulary size ( V = 1000 ).

Training : To train the V AE model, as described in §3.1, we use a combination of three objectives

where λ 1 = 100 , λ 2 = 1 , λ 3 = 0 . 001 . Now we provide more details about L kl: in practice, we follow [27] to model the latent z by estimating its mean µ and variance σ using the encoder. With the re-parameterization trick, the latent is sampled as z = µ + σ · ϵ where ϵ ∼ N (0 , I ) . By constraining the mean and variance to follow a Gaussian prior, Kingma and Welling [27] showed that

We implement our VAE model on Fairseq [36]. For optimization, we use the Adam [26] optimizer with betas (0 . 9 , 0 . 98) and we apply gradient clipping by setting -clip-norm=2.0. During training, we apply dropout with a probability of 0.1. We train the V AE model using a learning rate of 5e-4 with distributed data-parallel (DDP) on 4 A100 GPUs, where we set the maximum batch token to be 15000.

B.2 Latent Diffusion Model

Architecture : We modified DiT [37] to design a Transformer-based architecture for noise estimation. Since our input latent is a sequence of encoded speech feature z ∈ R M × Z , we first apply a 1D Convolution to convert the latent to our model's hidden dimension (set to 512). Then we also encode the timestep with a learnable Sinusoidal Positional Embedding [57] to obtain the time embedding.

Subsequently, we feed the latent and time embedding to our modified WaveNet Block, which applies an affine transformation of the latent using the time embedding, similar to the "Scale and Shift" operation in DiT's adaptive layer norm (adaLN) block. For our diffusion model, we use 4 stacks of such modified WaveNet Block, each containing 8 layers. The output feature from WaveNet is then passed through a Transformer-Encoder (12 layers with 8 attention heads) to further enhance model capacity. Lastly, a projection layer parameterized by the feedforward network is used to compress the transformed feature to the latent dimension, which is then used for noise estimation in equation (3).

Training : As described in §3.2, we train the diffusion model with a multitask objective:

and we empirically select γ 1 = 1 , γ 2 = 0 . 25 , γ 3 = 0 . 005 . For optimization, our latent diffusion model has the same setting as our VAE model (with Adam optimizer, clip-norm equals 2.0, and dropout with 0.1 probability). We train the diffusion model using a learning rate of 1e-4 with DDP on 4 A100 GPUs, where we set the maximum batch token to be 12000. The model is warmup-ed by 10000 steps.

C Non-autoregressive Transformer Details

C.1 Background on Non-Autoregressive Transformers

Non-autoregressive Transformers [12, 13, 10, inter alia. ] is a family of models that transduce sequences. In this work, we follow the formalization from the widely-used NAT: Conditional Masked Language Model [10, CMLM]. CMLM consists of an encoder ϕ enc that represents the input as high-dimensional features and a decoder ϕ dec that generates tokens conditioning on the encoded features. Different from auto-regressive Transformers [57], CMLM's decoder does not apply causal masking and is trained with an unmasking objective.

CMLMTraining Following formulation from §2, given the input speech x = ( x 1 , · · · , x N ) and target speech units y = ( y 1 , · · · , y M ) , CMLM first mask the target units into ˆ y . The total amount of masked token is uniformed sampled: n ∼ U [1 , M ] ; subsequently, n of M tokens from the target units y are randomly masked to form noisy target units ˆ y . Then, the decoder generates a distribution over vocabulary conditioning on the encoder representation and noisy target units v = f (ˆ y | g ; ϕ dec ) where g = f ( x ; ϕ enc ) is the encoder representation. Cross-entropy loss is then computed for the masked positions to update model parameters. CMLM also trains a length predictor that estimates the output length given input sequences and we refer readers to [21] for more details.

CMLMMask-Predict : For CMLM inference, an iterative decoding scheme is used through unmasking speech units and re-masking low-confident positions. Given the input sequence x , CMLM first predicts the length of output sequence M and initialize a sequence of M masked tokens ˆ y 0 = ([ mask ] 1 , · · · , [ mask ] M ) . Given the total number of iterations T and current iteration t ∈ [1 , T ] , CMLMdecodes tokens across all positions and also computes their log-probabilities:

From the generated tokens ˆ y t = ( y t 1 , · · · , y t M ) , we replace k = M · T -t T tokens with the lowest probabilities p t i with mask tokens. This re-masked sequence ˆ y t is then inputted into the CMLM for further decoding iterations. This process continues until the final step t = T , at which point no tokens are re-masked.

C.2 Architecture

We follow the same architecture and implementation from Huang et al. [21], where a Conformer-based encoder is used to obtain representation from the source speech and a Transformer-decoder (with no causal masking) is used to generate units. The Conformer-encoder has 12 layers with 512 hidden dimensions and 9 attention heads. The encoder also contains a CNN-based sub-sampler that has an effective stride size of 320 (i.e., it reduces the length of speech input by 320 × ). The Transformerdecoder has 6 layers with 8 attention heads and uses a fixed positional embedding for speech units. When classifier-free guidance is used, we randomly dropout encoding from the Conformer model with a probability of 15% as described in §4.

C.3 Hyper-parameter for Model Training

We train the model with a learning rate of 5e-4, using 10000 warmup steps We apply Adam optimizer with betas (0 . 9 , 0 . 98) and we clip the gradient by setting -clip-norm=10. We apply dropout with 0.1 probability and we use label smoothing of ratio 0.2 when computing the NLL loss. We train the model with DDP on 4 A100 GPUs, with a maximum batch tokens of 40,000.

D Unit-to-Speech Synthesis

Wefollow prior work [32, 38, 41] to convert predicted units (from S2UT model) into speech waveforms. Specifically, given units y , we use the HifiGAN-based unit-vocoder from [38] to synthesize speech. The unit-vocoder is trained using a generator G and discriminator network D . During training, the generator is updated with reconstruction loss based on the mel-spectrogram of the true and synthesized waveforms. Additionally, adversarial loss and feature-matching loss are added to enhance the fidelity of generated speech. To train the discriminator, Polyak et al. [38] follows the minmax objective from

GAN [11]. After training, the discriminator is discarded and only the generator is used to synthesize waveforms.

We follow [32] to use a unit-vocoder that is also trained with a duration predictor so that it can synthesize speech with reduced speech units. For more details, we refer readers to the discussion from the original paper [32, 41], as well as the open-sourced repository: github.com/facebookresearch/ fairseq/examples/speech_to_speech

E Limitations and Broader Impacts

Limitations : The major limitation of the work is the data pre-processing cost from DIFFNORM, as our method requires inferencing diffusion model on all training samples to obtain better speech units. With the CVSS benchmark, both En-Es and En-Fr have less than 1M pairs, which can be processed easily with our resource (4 A100 GPUs). However, when the dataset scales up, the corpus distillation cost could be large.

Broader Impacts : Our system helps bridge the communication gap between users of different languages. However, as our system involves speech synthesis, it also has a risk of being used to mimic a particular speaker.

F Ablation on Classifier-free Guidance

In this section, we compare non-autoregressive transformers trained with and without classifier-free guidance (CG). We train CMLM models with classifier-free guidance and evaluate them with 5 , 10 , 15 iterations of decoding. From Table 8, we observe that CG improves the quality of translation whether the model is trained on original or normalized speech units. We find CG brings more improvement on the original units than the normalized units. This happens because normalized speech units are more conformed and already result in a large improvement in their translation quality, making the regularization effect from CG less obvious. Nevertheless, the best-performing system is achieved with both DIFFNORM and CG.

Comparing different hyperparameters w , we find a small value like w = 0 . 5 or w = 1 brings the most improvement empirically, and such improvements are more noticeable when the number of decoding iterations is larger. For example, comparing the results under 5 and 15 iterations, we find w = 0 gives better results when the number of iterations is small while w = 0 . 5 obtains the best performance with 15 iterations.

Table 8: Speech-to-speech translation performance of CMLM models with different CG hyperparameters.

G Experiment Result Tables

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We propose strategies to improve speech-to-speech translation which is covered in our abstract and introduction sections.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See Appendix E for our discussion on limitations.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: There is no theoretical proof involved in this paper.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide details of our implementation (architecture and training hyper-parameters) in Appendix B and Appendix C. We will also release our code and the dataset we used (CVSS) is publicly available at https://github.com/ google-research-datasets/cvss .

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may

be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The data is publicly available at: https://github.com/ google-research-datasets/cvss . The code is publicly accessible at: https://github.com/steventan0110/DiffNorm

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide all our implementation detail in Appendix B and Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We follow the standard evaluation setting, using the same dev and test set as prior work, and evaluate our method in two language directions (English-Spanish and English-French). Our results are consistent over languages, and our ablation studies that introduce additional experiments also confirm our method's effectiveness. Lastly, we improve upon the baseline with a notable +7 BLEU increment for En-Es and +2 BLEU increment for En-Fr over a test set that has more than 10,000 samples. With the evidence above, we believe our improvement is solid and alleviates the need to re-run experiments for significance tests. We believe our practice is standard and follows from previous work on speech-to-speech translation.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: As noted in Appendix B and Appendix C, we use 4 A100 GPUs with Distributed Data-Parallel setting throughout our experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We follow the Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: See Appendix E for our discussions.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our work focuses on direct speech-to-speech translation and we do not believe our model has a high risk for misuse as it does not support voice cloning or other high-risk applications.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite and describe the used dataset or models throughout the paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not have new assets in this paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No human study is involved in this research.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No human subjects or crowdsourcing are involved.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work builds on TRANSPEECH (Huang et al., 2023) by applying the diffusion method to reduce noise, thereby normalizing speech units for further generation. The authors further use classifier-free guidance to enhance non-autoregressive generation. They conduct experiments on CVSS En-Fr and En-Es datasets, comparing their methods with the baseline. Both proposed methods show improvements.\n\nSTRENGTHS:\n1. This work proposes a diffusion method to normalize target speech units, which outperforms previous approaches.\n\n2. It explores classifier-free guidance to improve NAT generation for the speech-to-speech translation task.\n\nWEAKNESSES:\n1. The paper overstates its contributions. The authors claim to be the first to apply diffusion in the speech-to-speech translation task. However, they merely use it to generate auxiliary training targets and still follow the S2UT strategy (Lee et al., 2022b). This diminishes the novelty of the paper.\n2. Although the work is based on TRANSPEECH, the authors compare only two of the three translation directions. One of the core contributions is applying the diffusion method to normalize speech units, yet the method shows minimal improvement (only 0.3 BLEU) compared to BiP (Bilateral Perturbation) in the En-Fr task. Given the method significantly outperforms others in the En-De task, this result is perplexing.\n3. For a minor suggestion, in Tables 3 and 4, ensure consistent significant figures. Additionally, it would be more appropriate to place Table 3 before Table 4.\n\nQUESTIONS:\n1. How does the performance fare on the CVSS Fr-En task?\n2. In my opinion, the goal of normalizing the unit is to disentangle linguistic information from noisy speech features. If you use DDPM for this process, the target $z_0$, which is the output of the VAE encoder, should be the linguistic feature. How do you ensure that $z_0$ represents the desired linguistic information?\n\nLIMITATIONS:\nThe authors appropriately state the limitations and broader impacts of this work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes DiffNorm, a diffusion-based self-supervised method for speech data normalization, aiming to alleviate multimodal problem in non-autoregressive speech-to-speech translation (NAT). DiffNorm consists of a VAE to reconstruct the speech feature and a diffusion model to add and remove noise of latent vector. Experiment shows that DiffNorm significantly improve NAT translation quality compared to baselines.\n\nSTRENGTHS:\n1. It is surprising to see pure data normalization improves NAT so much. ASR-BLEU improves 7 BLEU on En-Es direction with DiffNorm. \n2. This technique has wide applicability. All NAT S2S unit-based models can potentially benefit from it. Not to mention it requires no handcrafting rules for noise injection.\n3. Ablation studies on noise level and training of DiffNorm further provide users a general guide on how to adapt DiffNorm to their own dataset and model.\n\nWEAKNESSES:\n1. The multi-modal problem has two aspects: semantic and acoustic. DiffNorm seems able to reduce acoustic modalities. Unclear if DiffNorm can also do that on semantic modalities, i.e., multiple feasible translations for the same source input.\n2. Classifier-free guidance combined with DiffNorm leads to worse performance than DiffNorm alone in Figure 4 and the authors ignore it in the text. Elaboration is needed here.\n3. Lack baseline comparison with several latest S2ST models like TransSentence [1], PolyVoice [2], SeamlessM4T and etc.\n\n[1] TranSentence: speech-to-speech Translation via Language-Agnostic Sentence-Level Speech Encoding without Language-Parallel Data.\n[2] PolyVoice: Language Models for Speech to Speech Translation\n[3] SeamlessM4T—Massively Multilingual & Multimodal Machine Translation\n\nQUESTIONS:\n1. Issues listed in weakness.\n2. What is the role of VAE here? Can we drop it and directly use a diffusion model here for denoising? \n3. Table 3 mentions larger dimension of latent vector brings better performance, why not use the original number of dimensions instead of compressing it?\n4. Why improvement on En-Fr is not that significant compared to En-Es? How does language direction impact the performance?\n5. Is there way to visualize the noise added on the latent vectors? Like what does it mean for the audio?\n\nLIMITATIONS:\nBesides what I have already mentioned in the weakness, experiments in the paper are only conducted on En-X, but not reverse. Also, it would be interesting to see how DiffNorm works on non-European languages like Chinese and Japanese.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce a process aiming to simplify the target distribution of speech-to-speech translation. This process uses a VAE model to map features to a latent space, followed by a diffusion model to normalize the features in the latent space. The authors use the generated dataset to train a non-autoregressive CMLM model on the CVSS-C dataset to validate its effectiveness.\n\nSTRENGTHS:\n1. The authors propose a novel speech normalization method using Denoising Diffusion Probabilistic Models.\n2. The quality gain in En-Es direction is impressive.\n\nWEAKNESSES:\n1.  I am confused by the rationale behind using a diffusion model to normalize the speech representation. The authors first add noise to the speech representation (Forward Process) and then remove the noise (Backward Process). This design seems awkward to me. Why do you think this process of adding and then removing noise can help with normalization?\n2. Using a Variational Autoencoder to map features to a latent space seems contradictory to the motivation of reducing data multimodality. The VAE provides an indefinite mapping, which may hinder efforts to reduce multimodality. Additionally, $z_0 = f(h; \\theta_{enc})$ is not correct; the VAE provides a distribution over the latent space. A more suitable expression would be $z_0 \\sim p(z; f(h; \\theta_{enc}))$.\n3. The experiments are only conducted on synthesised dataset. As the paper's contribution is to alleviate the multi-modality problem in data, conducting experiments on a real speech-to-speech dataset, like [1], is much better to support the major claims.\n4. The baseline results (Conformer in EnEs) seem quite low compared to the reported results in the literatre.\n\n[1] Lee et al., Textless speech-to-speech translation on real data.\n\nQUESTIONS:\n1. A simple CG method in En-Es can bring an improvement about 4.5 BLEU (Table 2, Line 7), any reasons behind it?\n2. In Table 4, Acc-Rec, BL-Rec, and BL-Dn seem highly correlated when applying your method. BL-Dn performs well with minor perturbation and vice versa. However, when there is no perturbation, BL-Dn degrades dramatically. What is the reason for this? Is there a borderline value of $T$ that separates these two regions? I think this exploration is important for understanding the effectiveness of using the diffusion process to normalize speech.\n\nLIMITATIONS:\nThe authors have addressed the limitations in Appendix E.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}