DiffNorm: Self-Supervised Normalization for Non-autoregressive Speech-to-speech Translation

Authors: weiting  tan
jingyu  zhang
lingfeng  shen
daniel  khashabi
philipp  koehn
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 06:45:18

Abstract: Non-autoregressive Transformers (NATs) are recently applied in direct speech-to-speech translation systems, which convert speech across different languages without intermediate text data. Although NATs generate high-quality outputs and offer faster inference than autoregressive models, they tend to produce incoherent and repetitive results due to complex data distribution (e.g., acoustic and linguistic variations in speech). In this work, we introduce DiffNorm, a diffusion-based normalization strategy that simplifies data distributions for training NAT models. After training with a self-supervised noise estimation objective, DiffNorm constructs normalized target data by denoising synthetically corrupted speech features. Additionally, we propose to regularize NATs with classifier-free guidance, improving model robustness and translation quality by randomly dropping out source information during training. Our strategies result in a notable improvement of about $+7$ ASR-BLEU for English-Spanish (En-Es) translation and $+2$ ASR-BLEU for English-French (En-Fr) on the CVSS benchmark, while attaining over $14\times$ speedup for En-Es and $5 \times$ speedup for En-Fr translations compared to autoregressive baselines.

Main paper: 
 introduction: 
 Speech-to-speech translation (S2ST) systems are essential to bridge communication gaps and have wide application potential. We focus on non-autoregressive modeling for direct speech-to-speech translation, converting source speech to the target without intermediate text data. Such direct S2ST systems [25, 23, 22, 32, 31, 21, 20] can preserve non-linguistic information, avoid error propagation from cascaded systems [29, 34] (e.g., a combination of speech recognition and machine translation systems), and achieve faster inference speed.Non-autoregressive Transformers (NAT) [21, 20] has played a central role in current S2ST work [31, 32, 21] . NAT translates source waveforms into target speech units via parallel decoding, achieving performance comparable to or better than autoregressive models while greatly reducing inference time. This process is often referred to as speech-to-unit (S2UT) translation [31] . The predicted speech units are then converted to target waveforms via a unit vocoder in the unit-to-speech synthesis stage [32, 38] . However, NATs suffer from incoherent and repetitive generations, referred to as the multi-modality problem [12] . This issue stems from NAT's assumption of conditional independence during parallel decoding, worsened by the complex and multi-modal nature of training data distribution.In this work, we propose DIFFNORM, a self-supervised speech normalization strategy that alleviates the multi-modality problem of NAT models by simplifying the target distribution. Instead of distilling training data from an autoregressive model [10] or utilizing perturbed speech to train a normalizer [32, 21] , we rely on the denoising objective of Denoising Diffusion Probabilistic Models [15, DDPM] to normalize target speech units. DIFFNORM inject synthetic noise to speech features and use diffusion model to gradually recover the feature, obtaining a simplified and more consistent data distribution that obscures non-crucial details. As the denoising objective is learned in a self-supervised manner over latent speech representations, it eliminates the necessity for transcription data [32] or manually crafted perturbation functions [21] . As illustrated in Fig. 1 , applying DIFFNORM to the target data obtains normalized speech units that lead to better NAT training for speech-to-unit translation.Besides using DIFFNORM as a data-centric strategy to mitigate the multi-modality problem, we also propose a regularization strategy to enhance the NAT model's robustness and generalizability when facing complex data distribution (e.g., linguistic diversity and acoustic variation [21, 20] ). Inspired by classifier-free guidance [17, CG] , during training, we occasionally drop out source information and replace it with a "null" representation, compelling the models to generate coherent units without conditioning on the source data. During iterative parallel decoding of the NAT model, we obtain higher-quality translation by mixing conditional and unconditional generation. Ultimately, combining DIFFNORM and CG results in our top-performing state-of-the-art system, achieving approximately +7 and +2 ASR-BLEU increment for En-Es and En-Fr translation compared to previous non-autoregressive systems on the CVSS [24] benchmark.In conclusion, we alleviate the multi-modality problem by proposing (1) diffusion-based normalization and (2) regularization with classifier-free guidance. To the best of our knowledge, we are the first to adapt diffusion models and classifier-free guidance into speech-to-speech translation and NAT modeling. Our methods obtain notable improvement compared to previous systems and maintain fast inference speed inherent in non-autoregressive modeling, achieving speedups of 14× for En-Es and 5× for En-Fr compared to autoregressive baselines. ...Figure 1 : Overview of our proposed system. We first normalize the target speech units with the denoising process from the latent diffusion model. Then speech-to-unit (S2UT) model is trained to predict normalized units, which are converted into waveform from an off-the-shelf unit-vocoder.We aim to develop a direct (textless) speech-to-speech translation system that transduces a source speech x = (x 1 , • • • , x N ) into target speech. We follow Lee et al. [31] to reduce speech-to-speech translation into two sub-tasks: speech-to-unit translation and unit-to-speech synthesis. In this work, we focus on speech-to-unit translation and follow prior work [31, 32, 21] to use the same unit-to-speech component for a fair comparison. To generate speech units for the target language, we first extract speech fea-ture h = (h 1 , • • • , h M ) ∈ R M ×H ,where each feature has H dimensions. Subsequently, a K-means clustering model is trained on extracted features and used to generate speech units y = (y 1 , • • • , y M ) ∈ R M ×1 . Once the source speech x and target speech unit y are prepared, we train sequence-to-sequence models to translate from source speech into target units. In practice, we follow Lee et al. [32] to use a multilingual-HuBERT (mHuBERT) model to extract H = 768 dimensional speech features h. Then we use a 1000-cluster K-means model to predict speech units given the encoded features. 2 For speech-to-unit translation, we follow Huang et al. [21] to adopt Conditional Masked Language Modeling [10, CMLM], a kind of non-autoregressive transformer (NAT).To mitigate NAT models' multi-modality problem, we propose DIFFNORM (section §3), which denoises synthetically corrupted speech features to construct normalized speech units y norm . As shown in Fig. 1 , such normalized units are then used to train the S2UT (CMLM) model, which generates better-translated units for the unit vocoder to synthesize target speech.Additionally, we propose to incorporate classifier-free guidance [17, CG] , a widely adopted strategy for diffusion-based image generation, as a regularization method to improve the non-autoregressive speech-to-unit system (section §4). As shown in Fig. 3 , by forcing NAT models to unmask target speech units without conditioning on source information, the model becomes more robust, generating coherent speech units that result in higher-quality translations. During inference, we follow classifierfree guidance to mix conditional and unconditional generation for the NAT model's iterative decoding, further enhancing its translation quality. 
diffnorm: denoising diffusion models for speech normalization: 
 Previous normalization strategy [32, 21] on speech-to-unit translation relies on connectionist temporal classification (CTC) fine-tuning [4] using the HuBERT [19] model. For example, Lee et al. [32] rely on single-speaker data from VoxPopuli [58] to produce normalized (speaker-invariant) speech units for HuBERT-CTC fine-tuning. On the other hand, Huang et al. [21] generate acoustic-agnostic units by perturbing rhythm, pitch, and energy information with (manually) pre-defined transformation functions. We propose to normalize speech units using self-supervised denoising objectives from Denoising Diffusion Probabilistic Models [15, DDPM] . We believe DDPM is more suitable for speech normalization because: (1) It only requires monolingual speech data, without the need of transcriptions to create a text-to-unit model as in [32] . (2) It learns to denoise features in a high-dimensional space rather than relying on hand-designed perturbation as in [21] . DIFFNORM consists of a variational auto-encoder (VAE) and a diffusion model. Since vanilla VAE [27] and DDPM [15] are applied to image generation, we modify them to support token generation and incorporate multitasking objectives. The VAE model reduces the dimension of the speech feature, mapping feature h into lower dimensional latent z. The diffusion model (visualized in Fig. 2 ) is then trained to denoise on the latent representation space, aiming to recover the original latent z 0 from the standard Gaussian noise z T . 3In the subsequent sections, we begin by outlining the architecture of our VAE model ( §3.1). Then, we delve into how the diffusion model is trained using the latent variables encoded by the VAE ( §3.2).Since speech features encoded by mHuBERT have a high dimension (H = 768), we compress the feature into lower-dimension latents for the diffusion model. The VAE model consists of an encoder, a decoder, and a language modeling head. Following our problem formulation ( §2), we first prepare a sequence of target speech features h = (h 1 , • • • , h M ) ∈ R M ×H and their corresponding speech units y = (y 1 , • • • , y M ). Our VAE model's encoder will map the feature into lower dimension latent z = f (h; θ enc ) ∈ R M ×Z where Z < H is the pre-defined latent dimension.Then VAE model's decoder reconstructs the speech feature ĥ = f (z; θ dec ) and we apply the language modeling head to convert the reconstructed feature into a distribution over speech units' vocabulary v = f ( ĥ; θ lm ) ∈ R M ×V . Following prior work [21, 32] , we cluster speech features into V = 1000 units. The training objective is a weighted combination of reconstruction loss (L recon ), negative log-likelihood (NLL) loss (L nll ), and a Kullback-Leibler (KL) divergence term resulted from the Gaussian constraint [27] to regularize the representation space:EQUATIONwhere the reconstruction loss is defined as L recon ( ĥ, h) = ||h -ĥ|| 2 , and the NLL loss is computed as the cross-entropy between ground-truth units y and the predicted vocabulary distribution v:L nll (v, y) = - M i=1 y i log v i, where y i is the one-hot version of y i . Lastly, L kl can be solved analytically when we assume Gaussian distribution for both prior and posterior approximation of latent z (more details in [27] and Appendix B). Though the Gaussian constraint has a small weight λ 3 , we show in the ablation study ( §6.2) that regularizing latent distribution is critical for the diffusion model. For details on the architecture of our VAE model, we direct readers to Appendix B.Training Once the VAE model is trained, we encode speech feature as the first step feature z 0 = f (h; θ enc ) for the diffusion model. Diffusion models consists of a (1) gradually transforms z 0 into a standard Gaussian distribution, and a (2) reverse process that denoise and recovers the original feature z 0 . Following DDPM [15] , with a pre-defined noise scheduler (let β t ∈ (0, 1) be the scaling of noise variance, define α t = 1 -β t and denote ᾱt = t s=1 α t as the noise level for time t), the forward and reverse process can be written as:EQUATIONwhere mean µ θ (z t , t) and variance σ 2 are parameterized by trainable models, typically based on U-Net [44] or Transformer [57] . We follow DDPM to train models using the re-weighted noise estimation objective:L noise (θ, t) = E x0,ϵ,t ||ϵ -ϵ θ (z 0 , t)|| 2(3) where the network learns to predict the injected Gaussian noise ϵ from the forward process. Besides noise estimation, we also train the model with auxiliary reconstruction and NLL loss using VAE model's decoder and language modeling head. Our training procedure is summarized in Alg. 1.1: Input: speech feature h, speech units y. 2: Pre-compute: βt, αt, ᾱt, t ∈ [1, T ] 3: while not converged :4: z0 = f (h; θenc) 5: t ∼ U [1, T ], ϵ ∼ N (0, I) 6: zt = √ ᾱtz0 + √ 1 -ᾱtϵ 7: ε = ϵ θ (zt, t) 8: Lnoise = ||ϵ -ε|| 2 9: ẑ0 = (zt - √ 1 -ᾱt ε)/√ ᾱt 10: ĥ = f ( ẑ0; θ dec ), Lrecon = ||h -ĥ|| 2 11: v = f ( ĥ; θ lm ), Lnll = NLL(v, y) 12: Take gradient descent step with loss 13:L = γ1Lnoise + γ2Lrecon + γ3LnllAlgorithm 2 Normalized Units ConstructionEQUATION11: ĥ = f (z0; θ dec ) 12: ynorm = argmaxf ( ĥ; θ lm ) 13: return denoised units ynorm As shown in Alg. 1, we randomly sample the current timestep t ∈ [1, T ] and compute corresponding scheduling parameters β t , α t , ᾱt 4 . The training process involves the regular DDPM objective that injects Gaussian noise for the model ϵ θ to perform noise estimation (Alg. 1, line 6-8). Additionally, we generate the pseudo latent ẑ0 by reversing the noise injection process (line 9), which is then decoded by the VAE into speech features and units for reconstruction loss (line 10) and NLL loss (line 11). Finally, the objective is a weighted sum of noise estimation loss, reconstruction loss, and NLL loss:EQUATIONIn our analysis ( §6.2), we show that adding NLL and reconstruction loss is indeed helpful in improving the diffusion model's reconstruction quality. Lastly, to parameterize our diffusion model for noise estimation, we modify Diffusion Transformer [37, DiT] to suit our task. For details of our architecture and hyper-parameters, we refer readers to Appendix B.Inference For speech normalization, we choose a start time T ∈ [0, 200] that decides the amount of noise to inject for the diffusion model to recover. Given the start time T , we follow Denoising Diffusion Implicit Models [48, DDIM] sampler to reverse noised latent z T back to z 0 . Then, our VAE model converts z 0 back to speech units with its decoder and language modeling head. Our inference procedure is summarized in Alg. 2 and visualized in Fig. 2 . Note that since our diffusion model is used to normalize speech units as a data preprocessing strategy, inference speed is not a concern. Therefore we use step size ∆t = 1 for DDIM sampling throughout our experiments. However, the inference speed could be easily improved by setting a larger step size.4 Classifier-free guidance for non-autoregressive transformerIn §3, we proposed DIFFNORM to normalize speech units and obtained speech-unit pairs (x, y norm ) that benefit NAT training. In this section, we propose to adapt classifier-free guidance [17] to regularize NAT models (visualized in Fig. 3 ), further enhancing NAT-based S2UT model's translation quality.Training We largely adhere to standard CMLM training [21] but introduce a small dropout probability p drop = 0.15 for the encoder representations. Specifically, we parameterize the encoder and decoder of the CMLM as ϕ enc and ϕ dec , respectively. The encoder processes the source speech into a representation g = f (x; ϕ enc ). The decoder then predicts the vocabulary distribution v = f ( ŷ|g; ϕ dec ) from g and the randomly masked target speech units ŷ. The total amount of masked token is uniformed sampled: n ∼ U[1, M ]; subsequently, n of M tokens from the target units y are randomly masked to form noisy target units ŷ. CMLM also trains a length predictor that estimates the output length given input sequences and we refer readers to [21] for more details.Using the predicted distribution v, we compute the NLL loss against the ground truth units y at the masked positions to train the model. If dropout is applied, the decoder receives a "null" representation g ∅ -a randomly-initialized learnable vector-forcing it to rely solely on the target information to unmask units. For more details, we refer readers to Appendix C and previous paper [10, 21] .Inference CMLM generates tokens through iterative unmasking. Given the input sequence x, CMLM first predicts the length of output sequence M and initialize a sequence ofM masked tokens ŷ0 = ([mask] 1 , • • • , [mask] M ).Given the total number of iterations T5 and current iteration t ∈ [1, T ], CMLM decodes all tokens y i , i ∈ [1, M ] in parallel using their probabilities:EQUATIONFrom the generated tokens ŷt = (y t 1 , • • • , y t M ), we replace k = M • T -tT tokens with the lowest probabilities p t i with mask tokens. This re-masked sequence ŷt is then inputted into the CMLM for further decoding iterations. This process continues until the final step t = T , at which point no tokens are re-masked. With our proposed regularization from classifier-free guidance, we compute two vocabulary distribution in each iteration step t:EQUATIONwhere p t orig is the same as vanilla CMLM and p t uncond is the unconditional probability obtained with the "null" representation. Then we select and re-mask tokens using the adjusted probabilityEQUATION)where the hyper-parameter ω is used to control the degree to push away from the unconditional distribution. In the special case of ω = 0, only conditional distribution is used during iterative decoding, resulting in the same generation process as standard CMLM. Through our intensive analysis (Appendix F), we determine that a relatively small ω (e.g., ω = 0.5) gives the best result, especially when the number of decoding iterations is large (i.e., T > 10). Notably, even when ω = 0 -which corresponds to the traditional CMLM decoding method -CMLMs regularized with classifier-free guidance during training can consistently outperform their standard counterparts. This observation further validates the effectiveness of our proposed regularization. Dataset We perform experiments using the established CVSS-C datasets [24] , which are created from COVOST2 by employing advanced text-tospeech models to synthesize translation texts into speech [59] . CVSS-C comprises aligned speech in multiple languages along with their respective transcriptions. Our methods are evaluated on two language pairs: English-Spanish (En-Es) and English-French (En-Fr), with detailed data statistics provided in Table 1 . As our focus is on direct speech-to-speech translation, transcriptions are solely utilized for evaluation purposes. Utilizing the speech data from CVSS, we preprocess the target speech to generate speech units using the mHuBERT and K-means model, as described in our problem formulation ( §2).Evaluation To evaluate the performance of various speech-to-speech translation systems, we adopt the standard methodology established in previous studies [32, 31, 21] to compute the ASR-BLEU score. Firstly, our speech-to-unit translation system generates speech units based on the input speech, which are then transformed into speech waveforms using a unit-vocoder. The unit-vocoder is built upon the HifiGAN architecture [28] with customized objectives, detailed in Appendix D. Once we have the waveforms, we employ an ASR model to transcribe them and calculate the BLEU score against the reference transcriptions. This ASR model is fine-tuned based on the WAV2VEC2.0 [5] model with the CTC objective. We direct readers to the original paper [7] for more details. Both the unit-vocoder and ASR models are sourced from an off-the-shelf repository, ensuring consistency in evaluation methodology with previous studies [32, 21] . 6Normalized dataset construction We adhere to the procedure outlined in Alg. 2 to generate normalized speech units. By manipulating start time T , we control the level of noise in the latent z T = √ ᾱt z 0 + √ 1 -ᾱt ϵ for the diffusion model, balancing between the reconstruction quality and normalization effect. We explore different levels of noise injection and choose T = 100 for En-Es and T = 120 for En-Fr (further details in §6.1). Hence, we adopt these settings to construct our normalized dataset, CVSS-NORM.In this section, we provide brief descriptions of evaluated speech-to-unit models. For autoregressive models, we evaluate the Transformer model trained following Lee et al. [31] . We also assess the Conformer model trained similar to the Transformer, but with its encoder replaced by a Conformer-Encoder [14] . Lastly, the Norm Transformer shares the same architecture as the Transformer, but it is trained on normalized speech units that are speaker-invariant. The normalized dataset is constructed following the strategy proposed by Lee et al. [32] .For non-autoregressive systems, we train the Conditional Masked Language Model (CMLM) following Huang et al. [21] , using a Conformer-based encoder and a Transformer decoder. The CMLM + Bilateral Perturbation (BiP) system retains the same architecture as CMLM but is trained on normalized speech units constructed with BiP [21] .For our improved systems, we train CMLM + DIFFNORM, which shares the same architecture as CMLM but is trained on CVSS-NORM, the normalized dataset obtained through DIFFNORM. Additionally, we train the CMLM + CG model that incorporates classifier-free guidance introduced in §4. Finally, the CMLM + DIFFNORM + CG system uses the architecture of CMLM + CG and is trained on our normalized dataset CVSS-NORM.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Justification: There is no theoretical proof involved in this paper. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experimental result reproducibility: 
 Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: We provide details of our implementation (architecture and training hyper-parameters) in Appendix B and Appendix C. We will also release our code and the dataset we used (CVSS) is publicly available at https://github.com/ google-research-datasets/cvss. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
results: 
 Table 2 summarizes the (ASR-BLEU) performances of various S2UT systems. Note that, for nonautoregressive methods, Table 2 shows their results obtained with 15 decoding iterations. For more details on the inference speedup achieved through non-autoregressive modeling, we plot Fig. 4 to provide details on the "quality vs. latency" trade-off. Observations from [21] . ‡ We use w = 0.5 for CG. Our NAT models achieve superior translation quality while maintaining their fast inference speed.Decoding speed In Fig. 4 , we illustrate the "quality-latency" trade-off of various non-autoregressive speech-to-unit systems. Quality is measured using ASR-BLEU, while latency is determined by the relative speedup over the autoregressive system, calculated as "generated units/second". For instance, the first marker on the line plot represents 15 decoding iterations, resulting in a speedup of 5.34× compared to the autoregressive baseline. Additionally, we include the performance of the Conformer-based autoregressive model as a horizontal dashed line. We observe that our improved system consistently outperforms the baseline CMLM model [21] and achieves better performance than the autoregressive model with more than 14× speedup for En-Es and 5× speedup for En-Fr.6 AnalysisQuestion: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
effect of synthetic noise injection: 
 In this section, we explore the effects of varying degrees of noise injection on DIFFNORM. Recall (from Alg. 2) that synthetic noise is injected into the latent representation as z T = √ ᾱT z 0 + √ 1 -ᾱT ϵ. Hence, adjusting the start time T allows us to control the degree of noise injection.We perturb the degree of noise injection by varying T , and assess the reconstruction quality of normalized speech units and downstream performance of CMLM models trained with different normalized units. For reconstruction quality, we measure the accuracy (Acc-Rec) of reconstructed units and ASR-BLEU (BL-Rec) of synthesized speech when using original units/speech as the reference. For downstream performance, we report the downstream ASR-BLEU of translated speech produced by CMLM models trained with the particular noise setup (BL-Dn). 7Results Shown in Table 5 , increasing T leads to more noise and a decline in reconstruction quality, which aligns with expectations that higher noise levels pose greater challenges for the diffusion model in restoring original features accurately. Interestingly, we find from Table 3 that when T is too small or large (e.g., T = 50 or T = 150), the normalized units do not result in an ideal downstream system. We hypothesize that there is barely any normalization effect when T is too small; and when T is too large, the reconstructed units are no longer semantically correct. This phenomenon is visualized in Fig. 5 , where we plot the log-mel spectrograms of the reconstructed speech. As more noise is introduced (larger T ), the reconstructed speech becomes smoother (e.g., portions of blank speech are filled in by diffusion), exhibiting a more pronounced deviation from the original speech. T = 10 to T = 100. This ablation study underscores that the optimal choice of T is affected by the dataset quality. Table 5 : Accuray of reconstructed speech units. KL: when applied, the latent space is regularized to be Gaussian [27] . Multitask: when not applied, the latent diffusion model is trained only with L noise .DIFFNORM requires an auto-encoder to map speech features into lower-dimension latents and then train denoising diffusion models using the encoded latents. In this section, we perform ablation experiments to investigate the effect of (1) latent dimension (2) Gaussian constraints (3) multitask objective for diffusion models. Firstly, to investigate the effect of latent dimension, we train auto-encoders that encode speech features into 16, 32, and 128 dimensions.As shown in Table 5 , it comes as no surprise that a larger latent dimension yields superior reconstruction results, evidenced by higher accuracy across systems trained with varying objectives. Next, we turn our attention to the importance of applying Gaussian constraints to the auto-encoder's latent representation space. Our results (comparing with and without KL) reveal that incorporating Gaussian constraints is crucial. Latent spaces not regularized by L kl lead to significantly poorer performance. Finally, we explore the effectiveness of our proposed multitasking objective for the diffusion model by training another vanilla DDPM solely with L noise (no Multitask). We find that the diffusion model's reconstruction capability indeed improves when employing the multitasking objective, particularly when denoising from a more noisy latent representation (e.g., T = 150). Through our ablation analysis, we identify the optimal setup for our DiffNorm model, which involves (1) mapping speech features to 128 dimensions, (2) regularizing latent space by Gaussian constraints, and (3) utilizing the multitask objective for diffusion training.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We provide all our implementation detail in Appendix B and Appendix C.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
related work: 
 Direct speech-to-speech translation (S2ST) Direct S2ST aims to directly translate speech into another language without cascaded systems that rely on transcriptions or translations. Translatotron [25] and Translatotron 2 [23] are among the first systems for direct S2ST, which uses sequence-tosequence model to map source speech into spectrograms. Then, a spectrogram decoder is used to synthesize the target language's speech. Instead of transducing speech into spectrogram, Tjandra et al. [53] , Zhang et al. [62] utilize Vector-Quantized Variational Auto-Encoder (VQ-VAE) [56] to discretize target speech and convert speech-to-speech translation into a speech-to-unit task. More recently, Lee et al. [31] improved the speech-to-unit models by obtaining such units with a k-means clustering model trained on self-supervised representation from HuBERT [19] . To convert units back to speech, Lee et al. [31] follows Polyak et al. [38] to train a unit-vocoder based on HifiGAN [2] .Speech normalization Speech representation are typically extracted from pre-trained encoders [19, 3, 40, 5] and can be compressed or adapted in different speech-to-speech/text tasks [61, 63, 50, 52, 51] . Inspired by previous work on speech enhancement [54, 1], Lee et al. [32] propose to normalize speech by synthesizing speaker-invariant waveforms through text-to-speech (TTS) systems [60, 46, 42, 41] . Huang et al. [21] propose to normalize speech with Bilateral Perturbation that focus on the rhythm, pitch, and energy information. Different from previous normalization methods that requires transcription [32] or manually designed perturbation [21] , our DIFFNORM strategy leverages diffusion models. Diffusion models [15, 48, 49] have achieved remarkable generative ability to produce high-quality images [8, 16, 43, 37, 45] and audio [33, 39, 47] . Using self-supervised denoising objectives [15] , we train effective denoising models capable of normalizing speech for training better speech-to-unit systems.Non-autoregressive speech-to-speech translation For sequence-to-sequence modeling, autoregressive [18, 6, 57] and non-autoregressive [12, 13, 10, 20] models have been widely explored. Lee et al. [31] reduce speech-to-speech into speech-to-unit task and follow the widely-used modeling strategy, Transformer [57] , to predict speech units. Later, non-autoregressive models [21, 9] have been explored and Huang et al. [21] is the first to use a non-autoregressive transformer model, Conditional Masked Language Model [10, CMLM], for speech-to-unit task. Fang et al. [9] , on the other hand, adopts Directed Acyclic Transformer [20] for speech-to-unit translation.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We follow the standard evaluation setting, using the same dev and test set as prior work, and evaluate our method in two language directions (English-Spanish and English-French). Our results are consistent over languages, and our ablation studies that introduce additional experiments also confirm our method's effectiveness. Lastly, we improve upon the baseline with a notable +7 BLEU increment for En-Es and +2 BLEU increment for En-Fr over a test set that has more than 10,000 samples. With the evidence above, we believe our improvement is solid and alleviates the need to re-run experiments for significance tests. We believe our practice is standard and follows from previous work on speech-to-speech translation.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
conclusion: 
 We improve speech-to-unit translation system through (1) corpus distillation by constructing normalized speech units with DIFFNORM and (2) regularization with classifier-free guidance. Our improved non-autoregressive (NAR) models, greatly outperform previous NAR models, yielding an increase of approximately +7 and +2 BLEU points for En-Es and En-Fr translation, respectively. Notably, DIFFNORM and classifier-free guidance maintain the inference speed advantages inherent in NAR models. Consequently, our approach obtains better performance to autoregressive baselines while achieving over 14× speedup for En-Es and 5× speedup for En-Fr translations.From §5.2 we demonstrate the effectiveness of DiffNorm and classifier-free guidance by evaluating ASR-BLEU scores. In this section, we provide example transcriptions of generated Spanish and French speech in Table 6 and Table 7 . Compared to the vanilla CMLM model that has erroneous or incomplete translation, our method results in higher-quality translation. to the original dimension. Additionally, we apply a Transformer [57] encoder on top of the decoded feature to further enhance model capacity. The Transformer-encoder has the same configuration as the vanilla Transformer-base model (6 layers, 8 attention heads, etc.). Lastly, to decode speech units from the feature, we use a language modeling head which is parameterized by a feedforward network that converts feature dimension (H = 768) into vocabulary size (V = 1000).Training: To train the VAE model, as described in §3.1, we use a combination of three objectivesEQUATION)where λ 1 = 100, λ 2 = 1, λ 3 = 0.001. Now we provide more details about L kl : in practice, we follow [27] to model the latent z by estimating its mean µ and variance σ using the encoder. With the re-parameterization trick, the latent is sampled as z = µ + σ • ϵ where ϵ ∼ N (0, I). By constraining the mean and variance to follow a Gaussian prior, Kingma and Welling [27] showed thatEQUATIONWe implement our VAE model on Fairseq [36] . For optimization, we use the Adam [26] optimizer with betas (0.9, 0.98) and we apply gradient clipping by setting -clip-norm=2.0. During training, we apply dropout with a probability of 0.1. We train the VAE model using a learning rate of 5e-4 with distributed data-parallel (DDP) on 4 A100 GPUs, where we set the maximum batch token to be 15000.Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: As noted in Appendix B and Appendix C, we use 4 A100 GPUs with Distributed Data-Parallel setting throughout our experiments.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: We follow the Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: See Appendix E for our discussions.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our work focuses on direct speech-to-speech translation and we do not believe our model has a high risk for misuse as it does not support voice cloning or other high-risk applications. 
 
 Appendix: Architecture: We modified DiT [37] to design a Transformer-based architecture for noise estimation. Since our input latent is a sequence of encoded speech feature z ∈ R M ×Z , we first apply a 1D Convolution to convert the latent to our model's hidden dimension (set to 512). Then we also encode the timestep with a learnable Sinusoidal Positional Embedding [57] to obtain the time embedding.Subsequently, we feed the latent and time embedding to our modified WaveNet Block, which applies an affine transformation of the latent using the time embedding, similar to the "Scale and Shift" operation in DiT's adaptive layer norm (adaLN) block. For our diffusion model, we use 4 stacks of such modified WaveNet Block, each containing 8 layers. The output feature from WaveNet is then passed through a Transformer-Encoder (12 layers with 8 attention heads) to further enhance model capacity. Lastly, a projection layer parameterized by the feedforward network is used to compress the transformed feature to the latent dimension, which is then used for noise estimation in equation (3) .Training: As described in §3.2, we train the diffusion model with a multitask objective:EQUATION)and we empirically select γ 1 = 1, γ 2 = 0.25, γ 3 = 0.005. For optimization, our latent diffusion model has the same setting as our VAE model (with Adam optimizer, clip-norm equals 2.0, and dropout with 0.1 probability). We train the diffusion model using a learning rate of 1e-4 with DDP on 4 A100 GPUs, where we set the maximum batch token to be 12000. The model is warmup-ed by 10000 steps.Non-autoregressive Transformers [12, 13, 10, inter alia.] is a family of models that transduce sequences. In this work, we follow the formalization from the widely-used NAT: Conditional Masked Language Model [10, CMLM] . CMLM consists of an encoder ϕ enc that represents the input as high-dimensional features and a decoder ϕ dec that generates tokens conditioning on the encoded features. Different from auto-regressive Transformers [57] , CMLM's decoder does not apply causal masking and is trained with an unmasking objective.CMLM Training Following formulation from §2, given the input speech x = (x 1 , • • • , x N ) and target speech units y = (y 1 , • • • , y M ), CMLM first mask the target units into ŷ. The total amount of masked token is uniformed sampled: n ∼ U[1, M ]; subsequently, n of M tokens from the target units y are randomly masked to form noisy target units ŷ. Then, the decoder generates a distribution over vocabulary conditioning on the encoder representation and noisy target units v = f ( ŷ|g; ϕ dec ) where g = f (x; ϕ enc ) is the encoder representation. Cross-entropy loss is then computed for the masked positions to update model parameters. CMLM also trains a length predictor that estimates the output length given input sequences and we refer readers to [21] for more details.CMLM Mask-Predict: For CMLM inference, an iterative decoding scheme is used through unmasking speech units and re-masking low-confident positions. Given the input sequence x, CMLM first predicts the length of output sequence M and initialize a sequence of M masked tokens ŷ0 = ([mask] 1 , • • • , [mask] M ).Given the total number of iterations T and current iteration t ∈ [1, T ], CMLM decodes tokens across all positions and also computes their log-probabilities:EQUATIONFrom the generated tokens ŷt = (y t 1 , • • • , y t M ), we replace k = M • T -tT tokens with the lowest probabilities p t i with mask tokens. This re-masked sequence ŷt is then inputted into the CMLM for further decoding iterations. This process continues until the final step t = T , at which point no tokens are re-masked.We follow the same architecture and implementation from Huang et al. [21] , where a Conformer-based encoder is used to obtain representation from the source speech and a Transformer-decoder (with no causal masking) is used to generate units. The Conformer-encoder has 12 layers with 512 hidden dimensions and 9 attention heads. The encoder also contains a CNN-based sub-sampler that has an effective stride size of 320 (i.e., it reduces the length of speech input by 320×). The Transformerdecoder has 6 layers with 8 attention heads and uses a fixed positional embedding for speech units. When classifier-free guidance is used, we randomly dropout encoding from the Conformer model with a probability of 15% as described in §4.We train the model with a learning rate of 5e-4, using 10000 warmup steps We apply Adam optimizer with betas (0.9, 0.98) and we clip the gradient by setting -clip-norm=10. We apply dropout with 0.1 probability and we use label smoothing of ratio 0.2 when computing the NLL loss. We train the model with DDP on 4 A100 GPUs, with a maximum batch tokens of 40,000.We follow prior work [32, 38, 41] to convert predicted units (from S2UT model) into speech waveforms. Specifically, given units y, we use the HifiGAN-based unit-vocoder from [38] to synthesize speech. The unit-vocoder is trained using a generator G and discriminator network D. During training, the generator is updated with reconstruction loss based on the mel-spectrogram of the true and synthesized waveforms. Additionally, adversarial loss and feature-matching loss are added to enhance the fidelity of generated speech. To train the discriminator, Polyak et al. [38] follows the minmax objective from GAN [11] . After training, the discriminator is discarded and only the generator is used to synthesize waveforms.We follow [32] to use a unit-vocoder that is also trained with a duration predictor so that it can synthesize speech with reduced speech units. For more details, we refer readers to the discussion from the original paper [32, 41] , as well as the open-sourced repository: github.com/facebookresearch/ fairseq/examples/speech_to_speech E Limitations and Broader Impacts Limitations: The major limitation of the work is the data pre-processing cost from DIFFNORM, as our method requires inferencing diffusion model on all training samples to obtain better speech units. With the CVSS benchmark, both En-Es and En-Fr have less than 1M pairs, which can be processed easily with our resource (4 A100 GPUs). However, when the dataset scales up, the corpus distillation cost could be large.Broader Impacts: Our system helps bridge the communication gap between users of different languages. However, as our system involves speech synthesis, it also has a risk of being used to mimic a particular speaker.In this section, we compare non-autoregressive transformers trained with and without classifier-free guidance (CG). We train CMLM models with classifier-free guidance and evaluate them with 5, 10, 15 iterations of decoding. From Table 8 , we observe that CG improves the quality of translation whether the model is trained on original or normalized speech units. We find CG brings more improvement on the original units than the normalized units. This happens because normalized speech units are more conformed and already result in a large improvement in their translation quality, making the regularization effect from CG less obvious. Nevertheless, the best-performing system is achieved with both DIFFNORM and CG.Comparing different hyperparameters w, we find a small value like w = 0.5 or w = 1 brings the most improvement empirically, and such improvements are more noticeable when the number of decoding iterations is larger. For example, comparing the results under 5 and 15 iterations, we find w = 0 gives better results when the number of iterations is small while w = 0.5 obtains the best performance with 15 iterations.En-Es En-Fr • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Code available at: https://github.com/steventan0110/DiffNorm. 38th Conference on Neural Information Processing Systems (NeurIPS2024).We take the mHuBERT and K-means models off-the-shelf to produce units consistent with prior work. See github.com/facebookresearch/fairseq/tree/main/examples/speech_to_speechThe subscript z0 is added to indicate the time step for diffusion process and z0 and z are equivalent.We follow the cosine schedule proposed by Nichol and Dhariwal[35] with a maximum timestep of T = 200T was used in §3 to denote total timesteps for diffusion models. We abuse the notation and re-use it here to represent the number of decoding iterations for CMLM.Repository available at:github.com/facebookresearch/fairseq/examples/speech_to_speech/ asr_bleu.ASR-BLEU is computed by synthesizing speech from the reconstructed units and transcribing it using the ASR model, as detailed in §5.1. 


Decision: 2 weak accepts, 1 borderline accept, 1 accept
strengths: interesting idea, wide applicability, solid improvements
weankesses: experiments can be extended to other tasks, novelty of paper can be improved.

still given the flaws i think this paper is wrong enough to accept.

Reviews: review: paper_summary: paper_summary: The authors introduce a process aiming to simplify the target distribution of speech-to-speech translation. This process uses a VAE model to map features to a latent space, followed by a diffusion model to normalize the features in the latent space. The authors use the generated dataset to train a non-autoregressive CMLM model on the CVSS-C dataset to validate its effectiveness.
strength_weakness: strength_weakness: 1.  I am confused by the rationale behind using a diffusion model to normalize the speech representation. The authors first add noise to the speech representation (Forward Process) and then remove the noise (Backward Process). This design seems awkward to me. Why do you think this process of adding and then removing noise can help with normalization?
2. Using a Variational Autoencoder to map features to a latent space seems contradictory to the motivation of reducing data multimodality. The VAE provides an indefinite mapping, which may hinder efforts to reduce multimodality. Additionally, $z_0 = f(h; \theta_{enc})$ is not correct; the VAE provides a distribution over the latent space. A more suitable expression would be $z_0 \sim p(z; f(h; \theta_{enc}))$.
3. The experiments are only conducted on synthesised dataset. As the paper's contribution is to alleviate the multi-modality problem in data, conducting experiments on a real speech-to-speech dataset, like [1], is much better to support the major claims.
4. The baseline results (Conformer in EnEs) seem quite low compared to the reported results in the literatre.

[1] Lee et al., Textless speech-to-speech translation on real data.
questions: questions: 1. A simple CG method in En-Es can bring an improvement about 4.5 BLEU (Table 2, Line 7), any reasons behind it?
2. In Table 4, Acc-Rec, BL-Rec, and BL-Dn seem highly correlated when applying your method. BL-Dn performs well with minor perturbation and vice versa. However, when there is no perturbation, BL-Dn degrades dramatically. What is the reason for this? Is there a borderline value of $T$ that separates these two regions? I think this exploration is important for understanding the effectiveness of using the diffusion process to normalize speech.
limitations: limitations: The authors have addressed the limitations in Appendix E.
paper_summary: paper_summary: This paper proposes DiffNorm, a diffusion-based self-supervised method for speech data normalization, aiming to alleviate multimodal problem in non-autoregressive speech-to-speech translation (NAT). DiffNorm consists of a VAE to reconstruct the speech feature and a diffusion model to add and remove noise of latent vector. Experiment shows that DiffNorm significantly improve NAT translation quality compared to baselines.
strength_weakness: strength_weakness: 1. The multi-modal problem has two aspects: semantic and acoustic. DiffNorm seems able to reduce acoustic modalities. Unclear if DiffNorm can also do that on semantic modalities, i.e., multiple feasible translations for the same source input.
2. Classifier-free guidance combined with DiffNorm leads to worse performance than DiffNorm alone in Figure 4 and the authors ignore it in the text. Elaboration is needed here.
3. Lack baseline comparison with several latest S2ST models like TransSentence [1], PolyVoice [2], SeamlessM4T and etc.

[1] TranSentence: speech-to-speech Translation via Language-Agnostic Sentence-Level Speech Encoding without Language-Parallel Data.
[2] PolyVoice: Language Models for Speech to Speech Translation
[3] SeamlessM4T—Massively Multilingual & Multimodal Machine Translation
questions: questions: 1. Issues listed in weakness.
2. What is the role of VAE here? Can we drop it and directly use a diffusion model here for denoising? 
3. Table 3 mentions larger dimension of latent vector brings better performance, why not use the original number of dimensions instead of compressing it?
4. Why improvement on En-Fr is not that significant compared to En-Es? How does language direction impact the performance?
5. Is there way to visualize the noise added on the latent vectors? Like what does it mean for the audio?
limitations: limitations: Besides what I have already mentioned in the weakness, experiments in the paper are only conducted on En-X, but not reverse. Also, it would be interesting to see how DiffNorm works on non-European languages like Chinese and Japanese.
paper_summary: paper_summary: This work builds on TRANSPEECH (Huang et al., 2023) by applying the diffusion method to reduce noise, thereby normalizing speech units for further generation. The authors further use classifier-free guidance to enhance non-autoregressive generation. They conduct experiments on CVSS En-Fr and En-Es datasets, comparing their methods with the baseline. Both proposed methods show improvements.
strength_weakness: strength_weakness: 1. The paper overstates its contributions. The authors claim to be the first to apply diffusion in the speech-to-speech translation task. However, they merely use it to generate auxiliary training targets and still follow the S2UT strategy (Lee et al., 2022b). This diminishes the novelty of the paper.
2. Although the work is based on TRANSPEECH, the authors compare only two of the three translation directions. One of the core contributions is applying the diffusion method to normalize speech units, yet the method shows minimal improvement (only 0.3 BLEU) compared to BiP (Bilateral Perturbation) in the En-Fr task. Given the method significantly outperforms others in the En-De task, this result is perplexing.
3. For a minor suggestion, in Tables 3 and 4, ensure consistent significant figures. Additionally, it would be more appropriate to place Table 3 before Table 4.
questions: questions: 1. How does the performance fare on the CVSS Fr-En task?
2. In my opinion, the goal of normalizing the unit is to disentangle linguistic information from noisy speech features. If you use DDPM for this process, the target $z_0$, which is the output of the VAE encoder, should be the linguistic feature. How do you ensure that $z_0$ represents the desired linguistic information?
limitations: limitations: The authors appropriately state the limitations and broader impacts of this work.

Comments: title: Invitation for Comments and Clarifications
Invitation for Comments and Clarifications
Thanks for the author response
Rebuttal by Authors (Cont'd)
Rebuttal by Authors (Cont'd)
General Response to All Reviewers
comment: Dear Reviewer DRdm,

We greatly value your feedback and have provided clarifiications to your questions and additional experiments on Fr-En task. To ensure that we have properly addressed your concerns, we would greatly appreciate if you could review our responses and provide any further comments. We are looking forward to engaging with you before the discussion period ends.

Thank you for your time and consideration.
Dear Reviewer g4Z7,

We greatly value your feedback and have provided clarifiications and additional experiments and analysis. To ensure that we have properly addressed your concerns, we would greatly appreciate if you could review our responses and provide any further comments. We are looking forward to engaging with you before the discussion period ends.

Thank you for your time and consideration.
I will keep my score.
This comment follows from **Rebuttal by Authors**.

> Why improvement on En-Fr is not that significant compared to En-Es? How does language direction impact the performance?
> 

Our method has demonstrated effective performance on both language pairs, with a notable 2 BLEU point improvement for En-Fr, which is considered substantial. However, the more dramatic improvement observed in the En-Es pair can likely be attributed to specific characteristics of the dataset. Three key factors may contribute to this significant enhancement:

1. **Dataset Size**: The En-Es dataset is smaller, which means that the impact of normalization on the dataset is more pronounced, leading to more noticeable improvements.
2. **Speech Length**: Spanish speech is, on average, longer than French speech, as indicated in Table 1. Longer sequences could be more susceptible to acoustic multimodality, which our normalization method effectively mitigates.
3. **Background Noise**: The Spanish speeches from the CommonVoice dataset contain background noise. Our proposed normalization technique is capable of removing such noise, thereby enhancing the clarity and quality of the speech data.


> Is there way to visualize the noise added on the latent vectors? Like what does it mean for the audio?
> 

For visualization, please refer to Figure 5, which displays the log-mel spectrogram of the reconstructed speech. The extent of corruption in the speech varies based on the level of noise injection. As the noise injection level changes, the speech feature can be quantized into speech units that produce sounds ranging from completely unrecognizable noise to those closely resembling the original audio.

> experiments in the paper are only conducted on En-X, but not reverse. Also, it would be interesting to see how DiffNorm works on non-European languages like Chinese and Japanese.
> 

To address the concern regarding the X-En translation direction, we have conducted additional experiments for the Fr-En pair. We invite you to refer to our general response to all reviewers, where the results are detailed in the accompanying table. We observe consistent improvements in the Fr-En direction with DiffNorm, which we hope will further convince reviewers of the effectiveness of our proposed strategy.
We agree that extending our method to non-European languages would be intriguing, and our self-supervised approach should be readily applicable to monolingual Chinese or Japanese speech features. However, we are currently unaware of a suitable speech-to-speech translation (S2ST) dataset for English-Chinese or English-Japanese pairs. If you could point us to such datasets, we would be eager to include them in our future studies!

------

References:

[1] Lee et al., (2022). Textless speech-to-speech translation on real data.

[2] Huang et al., (2023). TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation

[3] Ghazvininejad et al., (2019). Mask-Predict: Parallel Decoding of Conditional Masked Language Models

[4] Gu et al., (2018). Non-Autoregressive Neural Machine Translation
This comment follows from **Rebuttal by Authors**.

> A simple CG method in En-Es can bring an improvement about 4.5 BLEU (Table 2, Line 7), any reasons behind it?
> 

We are also intrigued by the significant improvement from CG on the En-Es dataset. We suspect this may be due to two factors: (1) the relatively small size of the En-Es dataset, which contains fewer than 80k data points for training, and (2) the longer sequence lengths of Spanish speech, averaging 256 tokens in the training set and 308 in the test set, which worsen the acoustic multimodality issue. CG operates by constraining the model to learn exclusively from the distribution of the target (Spanish) speech units, leading to a more consistent distribution for translation during inference. **This is particularly beneficial for longer target sequences.**

To further support our claim, we recently extended our exploration of CG to English-German translation using the WMT14 dataset. Following the same settings as the CMLM model [5] and incorporating our proposed CG for the MT task, we observed that while the CMLM+CG model does not show significant improvement using the full WMT14 dataset, it does enhance performance by approximately 1 BLEU point when the training data is filtered to only include sequences longer than 60 tokens and the test data to only include sequences longer than 30 tokens, as demonstrated below:

| Model | T=5  | T=10 | T=15  |
|-------------------|------|------|-------|
|          CMLM        | 19   | 20.2 | 20.5  |
|               CMLM+CG    | 19.8 | 20.8 | 21.04 |

For speech-to-speech translation that has much longer unit length, the effect of CG will be more noticeable, and therefore achieving large improvement over the baseline.

> ….when there is no perturbation, BL-Dn degrades dramatically. What is the reason for this? Is there a borderline value of 𝑇 that separates these two regions?
> 

We believe it is dataset-dependent to judge whether there is a borderline value of T that has drastic improvement. We performed more ablation studies using T=10 and T=30 as start time for noise injection and the result is shown below:

| Start Time | Acc-Rec | BL-Dn |
| --- | --- | --- |
| T=10 | 93.8 | 15.98 |
| T=30 | 91.6 | 17.29 |

As indicated in the result table, with a very small T=10, the reconstruction accuracy is approximately 0.94, leading to a downstream ASR-BLEU score of 15.98. This represents a significant improvement compared to the baseline CMLM result of 12.58. The accuracy of 0.94 suggests that around 6% of tokens differ after reconstruction, which is likely due to the regularization effect of the VAE model, as T is too small for the diffusion model to cause reconstruction errors. This implies that the Spanish speech dataset may contain significant acoustic variations and noisy information, which the VAE model can already partially remove. With a larger T, the diffusion model further cleans up the representation, resulting in even higher downstream results (as observed from T=10 to T=100, where there is an increase in ASR-BLEU score).

-----
References:

[1] Lee et al., (2022). Textless speech-to-speech translation on real data.

[2] Huang et al., (2023). TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation

[3]Chen et al., (2024). Robust Classification via a Single Diffusion Model

[4] Duquenn et al., (2021). Multimodal and Multilingual Embeddings for Large-Scale Speech Mining

[5] Ghazvininejad et al., (2019). Mask-Predict: Parallel Decoding of Conditional Masked Language Models

[6] Richter et al., (2023). Speech Enhancement and Dereverberation with Diffusion-based Generative Models
Thank you to all reviewers for their valuable feedback and for recognizing the innovation and effectiveness of our work. In response to reviewers AuR5 and DRdm, we conducted further experiments on the French-English translation direction to validate our method's effectiveness. The results are as follows:

| Model            | ASR-BLEU |
|------------------|----------|
| Transformer      | 15.44    |
| Norm Transformer | 15.81    |
| Conformer        | 18.02    |
| CMLM             | 16.95    |
| CMLM + BiP       | 18.03    |
| CMLM + DiffNorm  | 19.53    |

These results are consistent with our findings for En→Es and En→Fr, where the DiffNorm dataset showed a 2.58 improvement in ASR-BLEU score compared to the CMLM baseline. Due to limited computational resources and time, we have not yet conducted CG-related experiments, but we will include these results in the future. Additionally, we plan to release our code publicly, addressing suggestions from reviewers g4Z7 and DRdm, to facilitate its use with other languages and larger datasets.