DiffNorm: Self-Supervised Normalization for Non-autoregressive Speech-to-speech Translation
Authors: Weiting Tan
Jingyu Zhang
Lingfeng Shen
Daniel Khashabi
Philipp Koehn
Venue: NeurIPS 2024 poster (2024)
Keywords: speech-to-speech translation
non-autoregressive modeling
diffusion models
Primary area: speech_and_audio

Abstract: Non-autoregressive Transformers (NATs) are recently applied in direct speech-to-speech translation systems, which convert speech across different languages without intermediate text data. Although NATs generate high-quality outputs and offer faster inference than autoregressive models, they tend to produce incoherent and repetitive results due to complex data distribution (e.g., acoustic and linguistic variations in speech). In this work, we introduce DiffNorm, a diffusion-based normalization strategy that simplifies data distributions for training NAT models. After training with a self-supervised noise estimation objective, DiffNorm constructs normalized target data by denoising synthetically corrupted speech features. Additionally, we propose to regularize NATs with classifier-free guidance, improving model robustness and translation quality by randomly dropping out source information during training. Our strategies result in a notable improvement of about $+7$ ASR-BLEU for English-Spanish (En-Es) translation and $+2$ ASR-BLEU for English-French (En-Fr) on the CVSS benchmark, while attaining over $14\times$ speedup for En-Es and $5 \times$ speedup for En-Fr translations compared to autoregressive baselines.

DIFFNORM: Self-Supervised Normalization for
Non-autoregressive -peechh Translation
Weiting Tan Jingyu Zhang
Lingfeng Shen Daniel Khashabi Philipp Koehn
Department of Computer Science Johns Hopkins University
{wtan12, jzhan237, 1shen30, danielk, phi}ojhu.ed. edu
Abstract
Non-autoregressive Transformers (NATs) recently applied in direct speech-to- speech translation systems, which convert speech across different languages without intermediate text data. Although NATs generate high-quality outputs and offer faster inference than autoregressive models, they tend to produce incoherent and repetitive results due to complex data distribution (e.g., acoustic and linguistic variations in speech). In this work, we introduce DIFFNORM, a diffusion- based normalization strategy that simplifies data distributions for training NAT models. After training with a self-supervised noise estimation objective, DIFFNORM constructs normalized target data by denoising synthetically corrupted speech features. Additionally, we propose to regularize NATs with classifier-fre guidance, improving model robustness and translation quality by randomly dropping out source information during training. Our strategies result in a notable improvement of about +7 ASR- BLEU for Sggiish-Spaishh (En-Es) and +-2 ASR-BLEU for English-Frent Frenchh (En-Fr) translations on the CVSS benchmark, while attaining over 14x speedup for En-Es and 5x speedup for En-Fr translations compared to autoregressive baselines."
1 Introduction
Speech-to-spedd translation (S2ST) systems are essential to bridge communication gaps and have wide application potential. We focus on non-autoregressive modeling for direct speech-to-speect translation, converting source speech to the target without intermediate text data. Such direct S2ST systems [25, 23. 22, 32, 31, 21, 20] can preserve non- linguistic information, avoid error propagation from cascaded systems [29, 34] (c.g., combination of speech recognition and machine translation systems), and achieve faster inference speed.
Non-autoregressive Transformers (NAT) [21, 20] has played a central role in current S2ST work [31, 32, 21J. NAT translates source waveforms into target speech units via parallel decoding, achieving performance comparable to or better than autoregressive models while greatly reducing inference time. This process is often referred are then converted to target waveforms via a unit vocoder in the umil-to-speech synthesis stage [32, 38]. However, NATs suffer from incoherent and repetitive generations, referred to as the multi-modality problem [12]. This issue stems from NAT's s assumption of conditional independence during parallel decoding, worsened by the complex and multi-modal nature of training data distribution.
speech-to-unit (S2UT) translation
[31]. The predicted speech units
In this work, we propose DIFFNORM, a self-supervised speech normalization strategy that alleviates the multi-modality problem of NAT models by simplifying the target distribution. Instead of distilling training data from an autoregressive model [10] or utilizing perturbed speech to train a normalizer [32, 21], we rely on the denoising objective of Denoising Diffusion Probabilistic Models [15, DDPM] to normalize target speech units. DIFFNORM inject synthetic noise to speech features and use diffusion
Code available
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

model to gradually recover the feature, obtaining simplified and more consistent data distribution that obscures non-crucial details. As the denoising objective is learned in self superviee manner over latent speech representations, it eliminates the necessity for transcription data [32] or manually crafted perturbation functions [21]. As illustrated in Fig. 1, applying DIFFNORM to the target data obtains normalized speech units that lead to better NAT training for speech-to-ung unit translation.
Besides using DIFFNORM as a data-centric strategy to mitigate the multi-modality problem, we also propose regularization strategy to enhance the NAT model's robustness and generalizability when facing complex data distribution (e.g.. linguistic diversity and acoustic variation [21, 20]). Inspired by massifier-frre guidance [17, CG], during training, we occasionally drop out source information and replace it with a "null" representation, compelling the models generate coherent units without conditioning on the source data. During iterative parallel decoding of the NAT model, we obtain higher-quality translation by mixing conditional and unconditional generation. Ultimately, combining DIFFNORM and CG results in our porprrrongi state-of-the-ee system, achieving approximately +7 and +2 ASR-BLEU increment for En-Es and En-Fr translation compared to previous non-autoregressive systems on the CVSS [24] benchmark.
In conclusion, we alleviate the multi-modality problem by proposing (I) diffusion-basee normalization and (2) regularization with classífier-free guidance. To the best of our knowledge, we are the first to adapt diffusion models and classifier free guidance into speech-to-speede translation and NAT modeling. Our methods obtain notable improvement compared to previous systems and maintain fast inference speed inherent in non- con-autoregressive modeling, achieving speedups of 14x for En-Es and 5x for En-Fr compared to autoregressive baselines.
2 Problem formulation and overview
We aim develop direct (text- less) speech-to-speech translation sys- tem that transduces a source speech = (æ1, , .N) into target speech. We follow Lee et al. [31] to re- duce speech-to-speech translation into two sub-tasks: speech-to-ung transla- tion and unit-to-speech synthesis. In this work, we focus on speech-to-unie translation and follow prior work [31, 32, 21] to use the same unit-to-speech component for a fair comparison. To generate speech units for the target language, we first extract speech fea- ture h = (h),... hb)) E M x where each feature has H dimensions. Subsequently, a K-means clustering model is trained extracted features and used to generate speech units y =
DiffNorm: diffusion-based speech normalization
LM beel
Latent Diffusion
Largel spoech
spench feature
matmalized gold units
0000
Fasem
extropy
0000
SOUT model
aoxatic specch
predicted units y
Unitt Vocoder
synthesized target speech
Figure I: Overview of our proposed system. We first normal- ize the target speech units with the denoising process from the latent diffusion model. Then speech-to-und (S2UT) model is trained to predict normalized units, which are converted into waveform from an off-the-shelf unit-vocoder.
(y,,... , WM) ) IR A Once the source speech æ and target speech unit y are prepared, we train cequence-to-sequene
models to translate from source speech into target units. In practice, we follow Lee al. [32] to use multilingual-HHunt HuBRRT (mHuBERT)) model to extract H 768 dimensional speech features h. Then we use a 1000-cluster K-means model to predict speech units given the encoded features. 2 For speech-to-unil translation, we follow Huang al. [21] to adopt Conditional Masked Language Modeling [10, CMLM], a kind of non-autoregressive transformer (NAT).
To mitigate NAT models'
problem, propose DIFFNORM (section $3), which denoises synthetically corrupted speech features to construct normalized speech units ynorm- As shown in Fig. 1, such normalized units then used to train the S2UT (CMLM) model, which generates better-translated units for the unit vocoder to synthesize target speech.
2 We take the mHuBERT and K-means models the--fh--shef to produce units consistent with prior work. See wwtee//annneeeeaeeeeeeeeee exwmmlesaaee//eeeeeeeeeeeeeeeeee htspppecceepcpcceeeeeeeeeeeeecee speech_ to_ to_speech
github. wwt.////eebbboeeeeeeeeeeeeeeeeee wwweeee//aai//ee/eeeeeeeeeeeeeee

---

Additionally, we propose to incorporate classifier- free guidance [17. CG], widely adopted strategy for diffusion- based image generation, as a regularization method to improve the -oooeettrerrrsivvve (peech-to-unit system (section $4). As shown in Fig. 3, by forcing NAT models to unmask target speech units without conditioning on source information, the model becomes more robust, generating coherent speech units that result in higher-quality translations. During inference, we follow classifier- free guidance to mix conditional and unconditional generation for the NAT model's iterative decoding, further enhancing its translation quality.
3 DIFFNORM: denoising diffusion models for speech normalization
Previous normalization strategy [32. ] on speech-to-umi translation relies on connectionist temporal classification (CTC) fine-tuning [4] using the HuBERT [19] model. For example, Lee et al. [32] rely on single-speaker data from VoxPopuli [58] to produce normalized (speaker- invariant) speech units for HuBERT-CTC fine-tuning. On the other hand, Huang al. [21] generate acoustic-agnostic units by perturbing rhythm, pitch, and energy information with (manually) pre-defined transformation functions. We propose to normalize speech units using self-supervised denoising objectives from Denoising Diffusion Probabilistic Models [ [15, DDPMJ. We believe DDPM is more suitable for speech normalization because: (1) It only requires monolingual speech data, without the need of transcriptions to create a text-to-unit model as in [32]. (2) It learns to denoise features in a high-dimensions rather than relying on hand-designed perturbation as in [21]. DIFFNORM consists of a variational auto-encoder (VAE) and diffusion model. Since vanilla VAE [27] and DDPM [15] are applied to image generation, we modify them to support token generation and incorporate multitasking objectives. The VAE model reduces the dimension of the speech feature, mapping feature h into lower dimensional latent z. The diffusion model (visualized in Fig. 2) is then trained to denoise on the latent representation space, aiming to recover the original latent from the standard Gaussian noise ž7." In the subsequent sections, we begin by outlining the architecture of our VAE model ($3.1). Then, we delve into how the diffusion model is trained using the latent variables encoded by the VAE ($3.2).
space
3.1 Variational auto-encoders for latent speech representation
Since speech features encoded by mHuBERT have a high dimension (H 768), we compress the feature into lower-dimension latents for the diffusion model. The VAE model consists of an encoder, decoder, and a language modeling head. Following our problem formulation ($2), we first prepare a sequence of target speech features h = (h1,... h.r) E RMXH and their corresponding speech units y = (y) ... yM)) Our VAE model's 's encoder will map the feature into lower dimension latent = = fh: fene) M where Z < H is the pre-defined latent dimension.
Then VAE model's decoder reconstructs the speech feature h = f(; a and we apply the language modeling head to convert the reconstructed feature into a distribution over speech units' vocabulary ff; 0mm) E R.MXY, Following prior work [21, 32], we cluster speech features into V = 1000 weighted combination of reconstruction loss (Creoon), negative log-likelinn likeliihod (NLL) loss (Cmn), and Kullback- Leibler (KL) divergence term resulted from the Gaussian constraint [27] to regularize the representation space:
units. The training objective
L= AiCrron + d2 + A3
where the reconstruction loss is defined as frecon (h, h) = h h2 and the NLL loss is computed as the cross-entropy between ground-truth units and the predicted vocabulary distribution o: Lau(w, y) = EM y: log D; where y: is the one-hot version of yt- Lastly, Lu can be solved analytically when we assume Gaussian distribution for both prior and posterior approximation of latent z (more details in [27] and Appendix B). Though the Gaussian constraint has small weight As. we show in the ablation study ($6.2) that regularizing latent distribution critical for the diffusion model. For details on the architecture of our VAE model, we direct readers to Appendix B.
3.2 Diffusion model for denoising latent speech representation
Training Once the VAE model is trained, we encode speech feature as the first step feature zo = f(h; Dene) for the diffusion model. Diffusion models consists of a (1) forward process that
3 "The subscript zo is added to indicate the time step for diffuusion process and zo and z equivalent.
3

---

noiss onjection
enc
Zo
2/
2T4
20
de:
LM-head
speech feature
reconstructed feature h
normalized units Ynorm
denossing
Figure Visualization of our latent diffusion model's denoising process for speech normalization. The clean latent 2o is synthetically noised (into 2ạ) and the reverse diffusion process gradually denoise it to generate normalized speech units.
gradually transforms into standard Gaussian distribution, and (2) reverse process that denoise and recovers the original feature 26. Following DDPM [15], with a pre-defined noise scheduler (let B. E (0. be the scaling of noise variance, define a₂ = - 3 and denote ã = II By as the noise level for time t), the forward and reverse process can be written
(<z)|zo)= = -Ziz Vo+zo, -)))c) (o |z) 1o o2)
(2)
where mean (z t) and variance o² are parameterized by trainable models, typically based on U-Net [44] Transformer [57]. We follow DDPM train models using the re-weighted noise
estimation objective:
Laoise (0, F..g,.. (z, 4)12
where the network learns to predict the injected Gaussian noise from the forward process, Besides noise estimation, we also train the model with auxiliary reconstruction and NLL loss using VAE model's decoder and language modeling head. Our training procedure is summarized in Alg. 1.
Algorithm Latent Diffusion Model Training
Algorithm 2 Normalized Units Construction
1: Input: speech feature speech units y.
1: Input: speech feature h, start time T
Pre-compute: Br, ore, dee f [1,7] 7]
2: Pre-compute: Br, Qie, die E [1
3: while not converged:
zo = f'h;; Oonc)
4: f(:: Omne)
4: Zr Varzo /1 āre
5: 21,,
1)
5: = T
2: Vāizo + vI ase
6: while t 0:
7: = Eω(z,, t)
é (z(2).
8: Luois = |le - El12
8: e0 = (2:
V 2.,E) /
9: &0 (zt- VI- aoé)\
9: z = var-120 + VI ac-1
10: h = f(Zo; Odcc), Green = h h
10: t=t
11: v f(h; 0lm). Lall NLL(O, y)
11: h J(zo; Odar)
12: Take gradient descent step with loss
12: Yaorm argmax f fhh; 1m))
13: L 71 Cmiis 72 Crecon
Callll
13: return denoised units Ynorm
As shown in Alg. 1, we randomly sample the current timestep t E [1, T'] and compute corresponding scheduling parameters B,, at, ā; The training process involves the regular DDPM objective that injects Gaussian noise for the model co to perform noise estimation (Alg. 1, line 6-8). Additionally, we generate the pseudo latent &g by reversing the noise injection process (line 9), which then decoded by the VAE into speech features and units for reconstruction loss (line 10) and NLL loss (line 11). Finally, the objective is a weighted sum of noise estimation loss, reconstruction loss, and NLL loss:
L 71 Lmoise 7/2 Crecon + Call
In our analysis ($6.2). we show that adding NLL and reconstruction loss is indeed helpful in improving diffusion model's reconstruction quality. Lastly, to parameterize our diffusion model for noise estimation, we modify Diffusion Transformer [37, DiT] to suit our task. For details of our architecture and hyper-parameters, we refer readers to Appendix B.
Inference For speech normalization, we choose start time T' E [0, 200] that decides the amount of noise to inject for the diffusion model to recover. Given the start time T. we follow Denoising Diffusion Implicit Models [48, DDIM] sampler to reverse noised latent zr back to zq. Then, our VAE model converts 2g back to speech units with its decoder and language modeling head. Our inference procedure is summarized in Alg. 2 and visualized in Fig. 2.
4 We follow the cosine schedule proposed by Nichol and Dhariwal [35] with maximum timestep of 200

---

uninakk prediction
encoder represcotation
0000
source speech
Coufrrere Modss
Ravixn Merkig
GOO
Decoder
welacewwih pog
unit embedding
DiffNorm
sall reprosentation go
Yuormi
carget speech
Figure 3: Visualization of CMLM for speech-to-une uncitt translation where the model is trained with the unmasking objective to recover Ynorm- When classifier-fre frree guidance is used, with probability Pdrop- replace the encoded souree speech g by "null" representation Pt-
Note that since our diffusion model is used to normalize speech units as data preprocessing strategy, inference speed is not concern. Therefore use step size At for DDIM sampling throughout our experiments. However, the inference speed could be easily improved by setting larger step size.
4 Classifier-free guidance for non-autoregreesive transformer
In $3, we proposed DIFFNORM to normalize speech units and obtained speech-unit pairs (æx Yuorm) that benefit NAT training. In this section, we propose to adapt classifier-free guidance [17] to regularize NAT models (visualized Fig. 3), further enhancing NAT-based S2UT model's translation quality,
Training We largely adhere to standard CMLM training [21] but introduce a small dropout probability Parop 0.15 for the encoder representations. Specifically, we parameterize the encoder and decoder of the CMLM as dene and odec. respectively. The encoder processes the source speech into representation g = f(x; Dene). The decoder then predicts the vocabulary distribution fūlg; ddec) from g and the randomly masked target speech units ÿ. The total amount of masked token uniformed sampled: n ~ [111, M]; subsequently, n of M tokens from the target units y are randomly masked to form noisy target units CMLM also trains length predictor that estimates the output length given input sequences and we refer readers to [21] for more details.
Using the predicted distribution v, we compute the NLL loss against the ground truth units at the masked positions to train the model. If dropout is applied, the decoder receives "null" representation ge-a randomly-initialize learnable vector-forcing it to rely solely on the target information to unmask units. For more details, we refer readers Appendix C and previous paper [10, 21].
Inference CMLM generates tokens through iterative unmasking. Given the input sequence æ. CMLM first predicts the length of output sequence M and initialize sequence of M masked tokens ÿo = ( [m.as]]..".. ... [mask] M)) Given the total number of iterations T and current iteration e [1, T CMLM decodes all tokens yr, ė [1, .M in parallel using their probabilities:
v argmaxP (1= w w|.., Dr sPtor)
rf = log IP ((y/( ar, 11-1; Ddvc)
From the generated tokens 인. (vi.-. y.r), we replace ls M I-l tokens with the lowest probabilities p with mask tokens. This re-masked sequence ÿ: then inputted into the CMLM for further decoding iterations. This process continues until the final step t = T, at which point no tokens are re-maseed. With our proposed regularization from classifier free guidance, we compute two vocabulary distribution in each iteration step t:
Porig = PP-lg Rdcc) Piucoond = ((-lgo Úc-1; (dec)
where Porig is the same as vanilla CMLM and Pincoond is the unconditional probability obtained with the "null" representation. Then we select and re-mask tokens using the adjusted probability
n¹ = (y'ne - Pincondd + Pont
where the hyper- paper-parameter w is used to control the degree to push away from the unconditional distribution. In the special case of decoding, resulting in the same generation process as standard CMLM. Through our intensive analysis
O. only conditional distribution is used during iterative
$ 'I' was used in $3 to denote total timesteps for diffusion models. We abuse the notation and re-us it here to represent the number of decoding iterations for CMLM.
5

---

(Appendix F), we determine that a relatively small w (e.g., w = 0.5) gives the best result, especially when the number of decoding iterations is large (i.e., T 10). Notably, even when w = 0 which corresponds to the traditional CMLM decoding method - CMLMs regularized with classifier-free guidance during training can consistently outperform their standard counterparts. This observation further validates the effectiveness of our proposed regularization.
5
Experiments
5.1 Experimental setup
Dataset We perform experiments using the es- tablished CVSS-C datasets [24], which are created from CoVoST2 by employing advanced text-to- speech models to synthesize translation texts into speech [59]. CVSS-C comprises aligned speech in multiple languages along with their respec- tive transcriptions. Our methods are evaluated (En-Es) and
En-Es
En-Fr
Split
Size
Length
Size
Length
Train
79,012
256
207.366
228
Valid
13,212
296
14,759
264
Test 13,216
308
14,759
283
Table Data statisties for CVSS benchmarks. Length is the average number of speech units of the target speech.
two language pairs: English-Spaniie Frglish-Frenchh (En-Fr), with detailed data statis- ties provided in Table 1. As our focus is on direct speech-to-speech translation, transcriptions the speech data from CVSS, we preprocess the target speech to generate speech units using the mHuBERT and K-means model, described in our problem formulation ($2)..
solely utilized for evaluation purposes. Utilizing
translation systems, we adopt the standard methodology established in previous studies [32, 31, 21] to compute the ASR-BLEU score. Firstly, our speech-to-unit translation system generates speech units based on the input speech, which are then transformed into speech waveforms using unit-vocoder. The unit-vocoder is built upon the HifiGAN architecture [28] with customized objectives, detailed in Appendix D. Once we have the waveforms, against the reference transcriptions. This ASR model is fine-tuned based on the WAv2VEC2.0 [5] model with the CTC objective. We direct readers the original paper [7] for more details. Both the unit-vocoder and ASR models are sourced from an off-the-shelf repository, ensuring consistency in evaluation methodology with previous studies [32, 21J.
Evaluation To evaluate the performance of various
employ ASR model transcribe them and calculate the BLEU score
Normalized dataset construction We adhere to the procedure outlined in Alg. 2 to generate normalized speech units. By manipulating start time T, control the level of noise in the latent 3y = V0 + -I- ā,e for the diffusion model, balancing between the reconstruction quality and normalization effect. We explore different levels of noise injection and choose T 100 for En-Es and T = 120 for En-Fr (further details in $6.1). Hence, we adopt these settings to construct our normalized dataset, CVSS-NORM.
Compared systems In this section, we provide brief descriptions of evaluated speech-to-unie models. For autoregressive models, evaluate the Transformer model trained following Lee et al. [31]. We also assess the Conformer model trained similar to the Transformer, but with its encoder replaced by Conformer-Encoode Transformer, but it is trained on normalized speech units that are ppeaker-invarian The normalized dataset is constructed following the strategy proposed by Lee et al. [32].
[14]. Lastly, the Norm Transformer shares the same architecture as the
For non-autoreeresssvve systems, we train the Conditional Masked Language Model (CMLM) following Huang et al. [21]. J. using a Conformer- based encoder and a Transformer decoder. The CMLM + Bilateral Perturbation (BiP) system retains the same architecture as CMLM but is trained on normalized speech units constructed with BiP [21].
For our improved systems, we train CMLM DIFFNORM, which shares the same architecture as CMLM but is trained on NORM, the normalized dataset obtained through DIFFNORM. Additionally, we train the CMLM CG model that incorporates classifier-free guidance introduced in $4. Finally, the CMLM + DIFFNORM + CG system uses the architecture of CMLM + CG and is trained on our normalized dataset CVSS-NORM.
Repository asr asr_bles bleu.
ctm htt.ghthh..coeeeeeooeeeee//aaaaa
https//sspecchh___peeeeeeeeeee to_speech/

---

5.2 Results
Table 2 summarizes the (ASR-BLEU) performances of various S2U'T systems. Note that, for non- autoregressive methods, Table 2 shows their results obtained with 15 decoding iterations. For more details on the inference speedup achieved through non-autoregressive modeling, we plot Fig. 4 to provide details the "quality vs. latency" trade-off. Observations from Table are:
DIFFNORM greatly enhances translation quality compared to systems using original speech units (model vs. 4; 8 vs. 7). For example, CMLM with DUFFNORM achieves about +7 BLEU score on En-Es compared to CMLM trained on original units. DIFFNORM also outperforms previous normalization strategies (model 6 v.s. 2, 5), validating the superior quality of normalized speech units obtained through our methods. Besides normalization, classifier free guidance effectively improves speech-to-unts quality regularization strategy, leading to better translation quality (model 7 v.s. 4; v.s. 6). Finally, combining both classifier-free guidance and DIFENORM (model 8) results in the best overall system, outperforming both autoregressive and non-autoregressive baselines.
Quality 1
Inference Speed
ID
System
En-Es
En-Fr
Fr-En
Speed
Speedup
Autoregressive
Transformer [31]
10.07
15.28
15.44
870
1.00x
2
Norm Transformer' [32]
12.98
15.93
15.81
870
1.00
3
Conformer!
13.75
17.07
18.02
895
1.02x
Non-autoregressive Model
4
CMLM
12.58
15.62
16.95
4651
5.34x
5
CMLM + BiP
12.62
16.97
18.03
Our Improved Non-autoregreessve Model
6
CMLM + DIFFNORM
18.96
17.27
19.53
7
CMLM + CG
17.06
16.89
4651
5.34x
CMLM DIFFNORM CG#
19.49
17.54
Table 2: Comparison of speech-to-speccee models evaluated by quality (ASR-BLEU) and speed (units/seconds)." " Fr-En experiments are added during author response period and we leave model 7,8 for future work. Results with are taken from the prior work [21]. $ We use w = 0.5 for CG. Our NAT models achieve superior translation quality while maintaining their fast inference speed.
Decoding speed In Fig. 4, we illustrate the "quality-latency" trade-off of various non-autoregressive speech-to-une systems. Quality measured using ASR-BLEU, while latency is determined by the relative speedup over the autoregressive system, calculated as "generated units/second". For instance, the first marker on the line plot represents 15 decoding iterations, resulting in a speedup of 5.34x compared to the autoregressive baseline. Additionally, we include the performance of the Conformer-based autoregressive model as a horizontal dashed line. We observe that our improved system consistently outperforms the baseline CMLM model [21] and achieves better performance than the autoregressive model with more than 14x speedup for En-Es Ess and 5x speedup for En-Fr.
Analysis
6.1 Effect of synthetic noise injection
In this section, we explore the effects of varying degrees of noise injection on DIFFNRM. Recall (from Alg. 2) that synthetic noise injected into the latent representation as žT 20r + /11 ā c. Hence, adjusting the start time T allows control the degree of noise injection.
perturb the degree of noise injection by varying T. and assess the reconstruction quality of normalized speech units and downstream performance of CMLM models trained with different normalized units. For reconstruction quality, we measure the accuracy (Acc- Rec) of reconstructed units and ASR-BLEU (BL-Ree) of synthesized speech when using original
Setup and Evaluation
7

---

En-Es
En-Fr
Performance (BLEU)
20
17.5
5R-17.02
15.0
15
135
12.5
CMLM
10
10.0
CMLM CG
CMLM DMNomm
7.5
CMLM DINom CG
6X 9x 12x
16x
6x
9x
12x
16x
Relative Speedup
Figure 4: Trade-off between quality (ASR-BLEU) and latency for varying numbers of decoding iterations. Five markers correspond to (15, 10, 3) decoding iterations. Decreasing the number of iterations results in decline in model performance, traded off for faster speedup. With DIFFNORM and CG, our S2UT model achieves better quality-latency trade-off than CMLM and outperforms a strong autoregressive baseline with large speedups.
units/speech as the reference. For downstream performance, we report the downstream ASR-BLAB BR-BEEE of translated speech produced by CMLM models trained with the particular noise setup (BL-Dn)."
Results Shown in Table 5, increasing 7 leads to more noise and a decline in reconstruction quality, which aligns with expectations that higher noise levels pose greater challenges for the diffusion model in restoring original features accurately. Interestingly, we find from Table that when is too small or large (e.g., 7' 50 or T 150), the normalized units do not result in an ideal downstream system. We hypothesize that there is barely any normalization effect when 1' is too small; and when I' is large, the reconstructed units are longer semantically correct. This phenomenon visualized in Fig. 5, where we plot the log-mel spectrograms of the reconstructed speech. As more noise is introduced (larger T), the reconstructed speech becomes smoother (e.g., portions of blank speech are filled in by diffusion), exhibiting a more pronounced deviation from the original speech.
Start Step
Scheduler Parameters
En-Es
En-Fr
A
VT
Acc-Rec
BI-
BL.-D
Acc-Ree
BI.-Rec t
BL.DD
original units
N/A
NIA
N/A
100
48.7
12.58
100
40.64
15.62
T = 50
0.007
0.917
0.398
89.4
48.5
18.56
91.1
40.38
17.12
T = 100
0.016
0.697
0.717
81.2
47.63
18.96
83.0
39.9
16.69
T
0.022
0.577
0.816
75.3
16.7
77.7
17.27
T 150
0.038
0.373
0.928
57.8
31.43
14.77
60.0
28.3
14.03
Table 3: For different start steps T, we show corresponding noise scheduling parameter values, reconstruction quality (-Rec columns), and downstream translation quality (-Dn column). Noise injection that perturbs about 20% of units (i.e., 80% Acc-Rec) results in the best downstream S2UT performance (highlighted in bold text).
From the Table 3, we observed substantial enhancement in En-Es translation performance with T 50. Conse- quently, we conducted additional experiments for En-Es translation with start times T = 10 and T = 30, detailed in Table 4. The results show that at a minimal start time of T = 10, the reconstruction accuracy reaches 93.8%. yielding a downstream ASR-BLEU score of 15.98. This score significantly surpasses the baseline CMLM result of 12.58. The high accuracy indicates that about 6% of tokens vary pootstrcontt,uttion, likely due to the VAE model's regularization effect since the diffusion model does not cause notable reconstruction deviations at such small T. This suggests that the Spanish speech dataset might have considerable acoustic variations and noise, which the VAE model can partially mitigate. As T increases, the diffusion model further refines the representation, leading to improved downstream results, as evidenced by the rise in ASR BLEU score from from
Start Step
Ace-Ree
BL-Ree
T 10
93.8
15.98
T 30
91.6
17.92
Table 4: Reconstruction and downstream performance with small noise injection.
"ASR-BLE BLEU is computed by synthesizing speech from the reconstructed units and transcribing it using the ASR model, detailed in I.

---

Oiriginal
T 100
T 150
T 200
Mei frequency
Figure 5: Visualization of reconstructed speech's 's log-mel spectrograms. Noticeable divergence from the original speech is highlighted in the white bounding boxes.
10 to 7 = 100. This ablation study underscores that the optimal choice of T is affected by the dataset quality.
=
6.2 Ablation on training objectives for DIFFNORM
DIFFNORM requires an auto-encoder map speech features into lower-dimension latents and then train denoising diffusion models using the encoded latents. In this section, we perform ablation experiments to investigate the effect of (I) latent dimen- sion (2) Gaussian constraints (3) multitask objective for diffusion models. Firstly, to investigate the effect of latent dimension, we train auto-encoo -nooders that encode speech features into 16, 32, and 128 dimensions. sur-
Start Step
Objectives
Latent Dimension
KL
Multitask
32
128
X
51.4
69.1
85.8
50
x
80.9
86.4
89.3
/
80.9
86.5
89.4
x
V
13.3
32.5
73.1
=
V
x
64.5
76
80.8
V
65.3
76,2
81.2
As shown in Table 5, it comes prise that a larger latent dimension yields su- perior reconstruction results, evidenced by higher accuracy across systems trained with varying objectives. Next, turn atten- tion to the importance of applying Gaussian constraints to the auto-encoo nuto-encoder's latent rep- resentation space. Our results (comparing with and without KL) reveal that incorporating Gaussian constraints is crucial. Latent spaces not regularized by Lu lead to significantly poorer performance. Finally, we explore the effectiveness of our proposed multitasking objective for the diffusion model by training another vanilla DDPM solely with Luoise (no Multitask). We find that the diffusion model's reconstruction capability indeed improves when employing the multitasking objective, particularly when denoising from latent representation (c.g., g.. T 150). Through our ablation analysis, we identify the optimal setup for Dim model, which involves (I) mapping speech features 128 dimensions, (2) regularizing latent space by Gaussian constraints, and (3) utilizing the multitask objective for diffusion training.
X
/
2.9
5.3
34.6
= 150
x
19.8
38.2
56.8
20.2
57.8
Table 5: Accuray of reconstructed speech units. KL: when applied, the latent space regularized be Gaus- sian [27]. Multitask: when not applied, the latent diffu- sion model is trained only with Luoise.
noisy
7 Related work
Direct speech-to-speech translation (S2ST) Direct S2ST aims to directly translate speech into another language without cascaded systems that rely on transcriptions or translations. Translatotron [25] and Translatotron 2 [23] are among the first systems for direct S2ST, which uses sequence-to- sequence model to map source speech into spectrograms. Then, a spectrogram decoder is used to synthesize the target language's speech. Instead of transducing speech into spectrogram, Tjandra al. [53], Zhang et al. [62] utilize Vector- certor-Quantized Variational Auto-Encoder (VQ- VAE) [56] to discretize target speech and convert speech-to-speecl translation into a speech-to-unit task. More recently, Lee et al. [31] improved the speech-to- models by obtaining such units with a k-means clustering model trained self-supervised representation from HuBERT [19]. To convert units back to speech, Lee et al. [31] follows Polyak et al. [38] to train a unit-vocoder based on HifiGAN [2].
Speech normalization Speech representation are typically extracted from pre-trained encoders [19, 3, 40, 5] and can compressed or adapted in different speech-to waapch-tt-peeec/tee tasks [61, 63, 50, 52, 51]. Inspired by previous work speech enhancement [54, 1]. Lee et al. [32] propose to

---

normalize speech by synthesizing speaker-invariant waveforms through text-to- spoech (TTS) systems [60, 46, 42, 41]. Huang et al. [21] propose to normalize speech with Bilateral Perturbation that focus on the rhythm, pitch, and energy information. Different from previous normalization methods that requires transcription [32] or manually designed perturbation [21], our DiFFNORM strategy leverages diffusion models. Diffusion models [15, 48, 49] have achieved remarkable generative ability to produce high-quality images [8, 16, 43, 37, 45J and audio [33, 39, 47J. Using self-supervised denoising objectives [15], we train effective denoising models capable of normalizing speech for training better speech-to-une systems.
Non-autoregressive speech-to-specd translation For -ereeeeet---euuenee modeling, auto- regressive [18, 6, 57] and non -oooreeereerrsivvee [12, 13. 10, 20] models have been widely explored. Lee et al. [31] reduce ppeech-to-specee
into speech-to-unit task and follow the widely-used modeling strategy, Transformer [57], to predict speech units. Later, non-autoregressive models [21, 9] have been explored and Huang et al. [21] is the first to use non-autoregressive transformer model, Conditional Masked Language Model [10, CMLM], for speech-to-unie task. Fang al. [9], on the other hand,
adopts Directed Acyclic Transformer [20] for speech-to-unit translation.
8 Conclusion
We improve speech-to-unie translation system through (1) corpus distillation by constructing normal- ized speech units with DIFFNORM and (2) regularization with classifier-fres guidance. Our improved non-autoreerressive (NAR) models, greatly outperform previous NAR models, yielding an increase of approximately +7 and +2 BLEU points for En-Es and En-Fr translation, respectively. Notably, DIFFNORM and classifier-free guidance maintain the inference speed advantages inherent NAR models. Consequently, our approach obtains better performance autoregressive baselines while achieving over 14x speedup for En-Es and 5x speedup for En-Fr translations.
Acknowledgement
We express our profound appreciation anonymous reviewers for their helpful suggestions. We also thank Xiuyu Li for his valuable suggestions, which greatly enriched our work. Lastly, we thank JHU + Amazon Initiative for Interactive Al for sponsoring the work.
References
[1] Adiga, N., Pantazis, Y., Tsiaras, V., and Stylianou, Y. (2019). Speech Enhancement for Noise- Robust Speech Synthesis Using Wasserstein GAN, In Proc. Interspeech 2019, pages 1821-1825.
[2] Andreev, P., Alanov, A., Ivanov, O.,, and Vetrov, D. (2023). Hifi++: A unified framework for bandwidth extension and speech enhancement. In ICASSP 2023 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5.
[3] Babu, A., Wang, C., Tjandra, A., Lakhotia, K., Xu, Q., Goyal, N., Singh, K., von Platen, P., Saraf, Y., Pino, J. M., Baevski, A., Conneau, A., and Auli, M. (2021), XIs-r: Self-supervised cross-lingual speech representation learning at scale. In Interspeech.
[4] Baevski, A., Auli, M., and rahman Mohamed, A. (2019). Effectiveness of self-supervised pre-training for speech recognition. ArXiv, abs/1911.000 1.09912.
[5] Baevski, A., Zhou, H., Mohamed, A., and Auli, M. (2020). wav2vec 2.0: A framework for self-super learning of speech representations.
[6] Bahdanau. D., Cho, K., and Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. CoRR. abs/ fos//140/ 409.0473
[7] Chen, P.-J.. Tran, K., Yang, Y., Du, J., Kao, J., Chung, Y.-A., Tomasello, P., Duquenne, P.-A.,
Schwenk, H., Gong, H., Inaguma, H., Popuri, S., Wang, C., Pino, J., Hsu, W.-N., and Lee, A. (2023). Speech-to-speech translation for a real-world unwritten language. In Rogers, A., Boyd-Graber, J., and Okazaki, N., editors, Findings of the Association for Computational Linguisties: ACI. 2023, pages 9969-4983, Toronto, Canada. Association for Computational Linguistics.
10

---

[8] Dhariwal, P. and Nichol, A. (2021). Diffusion models beat gans on image synthesis.
[9] Fang, Q. Zhou, Y., and Feng, Y. (2023). DASpeech: Directed acyclic transformer for fast and high-quality speech-to-speech translation. In Advances in Neural Information Processing Systems.
[10] Ghazvininejad. M., Levy, O., Liu, Y., and Zettlemoyer. L. (2019). Mask-prediet: Parallel decoding of conditional masked language models. In Inui, K., Jiang, J., Ng, and Wan, X., editors, Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6112-6 2-21,,, Hong Kong, China. Association for Computational Linguisties.
[11] Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative adversarial networks.
[12] Gu, J., Bradbury, J., Xiong, C., Li, V. O., and Socher, R. (2018). Non-autoregrresive neural machine translation. In International Conference on Learning Representations.
[13] Gu, J., Wang, C.,, and Zhao, (2019). Levenshtein transformer. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buu Buc, F., Fox, E., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
[14) Gulati, A., Qin, J., Chiu, C., Parmar, N., Zhang. Y., Yu, J., Han, W., Wang, S., Zhang, Z., Wu, Y., and Pang. R. (2020). Conformer: Convolution-qugmente transformer for speech recognition. CoRR, abs/2005.08100
[15] Ho, J., Jain, A., and Abbeel, P. (2020). Denoising diffusion probabilistic models. Larochelle, H., Ranzato. M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Systems, volume 33, pages 6840-6851, Curran Associates, Inc.
[16] Ho, J., Saharia, C., Chan, W., Fleet, D. J., Norouzi, M., and Salimans, T. (2021). Cascaded diffusion models for high fidelity image generation.
[17] Ho, J. and Salimans, T. (2022). Classifier-free diffusion guidance.
[18] Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural computation, 98):1733-
[19] Hsu, W.-N., Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhutdinov, R., and Mohamed, A. (2021). Hubert: Self-supervised speech representation learning by masked prediction of hidden units.
[20] Huang, F., Zhou, H., Liu, Y., Li, H., and Huang, M. (2022). Directed acyelic transformer for non-autoregreesive machine translation. Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S., editors, Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 9410-9428, PMLR.
[21] Huang, R., Liu, J., Liu, H., Ren, Y., Zhang, L.., He, J., and Zhao, Z. (2023). Transpeech: Speech- to-speech translation with bilateral perturbation. In The Eleventh International Conference on Learning Representations.
[22] Inaguma, H., Popuri, S., Kulikov, I., Chen, P.-J., Wang, C., Chung, Y.-A., Tang, Y., Lee, A., Watanabe, S., and Pino, J. (2023). UnitY: Two-pass direct speech-to-speed translation with discrete units. In Rogers, A., Boyd-Graber, J., and Okazaki, N., editors, Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 15655-15680, Toronto, Canada. Association for Computational Linguistics.
[23] Jia, Y., Ramanovich, M. T., Remez, T., and Pomerantz, R. (2022a). Translatotron 2: High-quality direct speech-to-speech translation with voice preservation.
[24] Jia, Y., Ramanovich, M. T., Wang, Q. and Zen, H. (2022b), Cvss corpus and massively multilingual speech-to-speech translation.
[25] Jia, Y., Weiss, R. J., Biadsy, F,, Macherey, W., Johnson, M., Chen, Z., and Wu, Y. (2019). Direct speech-to-speech translation with sequence-to-sequeenc model. ArXiv, ab661994
[26] Kingma, D. P. and Ba, J. (2017). Adam: A method for stochastic optimization.
11

---

[27] Kingma, D. P. and Welling, M. (2022). Auto-encoding variational bayes.
[28] Kong, J., Kim, J., and Bae, J. (2020). Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis.
[29] Lavie, A., Waibel, A., Levin, L., Finke, M., Gates, D., Gavalda, M., Zeppenfeld, T., and Zhan, P. (1997). Janus-iii: speech-to-specch translation in multiple languages. 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing, volume I, pages 99-102 vol.1.
[30] Lecun, Y., Bottou, L., Bengio, Y., and Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2788-2324.
[31] Lee, A., Chen, P.-J., Wang. C., Gu, J., Popuri, S., Ma. X., Polyak, A., Adi, Y., Hc, Q. Tang, Y., Pino, J., and Hsu, W.-N. (2022a). Direct speech-to-speech translation with discrete units. In Muresan, S., Nakov, P., and Villavicencio, A., editors, Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3327-3339, Dublin, Ireland. Association for Computational Linguistics.
[32] Lee, A., Gong, H., Duquenne, P.-A., Schwenk, H., Chen, P.-.., Wang. C., Popuri, S., Adi, Y., Pino, J., Gu, J., and Hsu, W.-N. (2022b). Textless speech-to-specdt translation on real data. In Carpuat, M., de Marneffe, M.-C., and Meza Ruiz, I. V., editors, Pruceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 860-872, Seattle, United States. Association for Computational Linguistics.
[33] Liu, H., Chen, Z., Yuan, Y., Mei, X., Liu, X., Mandic, D., Wang, W., and Plumbley, M. D. (2023). AudioLDM: Text-to-audio generation with latent diffusion models. Proceedings of the International Conference on Machine Learning.
[34] Nakamura, S., Markov, K., Nakaiwa, H., Kikui, G., Kawai, H., Jitsuhiro, T., Zhang, J.-S., Ya- mamoto, H., Sumita, E., and Yamamoto, S. (2006). The atr multilingual speech-to-sye srrech-to-speech translation system. IEEE Transactions on Audio, Speech, and Language Processing, 14(2):365-376.
[35] Nichol, A. Q. and Dhariwal, P. (2021). Improved denoising diffusion probabilistic models. In Meila, M. and Zhang, T., editors, Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8162-8171. PMLR.
[36] Ou,, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng. N., Grangier, D., and Auli, M. (2019). fairseq: A fast, extensible toolkit for sequence modeling. In Ammar, W., Louis, A., and Mostafazadeh, N., editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 48-53, Minneapolis, Minnesota. Association for Computational Linguistics.
[37] Peebles, W. and Xie, S. (2023). Scalable diffusion models with transformers.
[38] Polyak, A., Adi, Y., Copet, J., Kharitonov, E., Lakhotia, K., Hsu, W.-N., Mohamed, A., and Dupoux, E. (2021). Speech resynthesis from discrete disentangled self-supervised representations.
[39] Popov, V., Vovk, I., Gogoryan, V., Sadekova, T., and Kudinov, M. (2021). Grad-tts: A diffusion probabilistic model for text-to-speech. In Meila, M. and Zhang. T., editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8599-8608. PMLR.
[40] Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. (2022). Robust speech recognition via large-scale weak supervision.
[41] Ren, Y., Hu, C., Tan, X., Qin, T., Zhao, S., Zhao, Z., and Liu, T.-Y. (2022). Fastspeech 2: Fast and high-quality end text to speech.
[42] Ren, Y., Ruan, Y., Tan, X., Qin, T., Zhao, S., Zhao, Z., and Liu, T.-Y. (2019). Fastspeech: Fast, robust and controllable text speech.
[43] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2022). High-resolution image synthesis with latent diffusion models.
12

---

[44] Ronneberger, O., Fischer, P., and Brox. T. (2015). U-net: Convolutional networks for biomedical image segmentation.
[45] Saharia, C., Chan, W., Saxena, S., Li, L.,, Whang, J., Denton, E., Ghasemipour, S. K. S., Ayan,
B. K., Mahdavi, S. S., Lopes, R. G., Salimans, T., Ho, J., Fleet, D. J., and Norouzi, M. (2022).
Photorealistic text-to-image diffusion models with deep language understanding.
[46] Shen, J., Pang, R., Weiss, R. J., Schuster, M., Jaitly, N., Yang. Z., Chen, Z., Zhang, Y., Wang, Y., Skerry-Ryan R., Saurous, R. A., Agiomyrgiannakis, Y., and Wu, Y. (2018). Natural uts synthesis by conditioning wavenet mel spectrogram predictions.
[47] Shen, K., Ju. Z., Tan, X., Liu, Y., Leng, Y., He, L., Qin, T., Zhao, S., and Bian, J. (2023). Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers.
[48] Song, J., Meng, C., and Ermon, S. (2021a). Denoising diffusion implicit models. In International Conference on Learning Representations.
[49] Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. (2021b). Score-based generative modeling through stochastic differential equations.
[50] Tan, W., Chen, Y., Chen, T., Qin, G., Xu, H., Zhang. H. C., Durme, B. V., and Koehn, P. (2024a). Streaming sequence transduction through dynamic compression.
[51] Tan, W., Inaguma, H., Dong, N., Tomasello, P., and Ma, X. (2024b). Ssr: Alignment-aware modality connector for speech language models.
[52] Tang. C., Yu, W., Sun, G., Chen, X., Tan. T., Li, W., Lu, L., Ma, Z., and Zhang, C. (2024). Salmonn: Towards generic hearing abilities for large language models.
[53] Tjandra, A., Sakti, S., and Nakamura. S. (2019). speech-to-specee translation between transcribed unknown languages. 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 593-600.
[54] Salentini--ottinha. C., Wang. X., Takaki, S., and Yamagishi, J. (2016). Speech Enhancement for Noise-Robust Text-to-Speech Synthesis System Using Deep Recurrent Neural Networks. In Proc. Interspeech 2016, pages 352-356.
[55] van den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., and Kavukcuoglu, K. (2016). Wavenet: A generative model for raw audio.
[56] van den Oord, A., Vinyals, O., and kavukcuoglu, k. (2017). Neural discrete representation learning. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
[57] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, N., Kaiser, L. u., and Polosukhin, I. (2017). Attention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
[58] Wang, C., Riviere, M., Lee, A., Wu, A., Talnikar, C., Haziza, D., Williamson, M., Pino, J., and Dupoux, E. (2021). VoxPopuli: A large-scale multilingual speech corpus for representation learning, sumi-supe learning and interpretation. In Zong, C., Xia. F., Li, W., and Navigli. R., editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume Long Papers), pages 993-1003, Online. Association for Computational Linguistics.
[59] Wang, C., Wu, A., and Pino, J. (2020). Covost 2 and massively multilingual speech-to-text translation.
[60] Wang, Y., Skerry-Ryan, R. J., Stanton, D., Wu, Y., Weiss, R. J., Jaitly, N., Yang, Z., Xiao, Y., Chen, Z.,, Bengio, S., Le, Q. V., Agiomyrgiannakis, Y., Clark, R. A. J., and Saurous, A. (2017). Tacotron: Towards end-to-end speech synthesis. In Interspeech.
13

---

[61] Yu, W., Tang, C., Sun, G., Chen, X., Tan, T., Li, W., Lu, L., Ma, Z., and Zhang, C. (2023). Connecting speech encoder and large language model for ast.
[62] Zhang, C., Tan, X., Ren, Y. Qin, T., Zhang, K., and Liu, T.-Y. (2020). Uwspeech: Speech to speech translation for unwritten languages.
[63] Zhang, D., Li, S., Zhang, X., Zhan, J., Wang, P., Zhou, Y., and Qiu, X. (2023). Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities.
Supplementary Material
Appendix Sections
Contents
Appendix A
Qualitative Examples Translated Speech
Appendix B
Architecture Hyper-parameterss Diffusson Model for DIFFNORM
Appendix C
Architecture Haper-parameters Non-autoregressive Models
Appendix
Unit-to-Speech Synthesis
Appendix
Limitations
Appendix
Full Results for Classifier free Guidance
Appendix G
Experiments Results Table
A
Qualitative Examples
From $5.2 we demonstrate the effectiveness of DiffNorm and classifier-free free guidance by evaluating ASR-BLEU scores. In this section, we provide example transcriptions of generated Spanish and French speech in Table 6 and Table 7. Compared to the vanilla CMLM model that has erroneous or incomplete translation, our method results in higher-quality translation.
Model
Generated Translation
English Version
Reference
este trabajo fue posible en cooperación con otros como alphonse milne-edwards y lcon vaillant.
This work was made possible coopera- tion with others such as Alphonse Milne- Edwards and lcon vaillant
CMLM
esta obra fue posible para lugar con otros como alphon y leol vigelante
This work was possible place with oth- ers such as Alphon and Leol Vigelante This time was made by cooperation with others with Walphonxe Mill and Lcon Fal- lent
CMLM CG
hora fue hizo por la cooperación
otros con walphonxe mill y leon fallent
CMLM DiffNorm
esta obra posible cooperación con otros como al von milo edwad leon val- lente
This work was possible through cooper- ation with others such as Al von Milo Edwad and Leon valente
CMLM + DiffNorm CG
este trabajo fue posihle por cooperación con otros como alphonso edos león va- lente
This work was possible through the co- operation with others such as Alphonso Edos and León Valente
Table 6: Example transcription of translated Spanish speech from different systems.
B Diffusion Model Hyperparameters
B.1 Variational Auto-Encoder
Architecture: Our Auto-encoder consists of an encoder, a decoder, and a language modeling head. The encoder architecture follows WaveNet [55] that combines Convolutional Neural Networks [30] with residual connections. In practice, we use 2 stacks of WaveNet Residual Blocks, where each block has 3 layers. Our decoder uses the same architecture as the encoder revert the latent dimension back
14

---

Model Reference
Generated Translation
English Version
mais la principauté est envahie et le prince- évéque joseph-clément de bavière doit s' s'exiler
but the principality is invaded and the prince-bishop Joseph-Clement of Bavaria must go into exile
CMLM
mais la précicipauté et le prince joseph clémont de bavier fisé en exil mais la principauté est lebséve joseth cré- mend de bavière et contré exil la princepóuté est envahic et le prince- évèque joser clémant de havière est contr mais la principauté est envahie et le prend s évêque joseph de bavier et borgé en exile
but the precicipality and prince joseph clemont of bavier is exiled
CMLM + CG
but the principality is lebséve joseth cre- mend of bavaria and countered in exile the princepóuté is invaded and the prince- bishop joser clemant of bavaria is opposed
CMLM + DiffNorm
CMLM + DifINorm + CG
but the principality is invaded and takes bishop joseph of bavaria and borgé into exile
Table 7: Example transcription of translated French speech from different systems.
to the original dimension. Additionally, we apply a Transformer [57] encoder on top of the decoded feature to further enhance model capacity. The Transoormer-eeccoder has the same configuration as the vamilla Transformer-baat model (6 layers. attention heads, etc.). Lastly, to decode speech units from the feature, we use a language modeling head which is parameterized by feedforward network that converts feature dimension (H 768) into vocabulary size (V 1000).
Training: To train the VAE model, as described in $3.1, we use a combination of three objectives
L A1 Crecom + X2 + uzhui
(8)
where x 100, 1, A3 0.001. Now provide more details about Lu: in practice, we follow [27] to model the latent z by estimating its mean μ and variance o using the encoder. With the re-pa re-parameterization trick, the latent sampled as = = f e where e (O, I). By constraining the mean and variance to follow a Gaussian prior, Kingma and Welling [27] showed that
Σ (1 + Lu 2
-
j=1
We implement our VAE model on Fairseq [36]. For optimization, we use the Adam [26] optimizer with betas (0.9, 0.98) and we apply gradient clipping by setting -clip-norm=2.0, During training, we apply dropout with a probability of 1. We train the VAE model using a learning rate of 5e-4 with distributed data-parallel (DDP) on 4 A GPUs, where we set the maximum batch token to be 15000.
B.2 Latent Diffusion Model
Architecture: We modified DiT [37] to design a Transformer-basee architecture for noise estimation. Since our input latent is a sequence of encoded speech feature z E R M2 we first apply a ID Convolution to convert the latent to model's hidden dimension (set to 512). Then we also encode the timestep with a learnable Sinusoidal Positional Embedding [57] to obtain the time embedding.
Subsequently, we feed the latent and time embedding to our modified WaveNet Block, which applies an affine transformation of the latent using the time embedding, similar to the "Scale and Shift" operation in DiT's adaptive layer norm (adaLN) block. For our diffusion model, we use 4 stacks of such modified WaveNet Block, each containing 8 layers. The output feature from WaveNet is then passed through Transformer- Encoder (12 layers with 8 attention heads) to further enhance model capacity. Lastly, a projection layer parameterized by the feedforward network is used to compress the transformed feature to the latent dimension, which is then used for noise estimation in equation (3).
Training: As described in $3.2, we train the diffusion model with multitask objective:
L = "/2 Caoise + 7/2 Crecon + Call
(10)
and we empirically select ?1 = 1. = 0.25, 7/- = 0.005. For optimization, our latent diffusion model has the same setting as our VAE model (with Adam optimizer, clip-norm equals 2.0, and dropout with 0.1 probability). We train the diffusion model using a learning rate of le-4 with DDP on 4 A 1100 GPUs, where we set the maximum batch token to be 12000. The model warmup-ed by 10000 steps.
15

---

C Non-autoregressive Transformer Details
C.1 Background on Non-Autoregreesivee Transformers
Non-autorrerresivve Transformers [12, 13, 10, inter alia. J is family of models that transduce sequences. In this work, we follow the formalization from the widely-used NAT: Conditional Masked Language Model [10, CMLMJ. CMLM consists of an encoder Pene that represents the input high-dimensional features and a decoder Odec that generates tokens conditioning on the encoded features. Different from auto-regressive Transformers [57], CMLM's decoder does not apply causal masking and is trained with an unmasking objective.
CMLM Training Following formulation from $2, given the input speech = (II, .NN and target speech units y = (y/1.... W/r), CMLM first mask the target units into ý. The total amount of masked token is uniformed sampled:
211, MM; subsequently. of M tokens from the target units are randomly masked to form noisy target units ÿ. Then, the decoder generates a distribution over vocabulary conditioning on the encoder representation and noisy target units v fWig Qdec) where == f(x Φene) is the encoder representation. Cross-entropp loss is then computed for the masked positions to update model parameters. CMLM also trains length predictor that estimates the output length given input sequences and we refer readers to [21] for more details.
CMLM Mask-Predice For CMLM inference, an iterative decoding scheme is used through unmasking speech units and re-masking low-confident positions. Given the input sequence æ, CMLM first predicts the length of output sequence M and initialize a sequence of M masked tokens úo ([mask]ı, ..... [mask]ar] J. Given the total number of iterations T and current iteration E [], CMLM decodes tokens across all positions and also computes their log- pe-prbbabilities:
vf argmax (1ys w y ddee) p log IP (3/) x. Øt-1i Qdec)
(11)
T-t tokens with the lowest From the generated tokens 인t = (vi..... fhr) we replace l M T
probabilities p with mask tokens. This re- re-masked sequence ÿ: is then inputted into the CMLM for further decoding iterations. This process continues until the final step t T, at which point no tokens are masked.
C.2 Architecture
We follow the same architecture and implementation from Huang al. [21], where a Conformer- based encoder is used to obtain representation from the source speech and Transformer-eccode (with no causal masking) is used to generate units. The cenformer-encode has 12 layers with 512 hidden dimensions and 9 attention heads. The encoder also contains CNN-based sub-sampler that has an effective stride size of 320 (i.e., it reduces the length of speech input by 320x). The Transformer- decoder has 6 layers with 8 attention heads and uses fixed positional embedding for speech units. When classifier-fres guidance is used, we randomly dropout encoding from the Conformer model with probability of 15% as described in $4.
C,3 Hyper-parammter for Model Training
We train the model with a learning rate of 5e-4, using 10000 warmup steps We apply Adam optimizer with betas (0.9, 0.98) and we clip the gradient by setting -llpp-nom==10. 10. We apply dropout with 0.1 probability and we use label smoothing of ratio 0.2 when computing the NLL loss. We train the model with DDP on 4 A100 GPUs, with a maximum batch tokens of 40,000.
D Unit-to-Speech Synthesis
We follow prior work [32, 38, 41] to convert predicted units (from S2UT model) into speech waveforms. Specifically, given units we use the HifiGAN-
unit-vocoder from [38] to synthesize speech. The unit-vocoder is trained using a generator G and discriminator network D. During training, the generator is updated with reconstruction loss based on the mel-spectrogram of the true and synthesized waveforms. Additionally, adversarial loss and feature-matching loss are added to enhance the fidelity of generated speech. To train the discriminator, Polyak et al. [38] follows the minmax objective from
16

---

GAN [11]. After training, the discriminator is discarded and only the generator is used to synthesize waveforms.
We follow [32] to use a unit-vocoder that is also trained with a duration predictor so that it can synthe- size speech with reduced speech units. For more details, refer readers the discussion from the original paper [32, 41], as well as the open-sourced repository: github. com comm/facebookeesearcl fairseq/ eeaeee/exaapleseee Sttpamppppsspeeeeeeeeeeeceeee
E
Limitations and Broader Impacts
The major limitation of the work is the data pre-processing cost from DIFFNORM, as our method requires inferencing diffusion model on all training samples obtain better speech units. With the CVSS benchmark, both En-Es and En-Fr have less than IM pairs, which can be processed easily with our resource (4 A100 GPUs). However, when dataset scales up, the corpus distillation cost could be large.
Limitations:
Our system helps bridge the communication gap between users of different languages, However, as our system involves speech synthesis, it also has risk of being used to mimic particular speaker.
Broader Impacts:
F Ablation on Classifier-free Guidance
In this section, we compare non-autoregressive transformers trained with and without classifier- guidance (CG). We train CMLM models with classifier-free guidance and evaluate them with 5, 10, 15 iterations of decoding. From Table 8, we observe that CG improves the quality of translation whether the model is trained on original or normalized speech units. We find CG brings more improvement on the original units than the normalized units. This happens because normalized speech units more conformed and already result in a large improvement in their translation quality, making the regularization effect from CG less obvious. Nevertheless, the best pest-performin system is achieved with both DIFFNORM and CG.
Comparing different hyperparameters w, we find a small value like 0.5 or brings the most improvement empirically, and such improvements are more noticeable when the number of decoding iterations is larger. For example, comparing the results under and 15 iterations, find w 0 gives better results when the number of iterations is small while w = 0.5 obtains the best performance with 15 iterations.
Model
En-Es
En-Fr
5
10
15
5
10
15
CMLM
9.45
11.85
12.58
12.73
14.78
15.62
CMLM + CG (w=0)
12.22
15.58
16.62
12.13
15.23
16.65
CMLM + CG (w=0.5)
11.37
15.51
17.06
11.63
15.39
16.89
CMLM + CG (w=l)
11.27
15.4
16.99
11.52
15.44
16.65
CMLM + CG (w=2)
10.98
14.97
16.57
11.12
15.04
16.50
CMLM + CG (w=3)
10.63
14.7
16.22
10.80
14.71
16.17
CMLM + DiffNorm
14.84
17.75
18.96
14.03
16.33
17.27
CMLM + DifiNorm CG (w=0)
14.69
17.57
18.63
13.03
16.05
17.13
CMLM + DiffNorm + CG (w=0.5)
15.02
18.24
19.49
13.48
16.53
17.54
CMLM + DifiNorm CG (w=l)
14.77
18.18
19.37
13.29
16.27
17.48
CMLM + DiffNorm + CG (w=2)
14.03
17.65
18.9
12.8
15.97
17.15
CMLM + DiffNorm CG (w=3)
13.45
17.09
18.5
12.23
15.44
16.52
Table 8: Speech-to-speech translation performance of CMLM models with different CG hyperparame-
ters.
17

---

G Experiment Result Tables
Model
Number of Iterations
3
5
7
19
15
CMLM
7.02
9,45
10.66
11.85
12.58
+ CG
7.17
11.37
13.59
15.51
17.06
+ DiffNoom
11.71
14.84
16.42
17.75
18.96
+ DifINNom + CG
10.96
15.02
16.83
18.24
19.49
Table 9: Experimental Results of different En-Es speech-to-unit translation systems.
Model
Number Iterations
3
5
19
15
CMLM
10.1
12.73
13.64
14.78
15.62
CMLM + CG
7.33
11.63
13.71
15.39
16.89
CMLM + DiffNorm
11.6
14.03
15.24
16.33
17.27
CMLM + DiffNorm + CG
9.14
13.48
15.24
16.53
17.54
Table 10: Experimental Results of different En-Fr speech-to-unit translation systems.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We propose strategies to improve speech-to-spest translation which is covered in our abstract and introduction sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See Appendix E for our discussion limitations.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well- sellsspecifiction, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
18

---

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general. empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-ters system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknow acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: There is no theoretical proof involved in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: We provide details of our implementation (architecture and training hyper-parameters) in Appendix B and Appendix C. We will also release our code and the dataset we used (CVSS) is publicly available at petpp::/githhe com/
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may
19

---

be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis-
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The data is publicly available at: https://github com/ wawpercccddaatttss/e/eeeaee The code is publicly accessible at:
bttps://gitahe com//tteventta0010//fffooo.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
Sudides/CodeSuumng wwwssssssioiiiioyoo oolicyyy for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nips. cctppblic////pppcccccccccccccclt wwtpc//gg/////eeeeeeceecccceeee/
/ttgg//e////eeuummiiiiinnn/e Swt////ooooSiissseeeeeea wswonnollll onoooicy))) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question:
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
20

---

Justification:
We provide all our implementation detail in Appendix B and Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We follow the standard evaluation setting, using the same dev and test set as prior work, and evaluate our method two language directions (English-Spanis) and (ngllis--French Our results consistent over languages, and our ablation studies that introduce additional experiments also confirm our method's effectiveness. Lastly, we improve upon the baseline with notable +7 BLEU increment for En-Es and +2 BLEU increment for En-Fr over a test set that has more than 10,000 samples. With the evidence above, we believe our improvement solid and alleviates the need re-run experiments for significance tests. We believe our practice is standard and follows from previous work on speech-to-speech translation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: As noted in Appendix B and Appendix C, we use 4 A100 GPUs with Distributed Data-Paralle setting throughout our experiments.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
21

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: //eurri/////..../e/e/e/ cc tttblliee/ehccsclideeeeeees
Question:
Answer:
[Yes]
Justification:
We follow the Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consider- ation due to laws or regulations in their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See Appendix E for our discussions.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications. let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality generative models could used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly. harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work focuses on direct our model has a high risk for misuse as it does not support voice cloning or other high-risk applications.
translation and we do not believe
22

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cite and describe the used dataset or models throughout the paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcee thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification:
We do not have new assets this paper.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
23

---

Answer:
[NA]
Justification:
No human study is involved in this research.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No human subjects or crowdsourcing are involved.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "JMXB7i32FT", "reviewer": "Reviewer_g4Z7", "summary": "The authors introduce a process aiming to simplify the target distribution of speech-to-speech translation. This process uses a VAE model to map features to a latent space, followed by a diffusion model to normalize the features in the latent space. The authors use the generated dataset to train a non-autoregressive CMLM model on the CVSS-C dataset to validate its effectiveness.", "questions": "1. A simple CG method in En-Es can bring an improvement about 4.5 BLEU (Table 2, Line 7), any reasons behind it?\n2. In Table 4, Acc-Rec, BL-Rec, and BL-Dn seem highly correlated when applying your method. BL-Dn performs well with minor perturbation and vice versa. However, when there is no perturbation, BL-Dn degrades dramatically. What is the reason for this? Is there a borderline value of $T$ that separates these two regions? I think this exploration is important for understanding the effectiveness of using the diffusion process to normalize speech.", "limitations": "The authors have addressed the limitations in Appendix E.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The authors propose a novel speech normalization method using Denoising Diffusion Probabilistic Models.\n2. The quality gain in En-Es direction is impressive.", "weaknesses": "1.  I am confused by the rationale behind using a diffusion model to normalize the speech representation. The authors first add noise to the speech representation (Forward Process) and then remove the noise (Backward Process). This design seems awkward to me. Why do you think this process of adding and then removing noise can help with normalization?\n2. Using a Variational Autoencoder to map features to a latent space seems contradictory to the motivation of reducing data multimodality. The VAE provides an indefinite mapping, which may hinder efforts to reduce multimodality. Additionally, $z_0 = f(h; \\theta_{enc})$ is not correct; the VAE provides a distribution over the latent space. A more suitable expression would be $z_0 \\sim p(z; f(h; \\theta_{enc}))$.\n3. The experiments are only conducted on synthesised dataset. As the paper's contribution is to alleviate the multi-modality problem in data, conducting experiments on a real speech-to-speech dataset, like [1], is much better to support the major claims.\n4. The baseline results (Conformer in EnEs) seem quite low compared to the reported results in the literatre.\n\n[1] Lee et al., Textless speech-to-speech translation on real data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the time and effort you've invested in reviewing our paper. We are grateful for your recognition of our approach's novelty and the significant improvements it offers. Below are our responses to your comments:\n\n> Why do you think this process of adding and then removing noise can help with normalization?\n> \n\nBy injecting noise and training models to denoise, Diffusion Models are capable of recovering noisy features. Due to the acoustic multimodality issue, where the same speech under varying acoustic conditions can exhibit slightly different speech features (refer to Figure 1(b) in [2] for examples), these models are particularly useful. Once well-trained, a Diffusion Model uses the prior information from the training data to reconstruct speech features. This is visually represented in Figure 5, where the more noise is injected, the more pronounced the \"smooth\" effect becomes. Thus, by corrupting the original audio and reconstructing it, we can mitigate its acoustic variation. Additionally, this process helps remove background noises when the original audio is noisy. Lastly, the self-supervised nature of diffusion-based normalization makes it widely applicable, as it only requires monolingual speech data and avoids the need for manual rules. This advantage was also acknowledged by reviewer AuR5.\n\nBeyond the normalization use case, the forward-backward process of the Diffusion Model has shown great potential in producing more robust and consistent features for other tasks as well, such as adversarial robustness [3], speech enhancement [6], and more.\n\n> The VAE provides an indefinite mapping, which may hinder efforts to reduce multimodality.\n> \n\nThank you for the insightful discussion on VAEs! We agree that VAEs transform inputs into a probability distribution over a latent space which provides an indefinite mapping. However, such latent space is smoother and more regularized due to the Gaussian constraints. In the end, whether VAEs help reduce the multimodality depends on the nature of input data. We believe your point is valid that VAE does not guarantee the reduction of multimodality **given very complex data distribution.** However, in the context of speech translation, where prosody information is generally consistent, we utilize the regularized latent space to capture linguistic information and filter out unwanted details, thereby mitigating acoustic multimodality. \n\nAdditionally, we are utilizing a combination of Variational Autoencoders (VAEs) and a Diffusion Model to achieve our objectives. The denoised features produced by the Diffusion Model are cleaner and more robust. Our empirical results have further verified the effectiveness of our DiffNorm strategy, demonstrating its capability to enhance the quality and consistency of the processed speech data.\n\n> A more suitable expression would be 𝑧0∼𝑝(𝑧;𝑓(ℎ;𝜃𝑒𝑛𝑐))\n> \n\nThanks for pointing out the notation issue, we have corrected the formula in the draft. \n\n> conducting experiments on a real speech-to-speech dataset, like [1], is much better to support the major claims\n> \n\nThank you for the suggestion. Due to resource constraints, we are unable to conduct further experiments on the larger-scale data used by Meta AI's researchers in [1]. However, we would like to highlight the following points:\n\n1. We have benchmarked our method against the speech normalization method proposed in [1] using the same dataset (CVSS). Our results, as shown in Table 2, demonstrate that our method is more effective.\n2. A significant portion of the data in [1] is mined S2ST data following [4], sourced from the same CommonCrawl project as the French and Spanish data used in our study. This similarity in data sources leads us to anticipate that our diffusion-based normalization will also be effective on the dataset used in [1].\n3. Additionally, we have conducted extra experiments for the French-English (Fr-En) translation direction and have provided additional data points that further validate the effectiveness of our method. For detailed results on Fr-En, please refer to our general response to all reviewers.\n\n> The baseline results (Conformer in EnEs) seem quite low compared to the reported results in the literature\n> \n\nAs indicated in the caption of Table 2, our baseline results for the Conformer-based model are sourced from the original TRANSPEECH paper (https://arxiv.org/pdf/2205.12523), specifically corresponding to the Basic Conformer Model (ID=3 in Table 1 of the TRANSPEECH paper). **Could you please provide clarification regarding the differing results you have observed in the literature?**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C811fPDYi6", "reviewer": "Reviewer_AuR5", "summary": "This paper proposes DiffNorm, a diffusion-based self-supervised method for speech data normalization, aiming to alleviate multimodal problem in non-autoregressive speech-to-speech translation (NAT). DiffNorm consists of a VAE to reconstruct the speech feature and a diffusion model to add and remove noise of latent vector. Experiment shows that DiffNorm significantly improve NAT translation quality compared to baselines.", "questions": "1. Issues listed in weakness.\n2. What is the role of VAE here? Can we drop it and directly use a diffusion model here for denoising? \n3. Table 3 mentions larger dimension of latent vector brings better performance, why not use the original number of dimensions instead of compressing it?\n4. Why improvement on En-Fr is not that significant compared to En-Es? How does language direction impact the performance?\n5. Is there way to visualize the noise added on the latent vectors? Like what does it mean for the audio?", "limitations": "Besides what I have already mentioned in the weakness, experiments in the paper are only conducted on En-X, but not reverse. Also, it would be interesting to see how DiffNorm works on non-European languages like Chinese and Japanese.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. It is surprising to see pure data normalization improves NAT so much. ASR-BLEU improves 7 BLEU on En-Es direction with DiffNorm. \n2. This technique has wide applicability. All NAT S2S unit-based models can potentially benefit from it. Not to mention it requires no handcrafting rules for noise injection.\n3. Ablation studies on noise level and training of DiffNorm further provide users a general guide on how to adapt DiffNorm to their own dataset and model.", "weaknesses": "1. The multi-modal problem has two aspects: semantic and acoustic. DiffNorm seems able to reduce acoustic modalities. Unclear if DiffNorm can also do that on semantic modalities, i.e., multiple feasible translations for the same source input.\n2. Classifier-free guidance combined with DiffNorm leads to worse performance than DiffNorm alone in Figure 4 and the authors ignore it in the text. Elaboration is needed here.\n3. Lack baseline comparison with several latest S2ST models like TransSentence [1], PolyVoice [2], SeamlessM4T and etc.\n\n[1] TranSentence: speech-to-speech Translation via Language-Agnostic Sentence-Level Speech Encoding without Language-Parallel Data.\n[2] PolyVoice: Language Models for Speech to Speech Translation\n[3] SeamlessM4T—Massively Multilingual & Multimodal Machine Translation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the time and effort you have dedicated to reviewing this paper. We are excited to see that you find our method effective and widely applicable! Below are our responses to your comments:\n\n> DiffNorm seems able to reduce acoustic modalities. Unclear if DiffNorm can also do that on semantic modalities\n> \n\nLike previous studies [1,2], DiffNorm also targets acoustic modalities. By injecting random Gaussian noise into all positions, DiffNorm effectively removes background noise and simplifies acoustic variations while preserving linguistic information. However, due to its design, DiffNorm is unable to simplify semantic modalities. Addressing this limitation, we plan to explore methods to simplify semantic modalities for speech units in future work, with using an auto-regressive model to generate output as a potential starting point [3,4]. Although the challenge of semantic multimodality persists, we believe that mitigating acoustic variations is crucial, and we have demonstrated significant improvements with our method.\n\n> Classifier-free guidance combined with DiffNorm leads to worse performance than DiffNorm alone in Figure 4 and the authors ignore it in the text\n> \n\nThank you for bringing this to our attention! We will include a discussion on this issue in our revised draft. As you may observe from Figure 4 and Table 7 (in the Appendix), CG performs poorly when the number of decoding iterations is small (e.g., 5 iterations). This occurs because, during inference, the logits are slightly perturbed in each decoding iteration (refer to eq.7). In the early decoding iterations, the probability difference ($p_{orig} - p_{uncond}$) may provide poor guidance, **as $p_{uncond}$, modeled from a sequence of [MASK] tokens, is essentially random**. In later iterations, $p_{uncond}$ becomes more useful because the sequence is already partially filled. When the number of iterations is larger, only a few most confident predictions are retained, allowing the model to disregard the early iteration’s negative influence from $p_{uncond}$ . However, when the total number of decoding iterations is small, more incorrect predictions are retained, which negatively impacts future generation.\n\n> Lack baseline comparison with several latest S2ST models like TransSentence, PolyVoice, SeamlessM4T and etc\n> \n\nThank you for bringing these notable works to our attention. We will incorporate a discussion to compare with them in our revised draft. Overall, we believe our contribution is orthogonal to these baselines.\n\nFor TransSentence, they employ an auto-regressive model based on the Transformer architecture, with its major novelty being the use of language-agnostic sentence-level encoding. Since they tested on the same dataset as ours, we can directly compare our results with theirs. As indicated in their result table (Table 2), our performance is superior: En→Fr (17.54 vs. 14.69), En→Es (19.49 vs. 18.9), and Fr→En (19.53 vs. 16.59).\n\nFor PolyVoice and SeamlessM4T, both utilize much larger training datasets and are based on autoregressive modeling and utilize multitask training (though SeamlessM4T incorporates NAR T2U decoding). Given these differences, a direct comparison with these systems may not be fair. We believe our contributions in speech normalization and CG regularization for non-autoregressive speech-to-speech translation are distinct from these baselines.\nNevertheless, we appreciate your pointing out these related works, and we will ensure to add a more thorough discussion in our paper to address these comparisons.\n\n> What is the role of VAE here? Can we drop it and directly use a diffusion model here for denoising?\n> \n\nThe role of the Variational Autoencoder (VAE) in our framework is twofold: (1) to reduce the feature dimension, making it more efficient for diffusion model training, and (2) to regularize the latent space with Gaussian constraints, which is crucial for the training of the diffusion model. The necessity of the VAE, particularly (2), is underscored in our ablation study. As demonstrated in Table 3, the reconstruction quality is significantly compromised when the Gaussian constraint is not applied, highlighting the essential role of the VAE in our methodology.\n\n\n> Table 3 mentions larger dimension of latent vector brings better performance, why not use the original number of dimensions instead of compressing it?\n> \n\nYes, technically it is possible to train a Variational Autoencoder (VAE) that maps features to their original dimension, but this approach would be highly inefficient. The larger the dimension, the more computationally expensive it becomes to train and perform inference with diffusion models. Utilizing the original dimension would significantly prolong the training process and increase costs for the diffusion model, with only marginal improvements in performance. Therefore, we have limited our experiments to a maximum dimension of 128.\n\nMoreover, while a dimension of 128 provides the best reconstruction quality, using a smaller dimension for the latent space could potentially achieve similar downstream results. This can be accomplished by appropriately adjusting the T value, which controls the amount of noise injection. We plan to explore the impact of varying the latent dimension and adjusting the T value in future research to optimize performance and efficiency.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Cef8WdKG17", "reviewer": "Reviewer_DRdm", "summary": "This work builds on TRANSPEECH (Huang et al., 2023) by applying the diffusion method to reduce noise, thereby normalizing speech units for further generation. The authors further use classifier-free guidance to enhance non-autoregressive generation. They conduct experiments on CVSS En-Fr and En-Es datasets, comparing their methods with the baseline. Both proposed methods show improvements.", "questions": "1. How does the performance fare on the CVSS Fr-En task?\n2. In my opinion, the goal of normalizing the unit is to disentangle linguistic information from noisy speech features. If you use DDPM for this process, the target $z_0$, which is the output of the VAE encoder, should be the linguistic feature. How do you ensure that $z_0$ represents the desired linguistic information?", "limitations": "The authors appropriately state the limitations and broader impacts of this work.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This work proposes a diffusion method to normalize target speech units, which outperforms previous approaches.\n\n2. It explores classifier-free guidance to improve NAT generation for the speech-to-speech translation task.", "weaknesses": "1. The paper overstates its contributions. The authors claim to be the first to apply diffusion in the speech-to-speech translation task. However, they merely use it to generate auxiliary training targets and still follow the S2UT strategy (Lee et al., 2022b). This diminishes the novelty of the paper.\n2. Although the work is based on TRANSPEECH, the authors compare only two of the three translation directions. One of the core contributions is applying the diffusion method to normalize speech units, yet the method shows minimal improvement (only 0.3 BLEU) compared to BiP (Bilateral Perturbation) in the En-Fr task. Given the method significantly outperforms others in the En-De task, this result is perplexing.\n3. For a minor suggestion, in Tables 3 and 4, ensure consistent significant figures. Additionally, it would be more appropriate to place Table 3 before Table 4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the time and effort you have dedicated to reviewing this paper. We are glad that you find our diffusion-based normalization and regularization strategy effective. Below are our responses to your comments:\n\n> However, they merely use it to generate auxiliary training targets and still follow the S2UT strategy. This diminishes the novelty of the paper.\n> \n\nWe believe that innovative data-centric methods represent significant contributions to the field, even if the diffusion model is not directly employed as the speech-to-speech translation model. In fact, it is explored and known that the diffusion model struggles with tasks that have weak source-target dependencies, such as translation [3].\n\nOur method distinguishes itself from previous normalization techniques [1,2] by requiring only target-side information and being learned in a self-supervised manner. Through empirical studies on En-Es, En-Fr, and the recently added experiments on Fr-En, our methods have consistently demonstrated substantial improvements for non-autoregressive S2UT models. As reviewer AuR5 noted, “This technique has wide applicability. All NAT S2S unit-based models can potentially benefit from it. Not to mention it requires no handcrafting rules for noise injection.” This endorsement underscores the potential and versatility of our approach in enhancing speech translation models.\n\n> The method shows minimal improvement (only 0.3 BLEU) compared to BiP (Bilateral Perturbation) in the En-Fr task. Given the method significantly outperforms others in the En-De task, this result is perplexing.\n> \n\nThe effectiveness of our normalization strategy is indeed dependent on the dataset. It is important to emphasize that our method should **primarily be compared with the CMLM baseline, rather than CMLM + BiP**, because our contribution lies in a self-supervised normalization strategy combined with a regularization strategy. These are completely orthogonal to BiP’s rule-based normalization approach. Our best system achieves an ASR-BLEU improvement of approximately 7 points on En-Es, 2 points on En-Fr, and 2.5 points on Fr-En over the CMLM baseline. In contrast, the improvement from BiP over the CMLM baseline is typically around or less than 1 point.\n\nWhen comparing the improvements between En-Es and En-Fr, the 7-point increase for En-Es is indeed very impressive. However, this does not diminish the significance of the 2-point improvement on En-Fr. We believe three factors contribute to the substantial gain observed in En-Es:\n\n1. **Dataset Size**: The En-Es dataset is smaller, which means that the impact of normalization on the dataset is more pronounced, leading to more noticeable improvements.\n2. **Speech Length**: Spanish speech is, on average, longer than French speech, as indicated in Table 1. Longer sequences could be more susceptible to acoustic variations, which our normalization method effectively mitigates.\n3. **Background Noise**: Spanish speeches from the CommonVoice dataset also contain background noise. Our proposed normalization strategy can remove this noise, thereby enhancing the clarity and quality of the speech data and benefiting the training of NAT models.\n\nThese factors combined suggest that while our method is broadly effective, its impact is particularly significant in scenarios where the data exhibits specific challenges that our approach directly addresses.\n\n> For a minor suggestion, in Tables 3 and 4, ensure consistent significant figures. Additionally, it would be more appropriate to place Table 3 before Table 4.\n> \n\nThanks for the suggestion! We will adjust it accordingly to make it more consistent and appropriate.\n\n> How does the performance fare on the CVSS Fr-En task?\n> \n\nThanks for the question. We kindly refer you to our response to all reviewers, where we show the result for Fr-En task as this is requested by multiple reviewers. Consistent with our paper’s results, DiffNorm also improves upon baseline CMLM by more than 2.5 ASR-BLEU on Fr-En (and more than 1.5 ASR-BLEU compared to CMLM+BiP). We hope this new set of results could further validate the effectiveness of our method and address your concerns.\n\n> How do you ensure that 𝑧0 represents the desired linguistic information?\n> \n\nSince the Variational Autoencoder (VAE) and the diffusion model are trained in a self-supervised manner on the target feature, they achieve high reconstruction quality and maintain linguistic information when properly trained. This is evidenced by the ASR-BLEU scores for the reconstructed units (BL-Rec column in Table 4), which show that transcription from reconstructed speech units achieves performance comparable to the original units (48.5 vs. 48.7 for Spanish and 40.38 vs. 40.64 for French speech). If the reconstructed units failed to preserve linguistic information, the speech synthesized from such units would exhibit significantly worse ASR-BLEU results.\n\nIt is important to note that CVSS-C is not a large dataset. By expanding the dataset size, we anticipate even higher quality reconstruction and a more pronounced normalization effect that reduces acoustic variations while preserving the linguistic content of the speeches.\n\n----\n\nReferences:\n\n[1] Lee et al., (2022). Textless speech-to-speech translation on real data.\n\n[2] Huang et al., (2023). TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation\n\n[3] Tan, X. (2024). Lessons From the Autoregressive/Nonautoregressive Battle in Speech Synthesis.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: 2 weak accepts, 1 borderline accept, 1 accept
strengths: interesting idea, wide applicability, solid improvements
weankesses: experiments can be extended to other tasks, novelty of paper can be improved.

still given the flaws i think this paper is wrong enough to accept.
Decision: Accept (poster)