DiffS2UT: A Semantic Preserving Diffusion Model for Textless Direct Speech-to-Speech Translation
Authors: Yongxin Zhu
Zhujin Gao
Xinyuan Zhou
Ye Zhongyi
Linli Xu
Venue: EMNLP 2023 Main (2023)
Keywords: Speech to Speech Translation
Diffusion Models

Abstract: While Diffusion Generative Models have achieved great success on image generation tasks, how to efficiently and effectively incorporate them into speech generation especially translation tasks remains a non-trivial problem. Specifically, due to the low information density of speech data, the transformed discrete speech unit sequence is much longer than the corresponding text transcription, posing significant challenges to existing auto-regressive models. Furthermore, it is not optimal to brutally apply discrete diffusion on the speech unit sequence while disregarding the continuous space structure, which will degrade the generation performance significantly. In this paper, we propose a novel diffusion model by applying the diffusion forward process in the continuous speech representation space, while employing the diffusion backward process in the discrete speech unit space. In this way, we preserve the semantic structure of the continuous speech representation space in the diffusion process and integrate the continuous and discrete diffusion models. We conduct extensive experiments on the textless direct speech-to-speech translation task, where the proposed method achieves comparable results to the computationally intensive auto-regressive baselines (500 steps on average) with significantly fewer decoding steps (50 steps).

DiffS2UT: A Semantic Preserving Diffusion Model for Textless Direct Speech-to-Speech Translation
Yongxin Zhu 1.3 Zhujin Gao?.,, Xinyuan Zhou', Zhongyi Ye², Linli Xu2.3 School of Data Science, University of Science and Technology of China School of Computer Science and Technology, University of Science and Technology of China State Key Laboratory of Cognitive Intelligence
'iFyytek Research
(zyx2016, gaozhujin]@mate wstc. edu. cn, linlixu@sstc.edu.cm (xyzhou15, zyye7jeiflytek. com
Abstract
would cause the problem of error propagation and incur expensive computational costs. On one hand, text-based cascaded systems face challenges when dealing with low low-resource languages that lack text annotations or even without written systems (Chen et al., 2022a). On the other hand, the speech-to- text process disregards acoustic information such as accentuation, attitude, and emotional nuances, which are crucial for effective human spoken com- munication.
While Diffusion Generative Models have achieved great success on image generation tasks, how to efficiently and effectively incor- porate them into speech generation especially translation tasks remains non-trivial problem. Specifically, due to the low information density of speech data, the transformed discrete speech unit sequence much longer than the corre- sponding text transcription, posing significant challenges to existing auto-regressive models. Furthermore, it is not optimal to brutally apply discrete diffusion on the speech unit sequence continuous space struc-
Recent works of textless direct S2ST models (Tjandra et al., 2019; Kano al., 2021; Jia et al., 2019, 2022; Zhang et al., 2021; Lee et al., 2021, 2022; Popuri et al., 2022; Wei et al., 2022) directly translate speech to speech in an end to end manner without the intermediate generation steps relying on text transcripts. Among them, S2UT model (Lee et al., 2021, 2022) takes the advantage of recent progress in spoken language modeling (Lakhotia et al., 2021) to obtain the discrete speech repre- sentations, or discrete units, to build the textless S2ST systems. The entire system consists of a sequence-to-ssquuence transformer (Vaswani et al., 2017) with a speech encoder and an auto-regressive unit decoder, followed by unit HiFi-GAN vocoder (Polyak et al., 2021) trained separately for unit-to- waveform conversion. Experiments (Chen et al., 2022a) have demonstrated notable improvements in translation for unwritten languages. Despite the promising translation capabilities of the aforemen- tioned auto-regressive-ban based S2ST models, they suffer from O decoding time complexity for predicting one token per step. Furthermore, the learned discrete speech unit sequences are typically much longer than the corresponding text transcripts due the low information density of speech data, with a rate of 50 units per second (sequence length up to 500 on average), where the generation latency bottleneck is exposed.
while disregarding ture, which will degrade the generation perfor- mance significantly. In this paper, we propose a novel diffusion model by applying the diffusion forward process in the continuous speech repre- sentation space, while employing the diffusion backward process in the discrete speech unit space. In this way, we preserve the semantic structure of the continuous speech representa- tion space in the diffusion process and integrate the continuous and discrete diffusion models. We conduct extensive experiments on the text- less direct speech-to-spect translation task. where the proposed method achieves compa- rable results the computationally intensive auto-regressive baselines (500 steps on aver- age) with significantly fewer decoding steps (50 steps).
1 Introduction
srrech-to-speede translation (S2ST) aims trans- lating speech of one language to speech another language, breaking the communication barriers be- tween people around the world who speak differ- ent languages. Conventional cascaded systems (Lavic et al., 1997; Nakamura et al., 2006) for S2ST typically consist of automatic speech recognition (ASR), machine translation (MT) or end-to-end speech-to-text translation (S2T), followed by text- to to-specch synthesis (TTS). However, integrating multiple separated modules into a single system
In the meantime, diffusion generative models -Sohl-Dickstein al., 2015; Ho et al., 2020; Song

---

K-mmeus Map
Discrete
Continuous
k
ASR transpription
more than half of plastics find beaches single plastics more than of placters we arbices are selus plasters
Ap
%
2
matenhaif of the past found on beaches are senro pasturds north
(a)
fondon
X, WS Wy. M3
5 moled oup the prices we alonte bechee ar simily it price
Ao have one apple
10 modon orout the olo brass tesen find fot true tos and e leat prassll celebration te fammy thorough poordnar aganto
[M] one [M]
Xy [M] [M] [M] [M]
Wi.
(d)
(c)
Figure 1: Illustration of different diffusion forward processes. (a) Continuous diffusion images. The image is corrupted gradually by the Gaussian noise. (b) Discrete absorbing diffusion on text sequences. The text sequence is corrupted with masks. (c) Our proposed diffusion process. The discrete units are converted between the continuous space and the discrete space by K-means mapping. The Gaussian noise is added in the continuous space. (d) The ASR transcription of the noisy speech converted from the perturbed units. The units are perturbed by replacing them with the K-NN search in the continuous K-means space.
al., 2020) have made remarkable advancements in generating high-quality images (Rombach et al.. 2022), audios (Popov et al., 2021; Shen et al., 2023), and texts (Austin et al., ; Hoogeboom et al., 2021). Diffusion generative models are typi- cally defined as a forward process and a backward process. The forward process gradually corrupts the data with noise, while the backward process starts with the pure noise and performs ancestral sampling to denoise the corrupted data. We com- pare different diffusion forward processes for spe- cific data structures in Figure 1. For continuous data like audios or images in Figure I(a), current diffusion models (Ho et al., 2020) typically add Gaussian noise gradually to the pixel space or au- dio spectrogram space (Popov et al., 2021). For discrete data like texts in Figure 1(b), current diffu- sion models (Austin et al., 2021; Hoogeboom et al., 2021) perturb the data with absorbing or multino- mial transition matrices.
that uses naive discrete diffusion models to gen- erate discrete units for S2ST. Our primary results (see Table 1) demonstrate that sub-optimal brutally apply discrete diffusion on the speech unit sequences, which would degrade the generation performance by large margin. Typically, the vo- cabulary for the text translation task is constructed by BPE (Sennrich et al., 2016), and the semantics of the tokens require large amount of training data to learn. In contrast, discrete speech units quired by the K-means clustering algorithm in the continuous speech representation space learned by the self-supervised learning models. Consequently, structural relationships exist among these units nat- urally. As shown in Figure I(d), we perturb discrete speech units by replacing them with k-NN search in the continuous K-means space, and then convert them to the waveform with the vocoder described in Section 3.2. We transcribe the noisy speech with the open-sourced ASR system to "see" the pronun- ciation. We find that when A is small, the noisy speech sounds similar to the original speech. The results show that the close K-means centroids share similar acoustical meanings.
However, speech is unique modality that lies between audio and text. On one hand, speech is a continuous form of acoustic data represented as a waveform. On the other hand, speech encom- passes high-level linguistic information similar to discrete text data and can be represented as dis- crete units with SSL models, which is crucial for the translation task. It is challenging to directly apply the diffusion forward process like images texts for S2ST. For example, Translatoron (Jia al., 2019) directly generates the continuous spec- trogram for speech translation but the performance is significantly inferior the discrete unit based model S2UT (Lee et al., 2021). We conduct ex- periments similar to DiffSound (Yang et al., 2023)
To tackle these challenges, this work, we propose a new paradigm called DiffS2UT for the S2ST task, a novel diffusion generative model that bridges the continuous representation space and discrete unit space of speech, while simultaneously preserving the semantic structure of the continuous speech representation space well. Specifically, as depicted in Figure (cc), there exists an one-to- one map between the discrete units and the con- tinuous K- K-means centroid vectors. Therefore, the diffusion forward process, we first map a unit

---

the continuous vector derived from the learned K-means space. Next, we gradually add Gaussian noise to corrupt it with the continuous diffusion process. Subsequently, the noisy embedding vector is converted into the discrete unit index through the nearest neighbor search. As a consequence, we obtain the perturbed discrete unit sequences which are derived from the semantically structured continuous space, rather than being uniformly sam- pled from the vocabulary in (Hoogeboom et al., 2021). The perturbed unit sequences are semanti- cally close to the unperturbed ones when the added noise is small, which cannot be achieved in the dis- crete diffusion process. In the diffusion backward process, we train the model to predict the uncor- rupted discrete unit sequences with cross-entropy in parallel.
aims translating the source speech to the tar- get speech directly without any text intermediates. Translatoron (Jia et al., 2019, 2022) is the first S2ST model to translate speeches by generating spectrograms directly. Tjandra et al. (2019); Zhang et al. (2021) build the S2ST system by pre-training VQ-VAE model to convert target speeches into discrete codes and learn speech-to-code transla- tion model. S2UT model (Lee et al., 2021) uti- lizes the self-supervised speech encoder HuBERT (Hsu al., 2021) pre-trained on large corpus of unlabeled speeches to convert speeches into dis- crete units with K-means clustering, which out- performs the VQ-VAE based approach in (Zhang et al., 2021). Lee et al. (2022) extend the S2UT model to the real-word S2ST data VoxPopuli-S2S (Wang et al., 2021) and propose a speech nor- malizer to obtain the speaker-invariant representa- tions. Duquenne et al. (2022) propose large-scale multilingual S2ST corpus with the speech min- ing method (Duquenne et al., 2021). By introduc- ing the pre-trained models including the wav2vec (Baevski et al., 2020) encoder and mBART (Liu et al., 2020) decoder to the S2UT model, the trans- lation quality is further boosted in (Popuri et al., 2022). TranSpeech (Huang et al., 2023) alleviates the acoustic multi-modal problem with bilateral perturbation. Speech2S (Wei et al., 2022) performs joint pre-training on speech and text datasets to align the acoustic and textual modalities. Although the above methods achieve good performance in the S2ST task, they are built on the auto-regressive generation models and suffer from the slow gener- ation speed due the excessive lengths of speech sequences. Our work leverages diffusion gener- ative models to speed up the translation process without compromising the translation quality.
We evaluate our framework DiffS2UT on the real-world S2ST datasets (Wang et al., 2021; Munzo-Sáncher et al., 2020). The model achieves 14.8/15.2/14.5/15 13.6 BLEU score on the Eurapal- BTTE--En/TT-EE/En-E///Ee
test sets, significantly surpassing the vanilla diffusion models, and achiev- ing comparable performance to auto-regressive models while requiring much fewer generation steps.
To summarize, the main contributions of this paper are:
To the best of our knowledge, this is the first work that effectively introduces diffusion gen- erative models to the textless S2ST task. The model significantly accelerate the generation process (50 steps v.s. 500 steps on average), while maintaining the generation quality com- parable to the auto-regressive models.
We propose a novel diffusion framework that integrates the continuous and diserete diffu- sion models by decoupling the diffusion for- ward and backward processes.
Diffusion Probabilistic Models Diffusion gener- ative models (Sohl-Dickstein al., 2015; Ho al., 2020; Song al., 2020) have achieved remarkable progress in generating high-quality images (Rom- bach et al., 2022), audios (Popov et al., 2021; Shen et al., 2023), and texts (Austin et al., 2021; Hooge- boom et al., 2021). DDPM (Ho et al., 2020) defines the generative process as the reverse of partic- ular Markovian diffusion process. By gradually corrupting an image with Gaussian noise and train- ing model based on U-Net denoise it, DDPM can generate high quality images. Multinomial diffusion (Hoogeboom et al., 2021) proposes a uni- form corruption process for categorical variables,
The proposed diffusion model DiffS2UT bridges the continuous and discrete spaces of speech and preserves the semantic structure of the speech representation. Experimental results demonstrate its superiority over vanilla diffusion models.
2 Related Work
Textless Direct Speech to Speech Translation Textless direct speech-to-speech translation (S2ST)

---

which is extended by D3PMs (Austin et al., 2021) support general transition matrices, including an absorbing variant that draws close connections to masked LMs. Recent works (Gong et al., 2022; Gao et al., 2022) aim at conducting the Gaussian diffusion process over token embeddings so that the continuous diffusion models can be applied to discrete texts. Some works (Chen al., 2022b) consider converting discrete tokens bit strings and model them as real values. However, these approaches neglect the semantic relationships be- tween tokens because the vocabulary constructed using BPE, limiting the model performance. In this work, we introduce the diffusion generative models into the S2ST task. We conduct a thorough investigation of the unique properties of the speech modality and propose novel diffusion paradigm decoupling the diffusion forward and backward pro- cess to bridge the discrete and continuous spaces of speech, while preserving the semantic structure of speech representations.
where 14g, Eg are the predicted mean and variance of g(x)-1{Xt), denotes the model parameters. Following the derivation in Ho et al. (2020), Eg is set as of1 and the objective function of the diffu- sion model can be written as a simplified variational lower-bound (VLB) of log (x(x)):
Lvih Exx, (lo (x,,, xo)]
(4)
For the discrete data represented as one-hot vec- tors Xo Pdata (xo) (0, 1)K, the discrete dif- fusion forward process usually involves model- ing noise through multinomial transition (Hooge- boom et al., 2021) and absorbing transition (Austin et al., 2021). The multinomial transition adopts a uniform noise distribution over the vocabulary {1...., K}; alternatively. the absorbing transition specifies noise to be a point mass with all of the probability
absorbing state. In the backward process, the model Py is adopted to predict xo with softmax function, representing the probability distribution for each token. The objective function of the discrete diffusion model can be written as the negative log-likelihood loss:
3 Background
In this section, we start with the formulations of both continuous and discrete diffusion generative models. Then we introduce some preliminaries for the proposed DiffS2UT system, such how convert speech to discrete units and units to waveform.
Lvbb Ex
[xo (x ()],
(5)
3.2 Pre-processing and Past-processing for DiffS2UT
We describe the speech-to- to-u method and the unit- to-speech method below.
3.1 Diffusion Generative Models
The diffusion generative models are typically de- fined as forward process and backward process. Given the continuous data X ~ Pdtta (xo) C Rd. the continuous diffusion forward process usually perturbs it into the Gaussian noise Xy ~ (0, I)
Speech to Unit Lee et al. (2022) pre-train Mul- tilingual HuBERT (mHuBERT) (Hsu al., 2021) on the data combined of En, Es and Fr from the unlabeled speech dataset VoxPopuli (Wang et al., 2021), which contains 4.5k hours of data for En, Es and Fr, respectively. We follow Lee et al. (2022) and use mHuBERT to discretize the target speech. As shown in the left of Figure 2(a), the speech-to- unit discretization method uses the last iteration of the mHuBERT model for feature extraction, fol- lowed by the K-means clustering algorithm. The learned K cluster centroids are used to convert the audio into sequence of cluster indices, which are referred to as units.
through a series of latent variables x1. xI...., xx in Markov transition:
(xx(xx=1) = Nxxi / Bix &I),
(1)
where 3, controls the noise level added at the timestep For any arbitrary sampling x; from Xg can be achieved in closed form with the repa- rameterization trick:
((x:|xo) (xxi V (xx--1,311)
(2)
Unit to Speech As illustrated in the right of Fig- ure 2(a), the unit-to-speech conversion is done with the discrete unit-based HiFi-GAN vocoder proposed (Polyak et al., 2021). The vocoder is trained separately from the S2UT model, with the combination of the generator-discrimiinn discriminator loss
where Br : : 1- (1 (,). In the backward process, the model learns to recover Xo by denois- ing from x, step by step:
Po (o(x-1(xx) = N (x(e t), (xt, t)):
(3)

---

Cross-Entropy Loss
Logits
units
speech
SoftMax
33, 93, 1,4
Transformer Encoder
Transformer
x L
Decoder
K-means
Input Embedding
CNN Block
x M
U(O
Code HiFi-GAN Vocoder
Hubert
33, 93,
source speech
Gaussian Noise
speech
units
noised target units
K-means mapping
clean target units
(b)
Figure 2: Illustration the model architecture. (a) Left: The speech-to-unit model with mHuBERT and K-means discretize the speech input. Right: The unit-based HiFi-GAN vocoder for unit-to-speech conversion. (b) Our proposed DiffS2UT translation model. The speech units are transformed and perturbed between the discrete space and contimuous space for diffusion.
from HiFi-GAN (Kong et al., 2020) and MSE of the predicted duration of each unit the logarith- mic domain.
cluster centroid vectors. And the inversion func- tion g works by conducting the nearest neighbor search the K-means Euclidean space, convert- ing the continuous vector to the K-means cluster indices (umits).
4 DiffS2UT
For each unit sequence the decoder input, we first convert it the continuous space that encapsu- lates the semantic structure learned by SSL models and K-means. Then we perturb with the Gaus- sian noise as in the continuous diffusion forward process, followed by the inversion function con- verting it back to discrete units. At last, the model learns to recover the correct unit sequence with the corrupted units and time step t as the input. The model is optimized with negative lig-likeli objective.
this section, we elaborate our proposed model DiffS2UT detail. We first describe the model ar- chitecture for S2ST, followed by the novel training and sampling algorithms for diffusion.
4.1 Model Architecture
Figure 2(b) depicts the overall architecture of the DiffS2UT system. The entire system consists of a speech encoder and unit decoder. The speech encoder constructed using a CNN-bascl speeeh down-sampling module and a stack of Transformer encoder blocks (Vaswani et al., 2017). The unit decoder consists of an input embedding layer that embeds discrete units and diffusion time steps, fol- lowed by stack of Transformer decoder blocks. Given the source speech the model learns to gen- erate the target unit sequence x [xI,..., xr,] by estimating Pe (x(c). Previous models factor- ize the unit sequence in an auto-regressive man- ner and estimate the units from left to right, i.e., (x([cc) - I (zxl c). In contrast, the diffusion decoder is trained with the non-causal attention mask and estimates all tokens simultanc- ously, which means each token can not only attend the tokens before it but the tokens after, i.c.. po px|c, t) = IT Po (a: /c, t). We denote the K- means mapping function as which can convert the discrete unit indices to the continuous K- K-means
4.2 Training
The training details are described in Algo- rithm 1. Following standard practices in diffu- sion models, we randomly sample a time step t and optimize the model parameter 0 with re- spect to the objective L(0). In our case, [()) x log (x,, which is the KI divergence of Xo and po [x, t). Given unit sequence xo Pdttta (xo) {1,..., ... K)", we first convert it the continuous speech representation vo = g(xo) Iaxd with the K-means mapping function g. Then we sample v, by perturbing Vo with the Gaussian noise as in continuous diffusion models
q(v:/vo) N(vi / 3 (6)
A After that we transform v, back to the discrete

---

Algorithm Training DiftS2UT
Algorithm 2 Sampling from DiffS2UT
Input: network fo. data distribution Pdata (xo), and K-means mapping function g.
Input:
trained network parameters and K-
means mapping function g. decoding steps K.
Output:
model parameters
repeat
Output: generated sample Xo.
Draw X Pdata (xn);
Initialize VT ~ (N I);
Draw E Uni.orm((1......
Convert continuous VT to discrete space XT 9 (VVT);
Convert discrete Xo to continuous space vo
g(x));
Sample subset [I...... rK} from the full dif- fusion trajectory .I....,T;, T):
Draw V2 v(v:
Convert continuous vi discrete space x
for T...... TK do
9 (v:):
Draw Xn- arg max po xxex 1);
((0) - Xo log (x,, , l);
Convert discrete Xo continuous space Vo g(xo):
Minimize [()) with respect to 8;
until converged
Draw Vt-1 ( vo):
Convert continuous v, to discrete space x; 9 (ve);
units x; (v:) E (I...., K} with the inver- sion function g 9_1 which preserves the semantic structure of the speech representation. The pertur- bation X₁ semantically close to the target units Xo with the time step as control.
end for
Return Xo.
1.0
linerr
linerrmm
4.3 Sampling
0.8-
The sampling details are described in Algorithm
Accuracy
2. Sampling in DifTSUT begins with a sequence comprising only continuous noisy vectors VT ~ (O, Then we convert to discrete units XT (VT) with the inversion function g g- At each time step first feed the noisy units x; into the neural network and get the predicted target &y with arg max:
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
Time Step
Xn = arg max po (xt, tt. t).
(7)
Figure 3: Accuracy of nearest neighbour search on the perturbation data in the K-means space.
Since we apply the diffusion forward process in the continuous space, we need to convert the model prediction back to the K-means space Vo (8o) with the K-means mapping function g. Then sample Vi-1 as in continuous diffusion models:
Note that the conversion between the discrete and continuous spaces only requires L- distance computation, which can be accelcrated with the FAISS library (Johnson et al., 2019) so that the time cost is negligible. Equipped with this method, execute the sampling algorithm more effec- tively.
Vz-1 = =V Vo. If
This strategy is more informative, as the noise is added to the semantically structured speech repre- sentation space, where the intensity is controlled by the time step t.
4.4
Noise Schedule
Due to the fact that K-means is robust to small Gaussian noises, the unit sequence would not be corrupted when the added noise small. As shown in Figure 3, the accuracy of the nearest neighbour search on the perturbation data remains 1.0 for the first 20% time steps with the linear noise sched- ule, which is useless since the model can directly
To accelerate the sampling speed, we pick sub- set (T....., T from the full diffusion trajectory {1...., ,T for generation (Nichol and Dhariwal, 2021). Then a reverse step can be obtained by:
Vy, ~ q(x,_) Vre vo)

---

Name
# Iter.
Es-En
Fr-En
En-Es
En-Fr
Param.
Speedup
Cascaded systems:
+f TTS (Lee al., 2022)
n.a.
19.2
19.8
21.7
18.5
n.a.
S2UT systems with auto-regressive generation:
S2UT (Lee et al., 2022)
N
13.1
15.4
16.4
15.8
71M
1.0x
S2UT w/ ED (Duquenne et al., 2022)
20.4
20.7
21.9
19.3
71M
1.0x
S2UT w/ PT (Popuri et al., 2022)
23.8
26.0
827M
S2UT systems with diffusion generation:
S2UT-Absorbing (Austin al., 2021)
50
5.5
5.1
3.9
4.3
71M
12.6x
S2UT-Multinomial (Hoogeboom et al., 2021)
50
11.8
12.0
11.2
8,5
71M
9.7x
13.5
14.1
13.8
12.9
71M
14.4x
10
14.6
15.0
14.2
13.5
71M
14.0x
DinS2UT (Ours)
20
14.8
15.1
14.4
13.6
71M
12,4x
14.8
15.2
14.5
13.6
71M
11.9x
Table 1: BLEU scores achieved by training on VoxPopuli--22S (Wang et al., 2021) training sets and evaluating EuroparI-ST (lranzo-Sánchez al., 2020) test sets. For the S2UT systems with diffusion generative models, the results reproduced by ourselves.
the training set. The statistics of the target unit sequence length in Es-En training set is shown in Figure 4. Different from the text translation data, the S2ST data is much longer with the average length larger than 500, inducing challenges auto-regressive models with the long generation process.
0.0014
0.0010
Density
0.0008
0.0006
0.0004
Evaluation Following prior works (Lee et al., 2022), all speech outputs are decoded with the same open-sourced ASR models! to compute BLEU with respect to the reference transcribed translations using SACREBLEU (Post, 2018).
0.0002
0.0000
500 1000 1500 2000 2500
3000 Length of target sequence
Figure 4: Statistics of the target sequence length in VoxPopuli-S25 Es-En training set.
Baselines For comparison, we choose three S2UT baselines and reproduce two discrete dif- fusion models for S2ST. Among them, S2UT is the baseline model proposed in (Lee et al., 2022). S2UT w/ ED (Duquenne al., 2022) trains the model on the mined S2ST datasets. S2UT PT (Popuri et al., 2022) initialized with the pre- trained wav2vec encoder and unit-based mBART decoder. We reproduce the absorbing (Austin et al., 2021) and multinomial (Hoogeboom et al., 2021) diffusion methods on the S2ST task, named S2UT- Absorbing and S2UT-Multnoomial The implemen- tation details deseribed in Appendix A.I.
copy the input as the target. To force the model to learn to recover the corrupted sequence, we propose a new noise schedule uniform for the DifIS2UT model. We initially set So 0.3 and decrease 1 1 3 uniformly. The proposed uniform schedule improves the level of data utilization.
5 Experiments
5.1 Setup
Datasets We use the real-world dataset VoxPopuli-S2S (Wang et al., 2021) for model training. We examine four directions of language pairs: Spanish-Englist (Es-En), English- Spanish (En-Es), French-English (Fr-En), and English-French (En-IFr) on the set from Europarl-ST (Tranzo-Sánchez al., 2020). In order to avoid duplication with the corpus of the test set, we deleted the data before 2013 in
5.2 Results
Table I summarizes the results from different S2UT systems trained with VoxPopuli-S22 data (Wang et al., 2021). First, by comparing our proposed
'https: / http://tt.ccmmeeeeeeoeeeeerre//a ttre//mainn/eeaamles/speeccc to https//ecccttttpeeeeeeeerr.bbbba

---

Models
BLEU
14.
DifIS2UT
14.8
w/o K-means mapping
12.1
w/o beam size
14.6
10
BLEU
Table 2:
The ablation study on the proposed methods. Results are conducted on the Es-En test set.
6
model with the models trained with conventional discrete diffusion processes 392UT-Absorbing and S2UT-Multinomial) results show that our model achieves improvements of 10.0 and 3.0 in BLEU scores the S2ST tasks. As shown in the results, existing discrete diffusion models struggle per- form well due to the complexity of the S2ST task over text translation. In comparison, our model can better utilize the semantic structures beneath the units through the K-means conversion. Further- more, we also compare with the S2UT model based on auto-regressive generation, and our method is able to achieve comparable results with the base- line model S2UT even with only 10 decoding steps, which boosts the generation speed a lot.
Di152UT
2
worbinb-orbing
S2UT-NNu tinemial
0%
25%
50%
75%
100%
Reverse Process
Figure The BLEU score of intermediate Xo within decoding process.
and evaluate the BLEU scores. The results are illustrated Figure 5. All the diffusion models can achieve high BLEU scores in the first quarter of the reverse process and converge slowly the rest steps. Among them, our DiffS2UT obtains the highest BLEU score in the first quarter, which is even higher than the final score of other diffusion models. According to the observation above, extract the intermediate Xn the final decod- ing results further accelerate the decoding speed with a negligible performance drop.
5.3 Analysis
On the Effect of Semantic Preserving Diffu- sion Process We study the effects of the pro- posed components, which are listed Table 2. can observed that without K-means mapping, DiffS2UT would degenerate to the multinomial dif- fusion and the BLEU score drops to 12.1, which is close to S2UT-Multinomial. We also study the effect of the predicted sequence length and ablate the beam search, we find that the model is slightly sensitive because the vocoder needs to predict the duration of each token which results in the time and speed the final audio.
Conclusion
We present diffusion generative model for the S2ST task named DiffS2UT that integrates the tinuous and discrete diffusion models by decou- pling the diffusion forward and backward processes. The model applies the diffusion forward process in the continuous speech representation space, while employing the diffusion backward process the discrete speech unit space. In this way, we effec- tively preserve the semantic structure of the con- tinuous speech representation space in the diffu- sion process. We evaluate the proposed DiffS2UT model on the real-world specch-to-speech datasets and demonstrate the significantly boosted gener- ation quality compared to the conventional dis- crete diffusion models. Furthermore, the model can achieve comparable results to auto-regressive models but with much fewer decoding steps.
On the Decoding Steps We list the BLEU scores different decoding step settings from steps to 50 steps for all diffusion models Table 1. We can see that the proposed DiffS2UT model achieves the BLEU score of 13.5 with 5 steps, which is even higher than the results of S2UT-Absorbing and S2UT-Multinomial with 50 steps. The results demonstrate the superiority of the proposed model DiffS2UT that it can surpass the conventional dif- fusion models with much fewer decoding steps.
Limitations
On the Intermediate Generation Results To analyze the generation quality during the reverse process, we extract & at some of the reverse steps
While the conversion between the continuous space and discrete space doses not add extra trainable model parameters, does introduce some com-

---

putational overhead. The dimensionality of the K-means space is D 768 and number of clusters is K = 1000. For each batch forward in training, the computation complexity O(B x L x K x D). Although we can accelerate the search process with the FAISS (Johnson et al., 2019) library without sac- rificing the training speed. the GPU memory usage is still not negligible. Developing faster and more efficient nearest neighbor search tools remains an active area of research (Guo et al., 2020).
Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and LingPeng Kong. 2022. Diffuseq: Sequence to se- quence text generation with diffusion models. arXiv preprint arXiv:2210.08933.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor O.K. Li, and Richard Socher. 2018. Non-autorogressive neural machine translation. In International Confer- ence on Learning Representations.
Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chem, and Sanjiv Kumar. 2020. Accelerating scaeescale inference with anisotropic vector quantization. In Proceedings of the 37th Inter- national Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 3887-3896. PMLR.
Acknowledgements
This research was supported by the National Nat- ural Science Foundation of China (Grant No. 62276245), and Anhui Provincial Natural Science Foundation (Grant No. 2008085131).
Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. De- noising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840- 6851.
References
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. 2021. Argmax flows and multinomial diffusion: Learning categorical dis- tributions. In Advances in Neural Information Pro- cessing Systems.
Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. 2021. Structured denoising diffusion models in discrete state-spaces. In Advances in Neural Information Processing Sys- lems.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdel- rahman Mohamed. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-3480
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33: 12449-12460.
Rongjie Huang, Jinglin Liu, Huadai Liu, Yi Ren, Lichao Zhang, Jinzheng He, and Zhou Zhao. 2023, Transpeech: Speech-to- so-cchhht--peechh translation with bi- lateral perturbation. In The Eleventh International Conference on Learning Representations.
Peng-Jen Chen, Kevin Tran, Yilin Yang. Jingfei Du, Justine Kao, Yu-An Chung. Paden Tomasello. Paul- Ambroise Duquenne. Holger Schwenk, Hongyu Gong, et al. 2022a. Speech-to-speech translation for real-world unwritten language. arXiv preprint ...64:2211.0647.
Javier Tranzo-Sánchez, Joan Albert Silvestre- Cerda, Javier Jorge, Nahuel Roselló, Adria Giménez, Al- bert Sanchis, Jorge Civera, and Alfons Juan. 2020. Europarl-st: A multilingual corpus for speech transla- tion parliamentary debates. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8229-8233. IEFF.
Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. 2022b. Analog bits: Generating discrete data us- ing diffusion models with celf-conditioning arXiv preprint arXiv:2208.04200
Paul-Ambroise Duquenne, Hongyu Gong, Ning Dong, Jingfei Du, Ann Lee. Vedanuj Goswani, Changhan Wang, Juan Pino, Benoît Sagot, and Holger Schwenk. 2022. Speechmatrix: A large-scale mined corpus of multilingual speech-to-se to-eeecch translations. arXiv preprint arXiv:2211.0455
Ye Jia, Michelle Tadmor Ramanovich, Tal Remez, and Roi Pomerantz. 2022. Translatotron 2: High-quality direct speech-to-speech translation with voice preser- vation. In International Conference on Machine Learning, pages 11120-10134 PMLR.
Paul-Ambroise Duquenne, Hongyu Gong, and Holger Schwenk. 2021. Multimodal and multilingual em- beddings for large-scale speech mining. Advances in Neural Information Processing Systems, 34:15748- 15761.
Ye Jia, Ron J Weiss, Fadi Biadsy, Wolfgang Macherey, Melvin Johnson, Zhifeng Chen, and Yonghui Wu. 2019. Direct speech-to-speech translation with a Sequence-to-sequeee model. arXiv preprint arXiv:1904.0603..
Zhujin Gao, Junliang Guo, Xu Tan, Yongxin Zhu, Fang Zhang, Jiang Bian, and Linli Xu. 2022. Difformer: Empowering diffusion model on embedding space for text generation. arXiv preprini 22f.:222220091...
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with GPUs. IEEE Trunsactions on Big Data 733:5355-547.

---

Takatomo Kano, Sakriani Sakti, and Satoshi Nakamura. 2021. Transformer- based direct speech- to- speech translation with transcoder. In 2021 IEFE Spoken Language Technology Workshop (SLT), pages 958- 965. IEEE
Alexander Quinn Nichol and Prafulla Dhariwal. 2021. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162-8171, PMLR.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLLT 2019; Demonstrations.
Yoon Kim and Alexander M. Rush. 2016. Sequence- level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language P Processing, pages 1317-1327, Austin, Texas. Association for Computational Linguistics.
Adam Polyak, Yossi Adi, Jade Copet, Bugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Ab- delrahman Mohamed, and Emmanuel Dupoux. 2021. Speech resynthesis from discrete disentan- gled self superviue representations. arXiv preprint arXiv:2104.0033.
Jungil Kong, Jachyeon Kim, and Jaekyoung Bae. 2020. Hifi-gan: Generative adversarial networks for effi- cient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 17:17 17033.
Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov. 2021. Grad-us: A diffusion probabilistic model for text-to-speech. In International Conference on Machine Learning. pages 8599-8608 8608. PMLR.
Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, and Emmanuel Dupoux. 2021. On gen- erative spoken language modeling from raw audio. Transactions of the Association for Computational Linguistics. 1334.-1354.
Sravya Popuri, Peng-Jen Chen, Changhan Wang. Juan Miguel Pino, Yossi Adi, Jiatao Gu, Wei-Ning Hsu, and Ann Lee. 2022. Enhanced direct speech-to- speech translation using self-supervised pre-training and data augmentation. In Interspeech.
Alon Lavie, Alexander H. Waihel, Lori S. Levin, Michael Finke, Donna Gates, Marsal Gavaldå, Torsten Zeppenfeld, and Puming Zhan. 1997. Janus- ii:: speech-to-speech translation in multiple lan- guages. 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing. 1:99-107 102 vol.1.
Matt Post. 2018, A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers. pages 186- 191, Brussels, Belgium. Association for Computa- tional Linguistics.
Ann Lee, Peng-Jen Chen, Changhan Wang, Jiatao Gu, Sravya Popuri, Xutai Ma, Adam Polyak, Yossi Adi, Qing He, Yun Tang, et al. 2021. Direct speech-to- speech translation with discrete units. arXiv preprint arXiv:2107.05604,
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. 2022. High- resolution image synthesis with latent diffusion mod- els. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695.
Ann Lee, Hongyu Gong, Paul-Ambroise Duquenne, Holger Schwenk, Peng-Jen Chen, Changhan Wang, Sravya Popuri, Yossi Adi, Juan Pino, Jiatao Gu, and Wei-Ning Hsu. 2022. Textless speech-to-speech translation on real data. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 860-872, Seattle, United States. Association for Computational Lin- guistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Pupers), pages 1715-1725, Berlin, Germany. Association for Computational Lin- guistics.
Kai Shen, Zeqian Ju, Xu Tan, Yanqing Liu, Yichong Leng, Lei He, Tao Qin, Sheng Zhao, and Jiang Bian. 2023. Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers. arXiv preprint arXiv:2304.091/
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis. and Luke Zettlemoyer. 2020. Multilingual denoising pre- training for neural machine translation. Transac- tions of the Association for Computational Linguis- tics, 8:726-742.
Jascha Sohl-Dickstein, Eric Weiss, Niru Mah- eswaranathan, and Surya Ganguli. 2015. Deep un- supervised learning using nonequilibrium thermody- namics. In International Conference on Machine Learning, pages 226562226..222 PMLR.
S. Nakamura, K. Markov, H. Nakaiwa, G. Kikui, H. Kawai, T. Jitsuhiro, J.-S. Zhang, H. Yamamoto. F. Sumita, and S. Yamamoto. 2006. The atr multi- lingual speech-to-speech translation system. IEEE Transactions on Audio, Speech, and Language Pro- cessing, 14(2):65-3376
Yang Song, Jascha Sohl-Dickstedi Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. 2020. Score-based generative modeling through

---

stochastic differential equations. arXiv:2011.13456.
arXiv preprint
Gu al., 2018) can boost the performance, we adopt the typical method the non-auto-regressiv translation task with S2UT w/ ED as the teacher. We set diffusion steps T = 1000 for all diffusion based models with the linear noise schedule. In the sampling process, we use the length beam size 5 which means generating 5 candidates of different lengths at the same time and select the final pre- diction according to perplexity. All experiments are conducted using the FAIRSEQ toolkit (Ott al., 2019). We use FAISS (Johnson al., 2019) for fast nearest neighbor search in high-dimensional spaces.
Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura. 2019. Speech-to-specch translation between untran- scribed unknown languages. 2019 IEEE Auto- matic Speech Recognition and Understanding Work- shop (ASRU), pages 593-600. IEEE.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza. Mary Williamson, Juan Pino, and Emmanuel Dupoux. 2021. VoxPop- uli: A large-scale multilingual speech corpus for rep- resentation learning, remi-supervised learning and interpretation. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the l 11th International Joint Conference on Natural Language Processing (Volume I: Long Pupers), pages 993-1003, Online. Association for Computational Linguistics.
Kun Wei, Long Zhou, Ziqiang Zhang, Liping Chen, Shujie Liu, Lei He, Jinyu Li. and Furu Wei. 2022. Joint pre-training with speech and bilingual text for direct speech to speech translation. arXiv preprint arXiv:2210.1 17027.
Dongchao Yang, Jianwei Yu, Helin Wang. Wen Wang, Chao Weng, Yuexian 7ou, and Dong 2023. Diffssund: Discrete diffusion model for text-to- sound generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing.
Chen Zhang, Xu Tan, Yi Ren, Tao Qin, Kejun Zhang, and Tie- Yan Liu. 2021. Uwspeech: Speech speech translation for unwritten languages. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14319-14327.
A Appendix
A.1 Implementation Details
We follow the model architecture settings in (Lee al., 2022), with a 12-layer speech encoder and 6-layer unit decoder with embedding size 512 and 8 attention heads. We train the model for 600k steps on VoxPopuli S2ST training datasets. We use Adam optimizer with learning rate 0.0003 for all translation directions, and inverse square root learn ing rate decay schedule with 10k warm up steps. To prevent overfitting, for training. All S2UT systems except S2UT w/ PT (pre-trained models) are trained with an auxiliary task as (Lee et al., 2022). We find that sentence level knowledge distillation (Kim and Rush, 2016;
label smoothing of 0.2


Reviews:
[{"review_id": "cTyOAZSIZM", "reviewer": "Reviewer_1oEp", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the valuable comments and constructive suggestions to help improve the paper. We give the response below.\n\nQ1: The comparison of generation steps.\n\nIt is true that the generation steps for auto-regressive (AR) models and diffusion models have different meanings, where AR models generate at the token level and diffusion models generate at the sequence level. However, it is an advantage of diffusion models over AR models that the number of generation steps is fixed and does not increase linearly with the sequence length. In fact, the sequence length, which is equivalent to the generation steps in AR models, is close to the diffusion models. Since both models generate sentences with the same meaning, the difference in sequence length is negligible. Fixing the target sequence length for auto-regressive models is not recommended because the model implicitly generates the [EOS] token, which means it may stop generating before reaching the maximum sequence length or continue generating even after reaching it.\n\nQ2: The comparison of the computational cost.\n\nWe conducted a benchmarking test following the official procedure in FAIRSEQ (examples/speech\\_to\\_speech/benchmarking) to compare the runtime costs of auto-regressive and DiffS2UT model (50 decoding steps) on the Es-En dataset, which is specifically designed for S2ST. As shown in the table below, DiffS2UT achieves a speedup of $11.9\\times$, demonstrating its efficiency. We appreciate the suggestions and will include a computation cost comparison of all models to Table 1 in the revised paper.\n\n|Model|Mem/MiB|Runtime/sec|Speedup|\n|:------|:----------:|:-------------:|:---------:|\n|S2UT [Lee et al., 2022]   | $6293.73$  | $90.57$ | $1.0 \\times$|\n|DiffS2UT (Ours)  | $8332.72$ | $7.61$  | $11.9 \\times$|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Drawing upon the one-to-one correspondence between discrete units and continuous K-means centroid vectors, this paper adeptly combines the continuous diffusion forward process with the discrete diffusion backward process. This idea is novel and reasonable to me.\n2. The experimental results illustrate that the proposed method outperforms a direct application of vanilla discrete diffusion models.\n3. The paper is well-written and easy to follow.", "reasons_to_reject": "The authors assert that the diffusion-based architecture significantly expedites the generation process by comparing it to the auto-regressive generation step, which scales with audio length, in contrast to the fixed number of steps in diffusion generation. However, these two types of “generation step” have different meanings, which I think cannot be used to compare these two generation methods directly. A more straightforward comparison could involve fixing the target sequence length (e.g., 500, 1000, 1500…) and then evaluating the time and space consumption of different methods.\n\nConsidering the lower performance of the diffusion generation methods compared to the auto-regressive ones, I want to see the more intuitive comparison of the computational cost between different methods.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces diffusion models to the textless S2ST task, aiming to expedite the generation process. The authors propose integrating continuous and discrete diffusion models by disentangling the forward and backward diffusion processes. Experimental results demonstrate the effectiveness of the proposed method.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4k96dPffHk", "reviewer": "Reviewer_n2co", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the valuable comments and constructive suggestions to help improve the paper. We give the response below.\n\nQ1: The introduction is too long and goes into too much detail.\n\n We will carefully revise the introduction section and try our best to reduce the length of the introduction while maintaining the article's clarity and coherence. We believe that it is crucial to introduce the direct S2ST task and continuous/discrete diffusion models meticulously and explain why it is important to integrate diffusion into S2ST, which includes substantial prior work that contributes to our concept development. Also, we believe it benefits both readers and the community to gain insight into the S2ST task and diffusion models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This is the first work that effectively introduces diffusion generative models to the textless S2ST task.\n2. significantly surpassing the vanilla diffusion models, and achieving comparable performance to auto-regressive models while requiring much fewer generation steps.\n3. The algorithm looks detailed.\n4. The research is innovative, and is an exploratory study of diffusion generative models on text-free S2TS.", "reasons_to_reject": "The introduction is too long and goes into too much detail. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "While Diffusion Generative Models have achieved great success on image generation tasks, how to efficiently and effectively incorporate them into speech generation especially translation tasks remains a non-trivial problem. The authors propose a novel diffusion model by applying the diffusion forward process in the continuous speech representation space, while employing the diffusion backward process in the discrete speech unit space. The model achieves 14.8/15.2/14.5/13.6 BLEU score on the Eurapal-ST Es-En/Fr-En/En-Es/En-Fr test sets.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2D8oJQbIlf", "reviewer": "Reviewer_787n", "summary": "", "questions": "1. Line 515, how do you get the conclusion for semantic preserving? Table 2 only shows BLEU score, which is not a sufficient metrics for semantics.\n2. Line 485, what is ED and PT from related work? I read the reference paper but still don't know the meaning of ED.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the valuable comments and constructive suggestions to help improve the paper. We give the response below.\n\nQ1: Lack of comparison with a baseline and the improvement of the proposed methods is minor.\n\nWe do compare our approach with the auto-regressive baseline [Lee. et al] at the second line of Table 1. Our approach achieves comparable results to it, outperforming in some directions while lagging behind in others though. However, our primary objectives are to expedite speech generation and improve the generation quality over vanilla diffusion models. Therefore, it is not appropriate to solely assess our approach by comparing BLEU scores to auto-regressive models. Our experimental results clearly demonstrate superior efficiency compared to auto-regressive models (approximately 10 times faster) and improved generation quality when contrasted with vanilla diffusion models (achieving BLEU scores ranging from 3 to 5).\n\nQ2: Too much introduction to diffusion models and insufficient analysis regarding why the diffusion model helps the units generation.\n\nWe will carefully revise the introduction section and try our best to reduce the length of the introduction while maintaining the article's clarity and coherence. We provide algorithms and implementation details of the proposed method in the paper. The diffusion model aims at helping accelerate the unit generation speed compared with auto-regressive models (about 10 times faster) and the incorporation of semantic structure improves the BLEU score compared to vanilla diffusion models.\n\nQ3: How do we get the conclusion for semantic preserving?\n\nIt is hard to design a specific metric for semantic preserving. We give an intuitive explanation of semantic preserving in Figure 1(d), on which the motivation of DiffSUT is based. We also conduct the ablation study of semantic preserving, i.e. K-means mapping, in the analysis section. The main difference between DiffS2UT and vanilla diffusion models lies in semantic preserving, so we contend that the improvement of the BLEU score adequately substantiates its efficacy.\n\nQ4: The meanings of ED and PT.\n\nWe explain the meanings of ED and PT in Section 5.1 Setup Baselines, which refer to Extra Data [1] and Pre-Training [2] respectively. We also introduce the related paper in the related work (line $220 \\sim 226$). We will clarify it more clearly in the revised version.\n\n[1] Duquenne, Paul-Ambroise, et al. \"SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations.\" arXiv preprint arXiv:2211.04508 (2022).\n\n[2] Popuri, Sravya, et al. \"Enhanced direct speech-to-speech translation using self-supervised pre-training and data augmentation.\" arXiv preprint arXiv:2204.02967 (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea in the paper is quite novel. The diffusion model has been introduced in many tasks and shows success. It is of great importance to show its potential to the speech-to-unit task.\n\n2. The adaptation of the diffusion model on units is well designed. Instead of adding mask directly to discrete units, the author designed a scheme to move these operations to embedding space.\n\n3. The paper gives a very comprehensive introduction to diffusion models and background.", "reasons_to_reject": "1. The performance of the proposed methods is not good compared with the baseline. In the paper, the authors compare the proposed model with other S2UT model with diffusion model. However, the proposed model should be compared to a simpler yet much stronger baseline from Lee. et al 2021. Comparing with Lee. et al 2021, the proposed model only shows improvement on Es->En directions. There is no improvement or even downgrades for other directions.\n\n2. The paper has too much introduction on the diffusion model and insufficient content of proposed model. The paper contains two pages of introduction section. The adaptation for diffusion to unit translation seems too abrupt and it would be good to have more analysis on the model behavior to show why actually the diffusion model helps the units generation. ", "questions_to_authors": "1. Line 515, how do you get the conclusion for semantic preserving? Table 2 only shows BLEU score, which is not a sufficient metrics for semantics.\n2. Line 485, what is ED and PT from related work? I read the reference paper but still don't know the meaning of ED.", "paper_topic_and_main_contributions": "The paper introduces the diffusion model to direct speech-to-unit translation. Diffusion models, which have gained considerable success on other tasks in computer vision and natural language processing, construct forward and backward processing during training. The forward pass constructs a series of noisy sampling processes and backward recovers the original samples. \n\nSpecifically in the paper, the diffusion process is applied on the target discrete units. The author designs a scheme to add noise and reconstruct on the embedding level to solve issue of the discreteness of the units. The experiments show improvement over S2UT system with other diffusion-based model.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yx1iIVJZ7b", "reviewer": "Reviewer_2aZv", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the valuable comments and constructive suggestions to help improve the paper. We give the response below.\n\nQ1: The comparison with TranSpeech.\n\nWe did notice the TranSpeech model and have cited it in the related work. As introduced in Section 3.2, a direct textless S2ST system typically comprises three components: (1) The speech unit extraction model. (2) A translation model built upon units. (3) The unit to waveform model. \nOur proposed model, DiffS2UT, and the selected baselines all focus on the second component to build a stronger model for translation. In contrast, TranSpeech focuses on the first component to extract more robust speech units. Consequently, it would be not appropriate to take it as a baseline for comparison. It is noteworthy that our method and the selected baselines are fully compatible with the TranSpeech method. We can achieve further improvement with the TranSpeech method, which is beyond the scope of our focus though. \n\nQ2: The gap between DiffS2UT and auto-regressive models.\n\nOur approach achieves comparable results to the auto-regressive S2UT baseline, outperforming in some directions while lagging behind in others though. However, our primary objectives are to expedite speech generation and improve the generation quality over vanilla diffusion models. Therefore, it is not appropriate to solely assess our approach by comparing BLEU scores to auto-regressive models. Our experimental results clearly demonstrate superior efficiency compared to AR models (approximately 10 times faster) and improved generation quality when contrasted with vanilla diffusion models (achieving BLEU scores ranging from 3 to 5).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors introduce diffusion generative models to the textless S2ST task.\nThe authors propose a novel diffusion framework that integrates the continuous and discrete diffusion models by decoupling the diffusion forward and backward processes.\nExperiments demonstrate the proposed method.", "reasons_to_reject": "The paper does not compare with other non-autoregressive methods, such as TranSpeech[1].\nThere still exists a certain gap between the method proposed in this paper and the auto-regressive methods.\n[1] TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation", "questions_to_authors": "", "paper_topic_and_main_contributions": "The topic of this paper is textless direct speech-to-speech translation.\nThe authors propose an effective diffusion model for speech-to-speech translation.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed diffusion models for textless speech-to-speech translation. Proposes integrating continuous and discrete diffusion models.

Pros:
- Performance is surpassing vanilla diffusion models
- introduces diffusion generative models for textless s2st task

Cons:
- Comparison with other stronger baseline models is not done
- introduction is too long
Decision: Accept-Main