Duplex Diffusion Models Improve Speech-to-Speech Translation
Year: 2023

Abstract:
Speech-to-speech translation is a typical sequence-to-sequence learning task that naturally has two directions. How to effectively leverage bidirectional supervision signals to produce high-fidelity audio for both directions? Existing approaches either train two separate models or a multitask-learned model with low efficiency and inferior performance. In this paper, we propose a duplex diffusion model that applies diffusion probabilistic models to both sides of a reversible duplex Conformer, so that either end can simultaneously input and output a distinct language’s speech. Our model enables reversible speech translation by simply flipping the input and output ends. Experiments show that our model achieves the first success of reversible speech translation with significant improvements of ASR-BLEU scores compared with a list of state-of-the-art baselines.

heading: 1 Introduction
text: Direct speech-to-speech translation (S2ST) (Lee et al., 2021; Inaguma et al., 2022), transforming a source language’s speech to the target language’s speech, is essential for online international communications and is friendly to numerous languages that do not have their own writing systems or textual vocabularies. S2ST circumvents a cascaded architecture (Lavie et al., 1997; Nakamura et al., 2006; Wahlster, 2000) of combining automatic speech recognition (ASR) of the source speech, textual source-to-target machine translation (MT), and target text-to-speech (TTS) synthesis where multiple types of datasets are required, error propagates, latency is high, and unavailable for thousands of (spoken) languages who do not have a writing system.
For S2ST, speech-to-speech parallel data is required, and it is costly to collect a comparable size dataset with textual counterparts. To alleviate the data scarcity problem, self-supervised pre-training and data augmentation techniques were used by
Popuri et al. (2022), and unsupervised and weaklysupervised speech and text data under Translatotron 2 (Jia et al., 2021) were leveraged by Jia et al. (2022a). Techniques such as multi-task learning (Weiss et al., 2017), pseudo labeling (Pino et al., 2020), and knowledge distillation (Inaguma et al., 2021) have also been adapted and achieved promising results.
From S2ST architecture’s point of view, Inaguma et al. (2022) describes four categories, (1) Translatotron (Jia et al., 2019) style which includes a speech encoder and a spectrogram decoder, (2) Translatotron2+ (Jia et al., 2021) style which inserts a first-pass text decoder followed by a TTS encoder between the two modules of Translatotron, (3) speech-to-unit translation (S2UT) (Lee et al., 2021) that uses discrete clustered units of the target language speech instead of spectrogram, and (4) UnitY (Inaguma et al., 2022) that inserts a firstpass text decoder followed by a text-to-unit (T2U) encoder between the two modules in S2UT.
In this paper, following the motivations of textual duplex machine translation (Zheng et al., 2021), we leverage S2ST’s two directions: effectively utilizing supervision signals from both directions is estimated to both relieve the pain of data scarcity and bring novel architectures of training and inferencing. Existing architectures (e.g., Translatotron1/2, S2UT, and UnitY) either train two separate models or a multitask-learned model with low efficiency and inferior performance. In contrast, we propose a duplex diffusion model that applies diffusion probabilistic models to both sides of a reversible duplex Conformer, so that either end can simultaneously input and output a distinct language’s speech. Our model enables reversible speech translation by simply flipping the input and output ends. Experiments show that our model achieves the first success of reversible speech translation with significant improvements of ASR-BLEU scores compared with a list of strong baselines.
8035
Our contributions are concluded as follows:
• a novel reversible duplex Conformer that extends the widely used Conformer (Gulati et al., 2020) architecture from ASR to S2ST, with reversible and symmetrical forward/reverse building blocks;
• a novel duplex diffusion model that jointly train one reversible duplex Conformer in diffusion ways to fit two translation directions;
• significantly better or comparable ASRBLEU scores are achieved by comparing with a list of state-of-the-art baselines including Translatotron, Translatotron2, S2UT, and UnitY.

heading: 2 Backgrounds

heading: 2.1 REDER
text: REDER, REversible Duplex TransformER, was proposed by Zheng et al. (2021) for reversible textual machine translation through duplex sequenceto-sequence (seq2seq) learning. A neural network with a parameter set θ is duplex when it satisfies the following conditions. First, the network has two ends, each end can take one language as its input or output. Second, the network defines a forward mapping function f→θ : X 7→ Y , and a backward (reverse) mapping function f←θ : Y 7→ X , that satisfies two reversibilities: f←θ = (f → θ ) −1 and f→θ = (f ← θ ) −1. Third, the network satisfies the cycle consistencies: ∀x ∈ X : f←θ (f→θ (x)) = x and ∀y ∈ Y : f→θ (f←θ (y)) = y.
However, building duplex seq2seq networks is non-trivial and must satisfy the following constraints, reversibility and homogeneity. First, vanilla encoder-decoder network, such as frequently used Transformer (Vaswani et al., 2017) and its variants, is irreversible. It is not feasible for the output end of the decoder side to take in input signals to exhibit the encoding functionality and vice versa. Second, the natural network architectures of the non-autoregressive encoder and the autoregressive decoder are heterogeneous. Therefore, REDER, leverages reversible Transformer layers (Gomez et al., 2017a) and fully nonautoregressive modeling without explicit encoder and decoder division, is designed to solve these two challenges. As reported in (Zheng et al., 2021), REDER worked in a duplex way that better exploited the bidirectional supervisions for achieving
better downstream reversible machine translation tasks’ performance.
The architecture of REDER is a stack of L reversible duplex transformer layers where the 1-st to L/2-th layers are mirror of the (L/2+1)-th to L-th layers to ensure the whole model being symmetric. In particular, each layer contains a multi-head selfattention (MHSA) module and a feed-forward network (FFN) module with a novel reversible design to ensure duplex behavior, where the input and output tensors of such a layer are split into two halves, Hl−1 = [H (1) l−1;H (2) l−1] and Hl = [H (1) l ;H (2) l ], respectively. Formally, the regular form of the l-th layer Fl performs as follow:
[H (1) l ;H (2) l ] = Fl([H (1) l−1;H (2) l−1]), (1)
H (1) l = H (1) l−1 + MHSA(H (2) l−1), (2) H (2) l = H (2) l−1 + FFN(H (1) l ). (3)
The reverse form F−1l can be computed by subtracting the residuals:
[H (1) l−1;H (2) l−1] = F−1l ([H (1) l ;H (2) l ]), (4)
H (2) l−1 = H (2) l − FFN(H (1) l ), (5) H (1) l−1 = H (1) l −MHSA(H (2) l−1). (6)

heading: 2.2 DDPM
text: We briefly introduce the diffusion and reconstruction processes in Denoising Diffusion Probabilistic Models (DDPM). Given a data point x0 sampled from a real data distribution q(x) (x0 ∼ q(x)), Ho et al. (2020) define a forward diffusion process in which small amount of Gaussian noise is added to sample x0 in T steps to obtain a sequence of noisy samples x0, ..., xT . A predefined (hyper-parameter)
variance schedule {βt ∈ (0, 1)}Tt=1 controls the step sizes:
q(xt|xt−1) = N (xt; √ 1− βtxt−1, βtI); (7)
q(x1:T |x0) := T∏
t=1
q(xt|xt−1). (8)
When T → ∞, xT is equivalent to following an isotropic Gaussian distribution. Note that, there are no trainable parameters used in this forward diffusion process.
Let αt = 1 − βt and ᾱt = ∏t
i=1 αi, we can express an arbitrary step t’s diffused sample xt by the initial data sample x0:
xt = √ ᾱtx0 + √ 1− ᾱtϵt. (9)
Here, noise ϵt ∼ N (0, I) shares the same shape with x0 and xt.
In order to reconstruct from a Gaussian noise input xT ∼ N (0, I), we need to learn a model pθ to approximate the conditional probabilities to run the reverse diffusion (reconstruction) process:
pθ(xt−1|xt) = N (xt−1;µθ(xt, t),Σθ(xt, t));
pθ(x0:T ) := p(xT ) T∏
t=1
pθ(xt−1|xt). (10)
Note that the reverse conditional probability is tractable by first applying Bayes’ rule to three Gaussian distributions and then completing the “quadratic component” in the exp(·) function:
q(xt−1|xt, x0) = N (xt−1; µ̃t(xt, x0), β̃tI) (11)
= q(xt|xt−1, x0) q(xt−1|x0) q(xt|x0) (12) ∝ exp(− 1 2β̃t (xt−1 − µ̃t)2). (13)
Here, variance β̃t is a scalar and mean µ̃t depends on xt and noise ϵt:
β̃t = 1− ᾱt−1 1− ᾱt βt; (14) µ̃t = 1√ αt (xt − 1− αt√ 1− ᾱt ϵt). (15)
Intuitively, q(xt−1|xt, x0) acts as a reference to learn pθ(xt−1|xt). We can use the variational lower bound (VLB) to optimize the negative loglikelihood:
− logpθ(x0) ≤ −logpθ(x0)+ DKL(q(x1:T |x0) ∥ pθ(x1:T |x0)). (16)
Using the definitions of q(x1:T |x0) in Equation 8 and pθ(x0:T ) in Equation 10, a loss item Lt (1 ≤ t ≤ T − 1) is expressed by:
Lt = DKL(q(xt|xt+1, x0) ∥ pθ(xt|xt+1)) (17)
= Ex0,ϵt [∥ µ̃t − µθ(xt, t) ∥2
2 ∥ Σθ(xt, t) ∥22
] .
We further reparameterize the Gaussian noise term instead to predict ϵt from time step t’s input xt and use a simplified objective that ignores the weighting term:
Lsimplet = Et∼[1,T ],x0,ϵt [ ∥ ϵt − ϵθ(xt, t) ∥2 ] (18)

heading: 3 Reversible Duplex Conformer
text: In this paper, we extend the widely used Conformer (Gulati et al., 2020) architecture for encoding the speech signals into dense and compact representations of both ends. Conformer has achieved impressive results in supervised ASR by leveraging transformer’s capturing of content-based global interactions and convolutional neural network’s exploiting of local features. In Conformer, two macaron-like FFN layers with half-step residual connections sandwich the MHSA and convolution (CNN) modules followed by a post layer normalization. Besides supervised ASR, Conformer has also been successfully used in self-supervised Wav2Vec (Schneider et al., 2019; Baevski et al., 2020) pretraining for downstream application tasks’ finetuning.

heading: 3.1 Forward and Reverse Building Blocks
text: Following (Gomez et al., 2017b; Zheng et al., 2021), we split the l-th layer’s (left-end) input tensor into two parts, Hl−1 = [x(1);x(2)]. The
(right-end) output tensor is split in the same way, Hl = [z
(1); z(2)]. Thus, the forward target of this REDER-style Conformer layer is [x(1);x(2)] 7→ [z(1); z(2)].
We introduce two intermediate tensors, y(1) and y(2), for intuitive understanding and mathematical convenient. Both Conformer’s four sub-modules (two FFNs, one MHSA and one CNN) and four residual connections are kept in our reversible duplex Conformer.
Figure 2 depicts the forward (a) and reverse (b) building blocks for one layer in our proposed reversible duplex Conformer. In Figure 2, the reverse block is a mirror of the forward block with symmetrical network connections and subtract residual connections. The forward block can be formally expressed as follows:
y(1) = x(1) + 0.5× FFN(x(2)); (19) y(2) = x(2) + MHSA(y(1)); (20)
z(1) = y(1) + CNN(y(2)); (21)
z(2) = y(2) + 0.5× FFN(z(1)). (22)
Symmetrically, the reverse block is expressed by:
y(2) = z(2) − 0.5× FFN(z(1)); (23) y(1) = z(1) − CNN(y(2)); (24) x(2) = y(2) −MHSA(y(1)); (25) x(1) = y(1) − 0.5× FFN(x(2)). (26)
We employ Layer Normalization (LN) (Ba et al., 2016) at the beginning of each module, i.e., PreLN (Xiong et al., 2020). The FFN module processes the input tensor x by six components:
FFN(x) = p2 ◦W2 ◦ p1 ◦ SiLU ◦W1 ◦ LN(x).
Here, ◦ means a layer takes ◦’s right-hand-side network’s output (e.g., LN(x)) as the input of ◦’s left-hand-side network (e.g., W1 to perform W1(LN(x))). W1 and W2 are two linear layers that preforms h 7→ 4h and 4h 7→ h linear projections, respectively. Two dropout layers p1 and p2 are used. The Sigmoid Linear Unit (SiLU) (Elfwing et al., 2017) activation function is inserted between the two linear layers. The MHSA module contains three components:
MHSA = p ◦ Attention ◦ LN(x).
We use multi-head attention with relative positional embedding (Shaw et al., 2018) for the “Attention”
component. Note that, the attention module is extendable to cross-attention cases where a source sequence’s encoded representation acts as memory (i.e., key and value) to the target sequence. Finally, the CNN module utilizes two types of convolutions, pointwise (PW) and 1D depthwise (DW), to capture local-range dependencies of the input speech. The idea of employing attention for global context modeling and convolution for local context modeling is also inspired by the long-short range attention mechanism used in the lite transformer (Wu et al., 2020). Formally,
CNN(x) = p ◦ PW2 ◦ Swish ◦ BN ◦ DW ◦ Glu ◦ PW1 ◦ LN(x).
Here, BN stands for batch normalization. Two types of activation functions, Glu (Dauphin et al., 2016) and Swish (Ramachandran et al., 2017), are inserted between convolution networks.

heading: 3.2 Symmetric Network Architecture
text: As depicted in Figure 3, the forward and reverse building blocks are arranged symmetrically in the whole architecture to achieve homogeneous computations. Specifically, in the L building blocks, the 1-st to L/2-th layers are set to be reverse blocks whereas the (L/2 + 1)-th to L-th layers be the regular forward form:
f→θ (x) = FL ◦ · · · ◦ FL/2+1 ◦ F−1L/2 ◦ · · · ◦ F −1 1 (x);
f←θ (z) = F1 ◦ · · · ◦ FL/2 ◦ F−1L/2+1 ◦ · · · ◦ F −1 L (z).
This design makes our reversible duplex Conformer to be homogeneous: the forward computational operation chain reads as a palindrome string < fcmf · · · fcmf |fmcf · · · fmcf > and so does the reverse chain, where f,m, c denotes FFN, MHSA and CNN, respectively.
Algorithm 1: Duplex Diffusion Model (DDM) Training Algorithm - One Step
1 Given: x, y, Ex, Ey 2 x0 = Ex(x) ▷ encode by pretrained
wav2vec models 3 y0 = Ey(y) ▷ encode by pretrained
wav2vec models 4 t ∼ Uniform(1, ..., T ) 5 ϵx ∼ Nx(0, I), ϵy ∼ Ny(0, I) 6 xt = √ ᾱt,xx0 + √ 1− ᾱt,xϵx 7 yt = √ ᾱt,yy0 + √ 1− ᾱt,yϵy 8 ϵθx = ←−−Mθ(xt, t,y0) ▷ reverse, given y0 9 ϵθy = −−→Mθ(yt, t,x0) ▷ forward, given x0
10 LDDM = λ1 ∥ ϵx − ϵθx ∥2 +λ2 ∥ ϵy − ϵθy ∥2
There are several selections of input types of the source and target ends in Figure 3. Popuri et al. (2022) explores self-supervised pretrained models such as (1) wav2vec2 (Baevski et al., 2020) to encode the source speech and (2) Unit mBART (Liu et al., 2020a) to encode the target discrete units (Lee et al., 2021), and then translate source speech into target clustered units through fine-tuning. The generated discrete unit sequence is then sent to an independently trained “text”-to-speech (TTS) model to obtain the final waves.
In this paper, we follow the usage of discrete units that are generated by first using pretrained HuBERT (Hsu et al., 2021) to encode the target speech and then perform k-means clustering. Then, we use the DiffWave (Kong et al., 2020b) vocoder to generate the final waves.

heading: 4 Duplex Diffusion Model
text: Cycle consistency has been utilized in textual neural machine translation (Zheng et al., 2021) and image-to-image translation (Su et al., 2022). In this paper, we propose a duplex diffusion model that alternatively optimizes both directions by two diffusion processes.
The training algorithm is described in Algorithm 1. Generally, we borrow DDPM (Ho et al., 2020)’s architecture and extend it to a duplex scenario where sequences of two ends are diffused alternatively during training. At the beginning, the source sequence x and target sequence y are encoded into dense representations by pretrained wav2vec models Ex, Ey through self-supervised learning on monolingual datasets, respectively. Then, time t
and two normal Gaussian noise signals ϵx, ϵy are sampled. Note that the lengths of the source and target sequences are diverse. We pre-define two variance schedules {βt,x ∈ (0, 1)}Tt=1 and {βt,y ∈ (0, 1)}Tt=1, for the source and target languages, respectively. Thus, we have αt,x = 1− βt,x, ᾱt,x =∏t
i=1 αi,x, αt,y = 1− βt,y and ᾱt,y = ∏t
i=1 αi,x, as used in Algorithm 1.
The variance schedules, initial sequence representations and normal Gaussian noises work together to give us diffused representations, xt and yt, respectively. They are then sent to the reversible duplex Conformer architectureMθ (Figure 3) to predict the noises.
Originally in Figure 3, we are intended to produce x0 from y0 in the reverse process of
←−−Mθ. Now, we have two additional inputs, t and xt. The output also changes from predicting x0 to estimating ϵθx which shares the same shape with x0. We thus have two ways to organize the network←−−Mθ: (1) reuse Figure 3’s architecture and predicting ϵθx from y0 by taking xt as the “memory” which acts as key and value in the cross-attention network in Conformer, or (2) follow traditional stable diffusion models (Rombach et al., 2021) and predict ϵθx from xt by taking y0 as the conditional “memory”. That is, in the MHSA function, we set query xt to be and key/value to be y0, MHSA(q = xt, k = y0, v = y0), so that the identical lengths of q = xt and ϵθx are ensured. Note that, in the second choice used in our experiments, we are not limited to use a reversible duplex Conformer, i.e., any transformer architecture with cross-attention are applicable. These two options still hold during inferencing from given y0, T , and xT to iteratively reconstruct x0.
Since the lengths of the source and target sequences are diverse, we follow textual duplex translation (Zheng et al., 2021) and double the source end’s length by a upsampling convolutional network.
We only describe the reverse process ←−−Mθ and the forward process −−→Mθ shares the similar strategies. To achieve a full cycle consistency, predicting the target Gaussian noise from source sequence by taking target noisy sequence as the “conditional memory” is more appropriate in current scenario setting so that both translation directions are achieved in one duplex diffusion model.
After the reverse and forward processes, we can compute the MSE losses of between the two pairs
of reference and predicted noises. They are interpolated together by hyper-parameter weights λ1 (=0.5) and λ2 (=0.5) to the final loss LDDM to be optimized.

heading: 5 Training
text: Our reversible duplex Conformer is largely inspired by REDER (Zheng et al., 2021). The novel parts are that (1) we select and reconstruct convolutionenhanced Conformer (Gulati et al., 2020) to synthetically capture global information by attentions and local context by convolutions, and (2) we extend from textual duplex machine translation to (dense) duplex speech-to-speech translation. When training our reversible duplex Conformer, we borrow and adapt the losses that are used in REDER to fit our scenario.
In REDER, three types of losses were used. The first loss is to model the variable-length of source and target sequences by a latent alignment approach, i.e., the Connectionist Temporal Classification (CTC) (Graves et al., 2006). Starting from the conditional independence assumption, CTC is capable of efficiently (by dynamic programming) finding all valid (yet monotonic) alignments a which derives from the target y by allowing consecutive repetitions and inserting blank tokens. The CTC loss is defined by:
LCTC = −logpCTC(y|x; θ) = −log ∑
a
pθ(a|x).
We adapt this loss for speech translation when the target are clustered unit sequences. We use the MSE loss instead when the target is a sequence of mel-spectrogram. Also, to ensure the source sequence is always longer than the target sequence, we upsample the source sequences by convolutional layers before sending them to the reversible duplex Conformer.
The second loss measures the layer-wise forward-backward agreement (fba, measured by cosine similarity) of between the forward l-th layer’s representation −→ Hl = Fl( −→ Hl−1) and the reverse representation ←− Hl = Fl( ←− Hl+1). Thus,
Lfba(y|x; θ) = 1
L
L∑
l=1
{ 1− cos(−→Hl, sg( ←− Hl)) } ,
where sg denotes the stop-gradient operation. The third loss explicitly describe the cycle consistency of a pair of seq2seq tasks, i.e., we minimize the distance between the original x and its
reconstruction f←θ (f → θ (x)) by,
Lcc(x; θ) = distance(x, f←θ (f→θ (x))).
For speech translation, the source sequence can be expressed by mel-spectrogram or clustered units so that MSE loss or CTC loss can be applied to them, respectively. Finally, these three types of losses are doubled to two directions and interpolated together for the final loss. That is, when predicting discrete units, the final loss function is:
Lunit = w1 ∗ LCTC(y|x) + w2 ∗ LCTC(x|y) + w3 ∗ Lfba(y|x) + w4 ∗ Lfba(x|y) + w5 ∗ Lcc(y) + w6 ∗ Lcc(x).
When predicting mel-spectrograms, the final loss function is:
Lmel = w1 ∗ LMSE(y|x) + w2 ∗ LMSE(x|y) + w3 ∗ Lfba(y|x) + w4 ∗ Lfba(x|y) + w5 ∗ Lcc(y) + w6 ∗ Lcc(x).
We reuse the default hyper-parameter values described in REDER (Zheng et al., 2021) for setting weights w1 to w6.
In our experiments, we first train the reversible duplex Conformer architecture by a predefined K1 (=200,000) iterations and then apply the duplex diffusion training algorithm as shown in Algorithm 1. After another predefined K2 (=200,000) iterations, we fix the diffusion processes and focus on updating the reversible duplex Conformer only so that traditional search algorithms such as beam search can be used for seeking target hypotheses.

heading: 6 Experimental Setups

heading: 6.1 Data
text: To compare with state-of-the-art baselines’ reported results, we align with UnitY (Inaguma et al., 2022) and use three S2ST datasets: (1) Fisher Es→En (Post et al., 2013) with 170-hour Spanish (Es) conversational telephone speech with textual transcriptions in Es and En. The English speech is synthesized by a single-female-speaker TTS model. (2) CVSS-C (Jia et al., 2022b), a public multilingual S2ST corpus from CoVoST2 (Wang et al., 2020). Again, a single-female-speaker TTS model is employed to synthesize the target speech. (3) Multi-domain En↔Es corpora (Popuri et al., 2022). We follow (Inaguma et al., 2022) to collect 1983- hour source speech for En→Es and 1404-hour source speech for Es→En.

heading: 6.2 Pre-training and Pre-processing
text: We use the pretrained wav2vec2.0 (Baevski et al., 2020) with a 24-layer Conformer (Gulati et al., 2020) self-trained on the Libri-Light dataset (Kahn et al., 2019), HuBERT (Hsu et al., 2021), mHuBERT (Popuri et al., 2022), and mBART (Liu et al., 2020b) given in Table 9 of (Inaguma et al., 2022).
For acoustic feature extraction, discrete unit extraction (100 clusters) and text normalization (e.g., for evaluation score computing), we follow (Popuri et al., 2022; Inaguma et al., 2022).

heading: 6.3 Vocoder
text: Instead of using the HiFi-GAN vocoder (Kong et al., 2020a; Polyak et al., 2021) which converts mel-spectrograms or discrete units to waveform sequences for TTS and direct speech-tospectrogram/unit models, we borrow a comparable diffusion based vocoder, DiffWave (Kong et al., 2020b), for reconstructing waveforms from spectrogram or unit sequences.

heading: 6.4 Training and Decoding Configurations
text: We implement our models based on the Fairseq toolkit1 (Ott et al., 2019). All our models are optimized with a mixed precision training for footprint saving. Our reversible duplex Conformer uses the settings of Conformer-Large with 135.1M parameters (Gulati et al., 2020). The two diffusion variance schedules used in our duplex diffusion model follow stable diffusion (Rombach et al., 2021). We use a NVIDIA DGX-A100*8 workstation to perform the training with a total of 2,500 GPU hours.
During inferencing, we set the beam size to be 10 which aligns with most of the baselines for fair comparison. Other configurations not mentioned here follow their default settings in their open-source repositories.

heading: 6.5 Evaluation
text: We use a pre-trained ASR model to transcribe the target speech into texts and then calculate 4- gram BLEU scores (Papineni et al., 2002), denoted as ASR-BLEU. The target languages’ ASR models are fine-tuned from pretrained wav2vec2.0 (Baevski et al., 2020) models with the CTC objective (Graves et al., 2006) when we taking discrete unit sequences as the prediction target. The same criterion has been used in (Inaguma et al., 2022).
1https://github.com/facebookresearch/fairseq

heading: 7 Experimental Results

heading: 7.1 Fisher Es→ En
text: In Table 1, we compare the ASR-BLEU scores of our systems (RDC and DDM) with three cascaded systems, four speech-to-spectrogram baselines which are variants of Translatotron (Jia et al., 2019, 2021), and four speech-to-unit baselines which are variants of (Lee et al., 2021) and UnitY (Inaguma et al., 2022). Baseline results are originally listed in (Inaguma et al., 2022).
We use RDC to denote our reversible duplex Conformer architecture that are trained in a similar way with textual REDER (Zheng et al., 2021). Our DDM further “boost” the quality of pretrained RDC models by bidirectional diffusion processes and can be recognized as an integration of the diffusion framework with RDC. Of the three categories, S2Sp and S2U achieved significantly better (p < 0.01) ASR-BLEU scores than the three traditional cascaded systems. In the S2Sp paradigm, our “S2Sp-DDM, w2v2” model achieves comparable results with the best baseline “S2Sp-Tn2+, C-w2v2”. In the S2U paradigm, our model “S2U-DDM, w2v2” achieves significantly better (p < 0.05) re-
sults than the best baseline “UnitY, C-w2v2”, with 1.2%, 1.5% and 1.5% absolute ASR-BLEU points. These reflects that our proposed duplex seq2seq learning can be boosted by the bidirectional diffusion processes to better capture the translation distributions of among the source and target sides. In addition, wav2vec2.0 acts as an essential component for all the model variants.
Table 1 also lists four variants of our models for ablation study. When we only use S2U-RDC, it performs better than the S2U+Conformer baseline. However, this advantage disappears when w2v2 is further employed to these two variants. S2URDC also performs relatively worse than UnitY which employs two pass decoding of basing on texts and units whereas our S2U-RDC uses units only. These reflect that, (1) additional textual information brings better results than duplex training, (2) diffusion processes can partly “hedge” the benefits from two-pass decoding used in UnitY and enhance the performance of duplex translations.

heading: 7.2 CVSS-C
text: The CVSS-C corpus’s ASR-BLEU scores of six baselines from three categories and our models are listed in Table 2. We observe almost the same tendencies with the result comparisons in the Fisher task (Table 1). The best baseline is still the two-pass UnitY model enhanced by a pretrained wav2vec-BERT model. Our S2U-DDM model improves UnitY by 0.4% ASR-BLEU points on average, comparable yet not significant.

heading: 7.3 Multi-domain En↔Es
text: The bidirectional multi-domain En↔Es results are listed in Table 3. We again compare with six state-
of-the-art baselines in three categories. On both directions, our model variants meet the best performances on the two test sets. We notice that the baselines perform less stable under the Europarl-ST corpus with ASR-BLEU ranges from 23.4% to 34.2%. In the S2Sp scenario, both our RDC and DDM variants perform significantly better (p < 0.01) than the two baselines. Our S2U-DDM variant performs significantly better (p < 0.05) than UnitY and is comparable to the best cascaded system. Note that we only require one run training for bidirectional translations.

heading: 7.4 Inference Speed
text: We use a NVIDIA DGX-A100*8 workstation to perform the inferencing comparison without additional engineering optimization. We randomly select 500 utterances from the multi-domain Es→En dev set. For end-to-end S2ST inferencing, our final RDC with one-pass decoding achieved 1.72× decoding speed-ups over the best-performance UnitY (Inaguma et al., 2022) baseline which requires a two-pass text+unit decoding.

heading: 7.5 Human Evaluation
text: Finally, we performed an audio-only human evaluation to evaluate the translation quality and acceptances of the best baseline UnitY and our DDM. For direct comparison, we use the mTEDx test with 989 samples. We obtained a mean translation quality score of 4.202(/5.0) which is comparable to UnitY’s 4.197 and an acceptable ratio of 92.89% which is also comparable to UnitY’s 92.94%.

heading: 8 Conclusion
text: Aiming at effectively leveraging bidirectional supervision signals of speech-to-speech translation (S2ST), we have proposed two models for duplex S2ST, a reversible duplex Conformer and a duplex diffusion model. We compare with cascaded S2ST models, single/multi-pass speech-tospectrogram/unit models and report significantly better or comparable ASR-BLEU and humanevaluated scores, with fewer training time and faster inference speed.

heading: 9 Limitations
text: Our duplex diffusion model and reversible duplex Conformer architecture do not explicitly take reordering as an essential challenge. However, depicts the language pairs described in the experiments, there are languages such as English and Japanese which shares subject-verb-object (SVO) and subject-object-verb (SOV) word orders. These limit the scalability of our proposed methods and external pre-ordering (Zhao et al., 2018; Wu et al., 2011) or post-ordering (Goto et al., 2013) techniques on clustered units of speech should be taken into consideration in the future work.
Large-scale unlabeled speech data is required to train self-supervised wav2vec2.0 or HuBERT models. However, this is frequently not easy to collect. Moreover, it is even more difficult to collect paired speech-to-speech data and existing TTS models for generating speech from text are still under developing. These are pre-conditions of applying our proposed approaches.
Finally, we still need to train pair-by-pair for S2ST which is quadratic to the number of languages. Our approach is less effective than textual multilingual machine translation architecture in which linear number of translation models are required and achieved comparable results than pairwise baselines. Multilingual S2ST requires novel training architectures and inferencing algorithms.

heading: 10 Ethics Statement
text: Our target is to build direct speech-to-speech translation systems with a duplex idea of training both directions in one run. We try our best to reuse existing pretrained wav2vec2.0, HuBERT, mHuBERT and mBART models to save energy consuming. In one run, we require much less GPU-hours for obtaining S2ST models for both direction usages. However, compared with textual duplex MT systems, pre-processing of speech signals still requires much higher costing of GPU-hours and as listed in our limitation section (Section 9), smarter ways of multilingual S2ST architectures are preferred in the future to reduce the cost of energy from current quadratic to linear number of models.
Generally, S2ST circumvents traditional cascaded systems which concatenate ASR, MT and TTS with high latency and high requirements of datasets. There are 3,000 around languages in the world who do not have their own writing systems or textual vocabularies. Through our duplex S2ST models, we hope to be friendly to these languages so that more and more languages can be covered.

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
9
3 A2. Did you discuss any potential risks of your work? 10
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? 7

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? 7
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 6

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 7

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 7
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 7
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? 6
3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? 6
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: However, compared with textual duplex MT systems, pre-processing of speech signals still requires much higher costing of GPU-hours and as listed in our limitation section (Section 9), smarter ways of multilingual S2ST architectures are preferred in the future to reduce the cost of energy from current quadratic to linear number of models. Generally, S2ST circumvents traditional cascaded systems which concatenate ASR, MT and TTS with high latency and high requirements of datasets. There are 3,000 around languages in the world who do not have their own writing systems or textual vocabularies. Through our duplex S2ST models, we hope to be friendly to these languages so that more and more languages can be covered