DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech Translation
Authors: Qingkai Fang
Yan Zhou
Yang Feng
Venue: NeurIPS 2023 poster (2023)
Keywords: speech-to-speech translation
non-autoregressive translation
speech translation
directed acyclic transformer
TL;DR: A non-autoregressive two-pass speech-to-speech translation model with comparable performance to translatotron 2 and faster decoding speed.

Abstract: Direct speech-to-speech translation (S2ST) translates speech from one language into another using a single model. However, due to the presence of linguistic and acoustic diversity, the target speech follows a complex multimodal distribution, posing challenges to achieving both high-quality translations and fast decoding speeds for S2ST models. In this paper, we propose DASpeech, a non-autoregressive direct S2ST model which realizes both fast and high-quality S2ST. To better capture the complex distribution of the target speech, DASpeech adopts the two-pass architecture to decompose the generation process into two steps, where a linguistic decoder first generates the target text, and an acoustic decoder then generates the target speech based on the hidden states of the linguistic decoder. Specifically, we use the decoder of DA-Transformer as the linguistic decoder, and use FastSpeech 2 as the acoustic decoder. DA-Transformer models translations with a directed acyclic graph (DAG). To consider all potential paths in the DAG during training, we calculate the expected hidden states for each target token via dynamic programming, and feed them into the acoustic decoder to predict the target mel-spectrogram. During inference, we select the most probable path and take hidden states on that path as input to the acoustic decoder. Experiments on the CVSS Fr$\rightarrow$En benchmark demonstrate that DASpeech can achieve comparable or even better performance than the state-of-the-art S2ST model Translatotron 2, while preserving up to 18.53$\times$ speedup compared to the autoregressive baseline. Compared with the previous non-autoregressive S2ST model, DASpeech does not rely on knowledge distillation and iterative decoding, achieving significant improvements in both translation quality and decoding speed. Furthermore, DASpeech shows the ability to preserve the speaker's voice of the source speech during translation.

DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech Translation
Qingkai Fang!.2, Yan Zhou Yang Fengl.2. 1.2-
Key Laboratory of Intelligent Information Processing Institute of Computing Technology. Chinese Academy of Sciences (ICT/CAS)
University of Chinese Academy of Sciences, Beijing, China fall zhouyan23z, fengyang @ Bict... ac
Abstract
Direct speech-to-speech translation (S2ST) translates speech from one language into another using a single model. However, due to the presence of linguistic and acoustic diversity, the target speech follows complex multimodal distribution, posing challenges to achieving both high-quality translations and fast decoding speeds for S2ST models. In this paper, we propose DASpeech, non-autoregressive direct S2ST model which realizes fast and high-quality S2ST. To better capture the complex distribution of the target speech, DASpeech adopts the two-pass architecture to decompose the generation process into two steps, where linguistic decoder first generates the target text, and an acoustic decoder then generates the target speech based on the hidden states of the linguistic decoder. Specifically, we use the decoder of DA- Transformer as the linguistic decoder, and use FastSpeech 2 as the acoustic decoder. DA-Transformer models translations with directed acyclic graph (DAG). To consider potential paths in the DAG during training, we calculate the expected hidden states for each target token via dynamic programming, and feed them into the acoustic decoder to predict the target mel-spectrogram. During inference, that path as input to the acoustic decoder. Experiments on the CVSS Fr->En better
select the most probable path and take hidden states
benchmark demonstrate that DASpeech can achieve comparable or performance than the state-of-the-art S2ST model Translatotron 2, while preserving up x speedup compared to the autoregressive baseline. Compared with the previous -on-autoreeressive S2ST model, DASpeech does not rely on knowledge distillation and iterative decoding, achieving significant improvements both translation quality and decoding speed. Furthermore, DASpeech shows the ability preserve the speaker's voice of the source speech during translation
Introduction
Direct speech-to-speecl translation (S2ST) directly translates speech of the source language into the target language, which can break the communication barriers between different language groups and has broad application prospects. Traditional S2ST usually consists of cascaded automatic speech recognition (ASR), machine translation (MT), and text-to-speech (TTS) models [ 22. contrast, direct S2ST achieves source- sartceettttarget speech conversion with unified model 3 which (1)
avoid error propagation across sub-models (2) reduce the decoding latency S ; and (3) preserve
*Correspondigg author:
Yang Feng.
AAdio samples are available at Cooee publicly available httppp/://gghhhb...////////////o
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

non-linguistic information (e.g,, the speaker's voice) during translation [6] Recent works show that direct S2ST can achieve comparable or even better performance than cascaded systems 7 8)
Despite the theoretical advantages of direct S2ST, it is still very challenging to train a direct S2ST model in practice. Due the linguistic diversity during translation, well the diverse acoustic variations (e.g., duration, pitch, energy, etc.), the target speech follows a complex multimodal distribution. To address this issue, Jia al. [6). Inaguma et al. ? propose the two-pass architecture, which first generates the target text with a linguistic decoder, and then uses an acoustic decoder to generate the target speech based on the hidden states of the linguistic decoder. The two-pass architecture decomposes the generation process into two steps: content translation and speech synthesis, making it easier to model the complex distribution of the target speech and achieving state-of-the-ar performance among direct S2ST models.
Although the two-pass architecture achieves better translation quality, two passes of autoregressive decoding also incur high decoding latency. To reduce the decoding latency. Huang et al. 9 recently proposes non-autoregressive (NAR) S2ST that generates target speech in parallel. However, due to the conditional independence assumption of NAR models, it becomes more difficult to capture the multimodal distribution of the target speech compared with autoregressive models") Therefore, the trade-off between translation quality and decoding speed of S2ST remains pressing issue.
In this paper, we introduce S2ST model with both high-quality translations and fast decoding speeds: DASpeech, a non- non-autoregressive two two-pass direct S2ST model. Like previous two-pass models, DASpeech includes a speech encoder, a linguistic decoder, and an acoustic decoder. Specifically, the linguistic decoder uses the structure of DA-Transformer 11 decoder, which models translations via directed acyelic graph (DAG). The acoustic decoder adopts the design of FastSpeech 2 [12), which takes the hidden states of the linguistic decoder as input and generates the target -ee-sppecto.ram. During training, we consider all possible paths in the DAG by calculating the expected hidden state for each target token via dynamic programming, which are fed to the acoustic decoder to predict the target mel-sppetro During inference, we first find the most probable path in DAG and take hidden states on that path as imput to the acoustic decoder. Due to the task decomposition of two-pass architecture, as well as the ability of DA Transformer and FastSpeech 2 themselves model linguistic diversity and acoustic diversity. DASpeech is able to capture the multimodal distribution of the target speech. Experiments on the CVSS Fr->En benchmark show that: (1) DASpeech achieves comparable or even better performance than the state-of-ther -hettt S2ST model Translatotron 2, while maintaining up to 18.53 speedup compared to the autoregressive model. (2) Compared with the previous NAR S2ST model TranSpeech []. DASpeech no longer relies on knowledge distillation and iterative decoding, achieving significant ad advantages both translation quality and decoding speed. (3) When training on speech-to-speech translation pairs of the same speaker, DASpeech emerges with the ability to preserve the source speaker's voice during translation.
2 Background
2.1 Directed Acyelic Transformer
is proposed for non-autoregressive machine translation (NAT), which achieves comparable results to autoregressive Transformer [14] without relying on knowledge distillation. DA-Transformer consists of a Transformer encoder and an NAT decoder. The hidden states of the last decoder layer are organized as a directed acyclic graph (DAG). The hidden states correspond to vertices of the DAG, and there are unidirectional edges that connect vertices with small indices to those with large indices. DA-Transformer successfully alleviates the linguistic multi-modality problem since DAG can capture multiple translations simultaneously by assigning different translations to different paths in DAG.
Directed Acyelic Transformer (DA-Transformer) | [3]
Formally, given a source sequence X = (I1, xN) and a target sequence Y = (y/1,.., Yad). the encoder takes X input and the decoder takes learnable positional embeddings G = (g) (gi,..., -.., gt. as input. Here T. is the graph size, which is set to À times the source length, i.e., I= X. N, and À is a hyperparameter. Tr-araosformer models the translation probability Po (Y by marginalizing all
'It is known the multi multi-modality problem [0] in non-autoregressivve sequence generation.

---

possible paths in DAG:
Px(Y [=
(1)
AET
where A = (a1.-.. aM) is a path represented by sequence of vertex indexes with 1 = a1 <... < aM L, and contains all paths with the same length as the target sequence Y. The probability of path A is defined as:
M-1 Po(AIX) = II
M-1
Po X) - II ==1
Eat0:111
where E R Lx the transition probability matrix. We apply lower triangular masking on E to allow only forward transitions. With the selected path A, all target tokens predicted in parallel:
M II
M III 1=1
Po(Y|A,
Po a, X)
P
where P RRL is the prediction probability matrix, and V indicates the vocabulary. Finally, train the DA-Transformer by minimizing the negative log-likelihood loss:
LDAT log = log Σ
Po(Y (AX))
(4)
AET
which can be calculated with dynamic programming.
2.2 FastSpeech
FastSpeech 2 [12] non-autoregressive text-to-speech (TTS) model that generates mel-spectrograms from input phoneme sequences in parallel. It is composed of three stacked modules: encoder, variance adaptor, and mel-spectrogram decoder. The encoder and mel-spectrogram decoder consist of several feed-forward Transformer blocks, each containing self-attention layer followed by ID-convolutional layer. The variance adaptor contains three variance predictors including duration predictor, pitch predictor, and energy predictor, which are used to reduce the information gap between input phoneme sequences and output mel- spectroorrrssss. During training. the ground truth duration, pitch and energy are used to train these variance predictors and also conditional inputs generate the mel-speectoorram. During inference, we use the predicted values of these variance predictors. The introduction of variation information greatly alleviates the acoustic leads to better voice quality. The training objective of FastSpeech consists four terms:
modality problem, which
Crts
LL1 Ldur + Lpttch Lenergy:
(5)
where LL1 measures the distance between the predicted and ground truth mel-spectrograms, Ldurs fpitch and Lenergy compute the mean square error (MSE) loss between predictions and ground truth for duration, pitch, and energy, respectively.
3 DASpeech
In this section, we introduce DASpeech. non-autoregressive two-pass direct S2ST model that generates target phonemes and target mel-spectrograms successively. Formally, the source speech sequence is denoted X (z,,..., xx, N. where N is the number of frames the source speech. The sequences of target phoneme and target mel-spectrogram are represented by Y = (V1...,, UMM) and
(s. .... ST), respectively. DASpeech first generates from X with speech-to-ters translation (S2TT Transfoorfrrmrr Subsequently, it generates S with FastSpeech 2-style decoder conditioned on the last-layer hidden states the DA-Transformer. We first overview the model architecture of DASpeech in Section 3.1 In Section 3.2 we introduce our proposed training techniques that leverage pretrained S2TT DA- Transformer and FastSpeech models and finetune the entire model for S2ST end-to-end. Finally, present several decoding algorithms for DASpeech in Section 3.3

---

Target Mel-spectrogram
Non-autoregressive Acoustic Decoder
Traning Infenence
Directed
Linear
Target
Acyclic Graph
Softmax
Speech Encoder
Conformer
Attn
Non-autoregreessive Linguistic Decoder (Self-Attn Cross-Attn FFN)
Encoder
Nx
Positional
Subsampler
91
92
93
94
95
96
97
Js
Embeddings
Figure 1: Overview of DASpeech. The last-layer hidden states the linguistic decoder are organized DAG. During training, the input the acoustic decoder is the sequence of expected hidden states. During inference, it is the sequence of hidden states on the most probable path.
3.1 Model Architecture
As shown Figure T DASpeech consists three parts: a speech encoder, a non-autoregressive linguistic decoder, and non-autoregressive acoustic decoder. Below are the details of each part.
Speech Encoder Since our model takes speech features as input, we replace the Transformer encoder in the original DA-Transformer with speech encoder. The speech encoder contains subsampler followed by several Conformer blocks [15]. Specifically, the subsampler consists of two IDDcoovolutional layers which shrink the length of input sequences by a factor of 4. Conformer combines multi-head attention modules and convolutional layers together to capture both global and local features. We use relative positional encoding [16] in the multi-head attention module.
Non-autoregressive Linguistic Decoder The linguistic decoder identical the decoder of DA- Transformer, which generates the target phoneme sequence from the source speech in parallel. Each decoder layer comprises a self-attention layer, a aots-atttntionn layer, and feed-forward layer. The decoder takes learnable positional embeddings input, and the last-layer hidden states are organized as DAG to model translations, we described in Section 2.1
Non-autoregressive Acoustic Decoder The acoustic decoder adopts the design of FastSpeech 2, which generates target mel-speetrograms from the last-layer hidden states of DA-Transformer in parallel. The model architecture is the same as that introduced in Section 2.2 except that the embedding matrix of the input phonemes is removed, since the input has changed from the phoneme sequence to the hidden state sequence.
3.2 Training
DASpeech has the advantage of easily utilizing pretrained S2TT DA Transraasfrrmrr and FastS models. We use the pretrained S2TT DA-Transformer initialize the speech encoder and the non-autoregressive linguistic decoder, and use the pretrained FastSpeech 2 model to initialize the non-autoregressive acoustic decoder. Finally, the entire model is finetuned for direct S2ST. This pretraining-finetuiii pipeline simplifies model training and enables the use of additional S2TT and TTS data. However, end-to-end finetuning presents a major challenge due to the length discrepancy between the output from the linguistic decoder and the input to the acoustic decoder. Specifically, the hidden state sequence output by the linguistic decoder has a length of L = À N, while the input sequence required by the acoustic decoder should have length of M, which is the length of the ground truth phoneme sequence. Therefore, it is necessary to determine how to obtain the input
'In this work, speech-to-text translation refers speech to- phoneme translation if not otherwise specitied.

---

sequence of acoustic decoder Z = (Zi,...,Zad from the last-layer hidden states of linguistic decoder (VI..... Vr.). The following introduces our proposed approach: Expect- Path Training.
Expect-Patt Path Training Intuitively, the i-th input element 2; should be the hidden state of the vertex responsible for generating Yt- However, since there may be multiple vertices capable of generating each due to numerous possible paths, we would like to consider all potential paths. To address this issue, define z, the expected hidden state under the posterior distribution Pe (a,
2; Σ Pa(a; = J|x,)
(6)
where (a = V|X,Y) refers to the probability of vertex j being the i-th vertex path A, which means that Yi is generated by vertex j. We can compute (a = j|X, Y) as follows:
Po(a== = j|x,8) = Σ
1(4 j) -o((AX,))
AET
Po(Y,
Σ
1 1(4 = Eveer PolY, A(
(8)
AET
Σxer 1(a = -)()) Po)
(9)
ΣAET Po(Y,A|X)
where 1(a; 5) is an indicator function to indicate whether the i-th vertex of path vertex j. To calculate EAT L(a; = -) PY(Y, A and Aee (Y, Al in Equation (9), we employ the aerward-backwad algorithm [17]. which involves two passes of dynamic programming.
Forward Algorithm The forward probability is defined a;(}) = Po(y....., Vi. a; x|X), which is the probability generating the partial target sequence (y1,-.. .... Ui) and ending in vertex j at the i-th step. By definition, we have (1) = P1 P1.g1 and a (1 < < L) = O. Due to the Markov property,
we can sequentially calculate (-)(-) from its previous step - (-) as follows:
- J.8
a Eg..
(10)
Backward Algorithm The backward probability is defined as Be(j) Po (o VM |a; = J. X).
which is the probability of starting from vertex j at the i-th step and generating the rest of the target sequence (Vi+1,..., yMr). By definition, we have 3M (L) = 1 and Ba (1 I j < L) 0. Similar to
the forward algorithm, we can sequentially calculate (j) from its next step B.41 (f) as follows:
L
B,(1)
EyA (k) PREAI k=j+1
Σ
(11)
Recalling Equation (9). the denominator is the sum of the probabilities of all valid paths, which is equal to a. (L). The numerator is the sum of the probabilities of all paths with a, = j. which is equal to a;(i) B.()). Therefore, the Equation (6) can be calculated as:
ax(1) 3x() #=1
z, Po V, 1=1
Vz.
(12)
aM(L)
The time complexity of the forward-backward algorithm is O(ML²). Finally, the training objective of DASpeech as follows:
LDASpeech LDAT + μ. CrTs,
(13)
where j is the weight of TTS loss. The definitions of LDAT and Crrs are the same as those in Equations (4) and (5).
3.3 Inference
During inference, we perform two-pass parallel decoding. First, we find the most probable path A in DAG with one of the decoding strategies proposed for DA-Transformer (see details below). We then
5

---

feed the last- hidden states on path A" to the autoorergrresvvv acoustic decoder to generate the mpeesteet.....m Finally, the predicted meesspectrogram will be converted into waveform using a pretrained HiFi-GAN vocoder [18]. Since both DAG and TTS decoding are fully parallel, DASpeech achieves significant improvements in decoding efficiency compared to previous two-pass models which rely on two passes of autoregressive decoding. Considering the trade-off between translation quality and decoding efficiency, we use the following two decoding strategies for DA-Transformer in our experiments: Lookahead and Joint-Viterbi. Viterbi.
Lookahead Lookahead decoding sequentially chooses a, and y: in a greedy way. At each decoding step, it jointly considers the transition probability and the prediction probability:
a yr = arg max (u la,,
(14)
Viterbi Joint- Viterbi decoding 19 finds the global joint optimal solution of the translation and decoding path via Viterbi decoding 20)
A",Y" arg max Po(Y,
(15)
A.Y
After Viterbi decoding, we first decide the target length M and obtain the optimal path by backtracking from 4M = L. More details can be found in the original paper.
4
Experiments
4.1 Experimental Setup
Dataset We conduct experiments on the CVSS dataset 4. a large-scale S2ST corpus containing speech-to-speech translation pairs from 21 languages to English. It is extended from the CoVoST [2 S2TT corpus by synthesizing the target text into speech with state-of-the-art TTS models. It includes two versions: CVSS-C and CVSS-T. For CVSS-C, all target speeches are in a single speaker's 's voice. For CVSS-T, the target speeches are in voices transferred from the corresponding source speeches. We evaluate the models the CVSS-C French-Engllss (Fr-EEn) and CVSS-T Ernnch-Enggiish (Fr-=En) datasets. We also conduct a multilingual experiment by combining all 21 language directions in CVSS-C together to train a single Many-to-Engliise S2ST model.
Pre-processing We convert the source speech to 16000Hz and generate target speech with 22050Hz. We compute the 80-dimensional mel-filterbank features for the source speech, and transform the target waveform into mel-spectrograms following Ren et al. [2]. We apply utterance-level and global-level cepstral mean-variance normalization for source speech and target speech, respectively. We follow Ren et al. to extract the duration, pitch, and energy information of the target speech.
Model Configurations The speech encoder, linguistic decoder, and acoustic decoder contain 12 Conformer layers, 4 Transformer decoder layers, and feed-forward Transformer blocks, respectively. The detailed configurations can be found in Table in Appendix A For model regularization, we set dropout to 0.1 and weight decay 0.01, and no label smoothing is used. We use the HiFi-GAN vocoder pretrained on the VCTK datase (22) to convert the mel-spectrogram into waveform.
Training DASpeech follows the preraininggfinettning pipeline. During pretraining. the speech encoder and the linguistic decoder are trained on the S2TT task for 100k updates with a batch of 320k audio frames. The learning rate warms up to 5e-4 within 10k steps. The acoustic decoder is pretrained on the TTS task for 100k updates with a batch size of 512. The learning rate warms up to 5e-4 within 4k steps. During finetuning, batch of 320k audio frames. The learning rate warms up to le-3 within 4k steps. We use Adam optimizer 2 for both pretraining and finetuning. For the weight of TTS loss μı, we experiment with E {1.0, 2.0, 5.0, 10.0} and choose = 5.0 according to results on the dev set. We implement our model with the open-source toolkit toilkit 224. All models are trained on RTX 3090 GPUs.
train the entire model for 50k updates with
In the multilingual experiment, the presence of languages with limited data or substantial interlingual variations makes the mapping from source speech to target phonemes particularly challenging. To address this, we adopt a two-stage pretraining strategy. Initially, we pretrain the speech encoder and
Seee VCTK V1 in https: http::://githubl http:h/w......m/./////t../......

---

Table 1: Results on CVSS-CFr- and CVSS-T test sets. indicates results quoted from
Huang et al. 9 indicates results of re-implementation. t: target length beam=15 and noisy parallel decoding (NPD). Thonne Tunit, and mm indicate the sequence length of phonemes, discrete units, and sel-spectr,grams, respectively. means the improvements over S2UT are statistically
significant (p < 0.01).
ASR-BLEU (Fr->En)
Models
Decoding
#lter
#Param
Speedup
CVSS-C
CVSS-T
Ground Truth
84.52
81.48
/
Single single-pass antoregressive decoding
A1 A2
S2UT S
Tumit Autoregressive
Beam=10 10
73M
22.23
22.28
1.00x 2.32x
Translatotron
Tial
79M
16.96
11.25
Two-puss autoregressive decoding
B1 B2
UnitY 7
Bcam=
T Tphone + Tusis
64M
24.09 25.21
24.29
1.43x
Translatotron
Beam=10
phone T mel
87M
24.39
1.42x
Single-pass non cetrrrrtreeeesivv decoding
C1
TranSpeech
9
Iteration
5
67M
17.24
/
11,0x
c2
b=15 NPD1
Iteration
15 67M
18.39
/
2.53x
TranSpeech
Iteration
67M
16.38
16.49
12.45x
c4
b=15 NPD1
Iteration
15
67M
19.05
18.60
3.35x
Two-pass non-guuooregresivve decoding
D1
DASpeech (A = 0.5)
Lookahead
1+1
93M
24.71**
24.45**
18.53x
D2
Joint- Viterbi
1+1
93M
25.03
25.26
16.29x
D3
DASpeech
Lookahead
1+1
93M
24.41
24.17**
18.45
D4
(A = 1.0)
Joint-Viterbi
1+1
93M
24.80*
24.48
15.65
Cascaded systems
B1
S2T FastSpeech
Beam=10
Tohoue +
49M+4IM
24.71
24.49
E2
DAT FastSpeech 2 (A = 0.5)
Lookahead Joint-Viterbl
1+1
51M+4IM
22.19
22.10
/
R3
1+1
5IM+4IM
22.80
22.75
/
E4
FastSpeech (A L.0)
Lookahead Joint-Viterbi
1+1
5IM+4IM
22.68
22.57
E5
1+1
51M+4IM
23.20
23.15
the linguistic decoder using the speech-to- speech-to-subword task, followed by pretraining on the speech-to- phoneme task. In the second stage of pretraining, the embedding and output projection matrices of the decoder replaced and trained from scratch accommodate changes in the vocabulary. We employ this pretraining strategy for DASpeech, UnitY and Translatotron in the multilingual experiment. We learn the subword vocabulary with size of 6K using the SentencePiece toolkit.
We also adopt the glancing strategy 25 during training, which shows effectiveness in alleviating the multi-modality problem for NAT. It first assigns target tokens appropriate vertices following the most probable path Å arg max Po (Y, A|X), and then masks some tokens. We linearly anneal the unmasking ratio т from 0.5 to 0.1 during pretraining and fix T to 0.1 during finetuning.
Evaluation During finetuning, we save checkpoints every 2000 steps and average the last 5 check- points for evaluation. We use the open-source ASR-BI BLÜU toolkit" to evaluate the translation quality. The translated speech is first transcribed into text using a pretrained ASR model. SacreBI.EU (26) is then used to compute the BLEU score [27 and the statistical significance of translation results. The decoding speedup is measured on the test set using RTX 3090 GPU with a batch size of 1.
Baseline Systems We implement the following baseline systems for comparison. More details about the model architectures and hyperparameters can be found in Appendix A
S2UT (5]
Speech-to-unit translation (S2UT) model generates discrete units corresponding to the target speech with http://stt..oolooooeeeee////eres
model. We introduce the auxiliary task of predicting target phonemes to help the model converge.
www.speechhaee altp//sss__beuee

---

Translatotron B Translatotron generates the target mel- spectroggrmm with sequence -e-uueeeee model. We also introduce the auxiliary task of predicting the target phonemes.
Unity 7 UnitY is two-pass model which generates target phonemes and discrete units succes- sively 8
We remove the R- R-Drop training [28] for simplification. We pretrain the speech encoder and first-pass decoder on the S2TT task.
Translatotron [ Translatotron 2 is two-pass model which generates target phonemes and mpel-sppetrograms successively. We enhance Translatotron 2 by replacing LSTM with Transformer, and introducing an additional encoder between two decoders following Inaguma al. The speech encoder and first-pass decoder are pretrained on the S2TT task.
TranSpeech [9] TranSpeech is the first non-autoregressive S2ST model that generates target discrete units in parallel. To alleviate the acoustic multi-modality problem, TranSpeech introduces bilateral perturbation (BiP) to disentangle the acoustic variations from the discrete units. We re-implement TranSpeech following the configurations in the original paper.
S2T + FastSpeech The cascaded system that combines an autoregressive S2TT model and FastSpeech 2. The S2T model contains 12 Conformer layers and 4 Transformer decoder layers, which is also used in YnitY and Translatotron 2 pretraining.
DAT FastSpeech The cascaded system that combines the S2TT DA- Transformer model and FastSpeech 2. Both models are used in DASpeech pretraining.
4.2 Main Results
Table summarizes the results on the CV and SV-S-TFr--BR datasets. (1) Compared with previous autoregressive models, DASpeech (D1-D4) obviously surpasses single-pass models (A1, A2) and achieves comparable or even better performance than two-pass models (B1, B2), while preserving up to 18.53 times decoding speedup compared to S2UT. (2) Compared with the previous NAR model TranSpeech (C1-C4), DASpeech does not rely on knowledge distillation and iterative decoding, achieving significant advantages in both translation quality and decoding speedup. (3) DASpeech obviously outperforms the corresponding cascaded systems (D1-D4 vs. E2-E5), demonstrating the effectiveness of our expect-path training approach. We also find that the cascaded model prefers larger graph size (A = 1.0 is better) while DASpeech prefers smaller graph size (A 0.5 better). We think the reason is that a larger graph size can improve S2TT performance, but it also makes end-to-end training more challenging. We further study the effects of the graph size in Appendix C (4) On the CVSS-T dataset, which includes target speeches from various speakers, observe performance degradation in Translatotron and Translatotron 2 as the target mel-spectrogram becomes more difficult to predict. In contrast, DASpeech still performs well since its acoustic decoder explicitly incorporates variation information to alleviate the acoustic multi-modality, demonstrating the potential of DASpeech handling complex and diverse target speeches.
Table 2 shows the results on CVSS-C dataset of the multilingual X-+En S2ST model. We report the average ASR-BLE scores all languages, well the average scores high/middle/loo- resource languages [)) We find that DASpeech still obviously outperforms S2UT but performs slightly worse than Translatotron and YnitY in the multilingual setting, with an average gap of about 1.3 ASR-BLEU compared to Translatotron 2. However, DASpeech has about 13 times decoding speedup compared to Translatotron achieving better quality-speed trade-off.
Table 2: ASU- scores on CVSS-C test sets of the multilingual En S2ST model.
Table 3: ASR-BLEU scores the CVSS-C Fr-xEn test set with best-path training and expect- training.
Models
Avg.
High
Mid
Low
S2UT 5
5.15
16.74
6.24
0.84
Models
Best
Expect
YnitY 7
8.15
24.97
9.78
1.86
DASpeech (A = 0.5)
+ Lookahcad Joint- Viterbi
24.45
24.71
+0.26
Translatotron
66
8.74
25.92
11.07
2.04
24.84
25.03
+0.19
DASpeech
+ Lookahcad
7.42
22.84
9.51
1.41
DASpeech (A = 1.0)
Lookahead
24.18
24.41
+0,23
(A 0.5)
Joint- Viter-iterii
7.43
22.80
9.49
1.45
Viterbi
24.46
24.80
+0.34
Note that original UnitY uses subwords instead of phonemes. Here we use phonemes just for consistency with other systems.
The detailed results of each language pair be found in Table 6 Appendix
8

---

26,00
800
700
24.00
Decoding Time (mstaamgle))
600
22.00
DASpeech (Lookahead] DASpeech (jintViterbii
500
TranSpeech
20.00
S2UT
800
Unity
300-
18.00
Translatutron
2
200
16.00
100
14.00
10. 300)
[300, 600)
1600, 900)
1900, 1200)
1
4
10
12
14
16
18
20
Length of Source Speeich 14) Frames)
Speedup Ratio
Figure 2: Translation latency of different models across different source speech lengths, catego- rized into 4 groups based on source speech frame counts. The translation latency is computed as the decoding time on RTX 3090 GPU.
Figure 3: Trade-off between translation qual- ity and decoding speed. X-axis represents the speedup ratio relative to S2UT, and Y-axis rep- resents ASR-BLEU. The upper right represents better trade-off.
Table 4: Average speaker similar- ity on CVSS-TFr-BB test set.
4.3 Alternative Training Approach: Best-Path Training
In addition to the expect appect-path training approach proposed in Section 3.2 also experiment with simpler approach: Best- Path Training. The core idea is to select the most probable path Á = arg maX PY,Y, A|X) via Viterbi algorithm (20), and take the hidden states on path À = (a1...., â.s) input to the acoustic decoder, i.e., z, Va As shown Table 3] best- path training also performs well but is inferior to expect-path training. We attribute this to the fact that when using best-path training, only hidden states the most probable path participate TTS training, which may result insufficient training for the remaining hidden states. In contrast, all hidden states participate in TTS training with our expect-pate path training, which achieves better performance. The time complexity of Viterbi algorithm used in the best-path training is also O (MM12). More details about the best-path training can be found in Appendix
Speaker Similarity
Models
Ground Truth
0.48
Unit-based S2ST
S2UT S
0.03
UnitY 7
0.03
TranSpeech
Mel-spectrogian-bouse -hased S2S7
Trunslatotron [3]
0.04
Trunslatotron 2
6
0.05
DASpeech
Lookahead
0.14
(A 0.5)
Viterbii
0.10
4.4 Analysis Decoding Speed
In this section, we provide more detailed analysis of decoding speed. Figure Seuuree]stee shows the translation latency of different models for speech inputs of different lengths. The results indicate that the translation latency of autoregressive models (S2UT, Translatotron, UnitY, and Translatotron 2) significantly increases with the length of the source speech. contrast, the translation latency of nun-autoregresssvve models (TranSpeech and DASpeech) are hardly affected by the source speech length. When translating longer speech inputs, DASpeech's decoding speed can reach more than 20 times that of S2UT. Furthermore, illustrate the quality-speed trade-off of different models in Figure 3 By adjusting the hyperparameter X, the translation quality and decoding latency will change. Specifically,
increases, the decoding latency of the model will increase, and it achieves the best translation quality when À = 0.5. It is evident that DASpeech achieves the best trade-off between translation quality and decoding latency among all models. We further study the speedup under batch decoding in Appendix D
4.5 Voice Preservation
In this section, we investigate the voice preservation ability of direct S2ST models on the CVSS-T Fn dataset, where target speeches are in voices transferred from source speeches. Specifically, we use pretrained speaker verification mode 29 to extract the speaker embedding of the source

---

speech and generated target speech. We define the cosine similarity between source and target speaker embeddings as speaker similarity, and report the average speaker similarity on the test set in Table We find that: (1) unit-hased S2ST model can not preserve the speaker's voice since discrete units contain little speaker information; and (2) DASpeech can better preserve the speaker's voice than Translatotron and Translatotron 2, since its acoustic decoder explicitly introduces variation information of the target speech, allowing the model to learn more complex target distribution.
4
Related Work
translation (S2ST) extends speech-to-text translation 30 which further synthesizes the target speech. Translatotron [ the first S2ST model that directly generates target mel-spectrograms from the source speech. Since continuous speech features contain a lot of variance information that makes training challenging, Tjandra et al. [34], Zhang et al. (35 use the discrete tokens derived from VQ-VAE model [36] as the target. Lee et al. [537 extend this research line by leveraging discrete units derived from the pretrained HuBERT model 38 the target. To further reduce the learning difficulty, Inaguma et al. 77. Jia et al. [6. Chen et al. [39] introduce a two-pass architecture that generates target text and target speech successively. To address the data scarcity issue, some techniques like pretraining and data augmentation
Direct Speech-to-Spestt Translation
used to enhance S2ST [8 44). Huang et al. [9] proposes the first non-autoregressive S2ST model which achieves faster decoding speed. Our DASpeech extends this line of research and achieves better translation quality and faster decoding speed.
Von-autoregressive Machine Translation Machine translation based autoregressive decoding usually has a high decoding latency [45] Gu et al. [10] first proposes NAT for faster decoding speed.
To alleviate the multi-modality problem in NAT, many approaches have been proposed [46] like
knowledge distillation 49] latent-variable models [50] [1], learning latent alignments [52 56).
sequence-level training (8). and curriculum learning [25). Recently, Huang et al. [1I] introduce
DA- Drnnnosffrrmer, which models different translations with DAG to alleviate the mudti-modality
problem, achieving competitive results with autoregressive models. Ma et al. [59], Gui et al. [60)
further enhance DA- MA-Transformer with fuzzy alignment and probabilistic context-free grammar.
Non-autoregressive Text-to-Speech Ren et al. [61], Peng et al. [62] first propose non-autoregressive TTS that generates mel-speetrograms in parallel. FastSpeech 2 [12] explicitly models variance information to alleviate the issue of acoustic multi-modality. Many subsequent works enhance non-autoregressive TTS with more powerful generative models like variational auto-encoder ( (VAE) 63 64], normalizing flows 67), and denoising diffusion probabilistic models (DDPM) [68] 69]
DASpeech adopts the design of FastSpeech 2 for training stability and good voice quality.
Conclusion
In this paper, we introduce DASpeech, a non-autoregressive two-pass direct S2ST model. DASpeech built upon DA-Transformer and FastSpeech 2, and we propose an expect-path training approach to train the model end-to-end. DASpeech achieves comparable or even better performance than the state-of-the-ar S2ST model Translatotron 2, while maintaining up to 18.53 x speedup compared to the autoregressive model. DASpeech also significantly outperforms previous non-autoregreesivv model in both translation quality and decoding speed. In the future, we will investigate how to enhance DASpeech using techniques like pretraining and data augmentation.
7 Limitations & Broader Impacts
Limitations Although DASpeech achieves impressive performance in both translation quality and decoding speed, it still has some limitations: (1) the translation quality of DASpeech still lags behind Translatotron 2 in the multilingual setting; (2) the training cost of DASpeech is higher than Translatotron 2 (96 vs. 18 GPU hours) since it requires dynamic programming during training; and (3) the outputs of DASpeech are not always reliable, especially for some low-resource languages.
Broader Impacts In our experiments, we find that DASpeech emerges with the ability to maintain the speaker identity during translation. It raises potential risks in terms of model misuse, such as mimicking a particular speaker voice identification spoofing.
10

---

Acknowledgements
We thank all the anonymous reviewers for their insightful and valuable comments.
References
[1] A. Lavie, A. Waibel, L. Levin, M. Finke, D. Gates, M. Gavalda, T. Zeppenfeld, and Puming Zhan. Janus-ii: speech-to-speech translation in multiple languages. In 1997 IEEE International Conference Acoustics, Speech, and Signal Processing, volume 1, pages 99-102 vol.1, 1997. doi: 10.1109/CASSP. 199.AASSPP..99.9999 1997.599557
[2] S. Nakamura, K. Markov, H. Nakaiwa, G. Kikui, H. Kawai, T. Jitsuhiro, J.-S. Zhang, H. Ya- mamoto, E. Sumita, and S. Yamamoto. The atr multilingual speech-to-speech translation system. IEEE Transactions on Audio, Speech, and Language Processing. 14(2):365-376, 2006. doi: 00..9//SA..00..800777
[3] Ye Jia, Ron J. Weiss, Fadi Biadsy, Wolfgang Macherey, Melvin Johnson, Zhifeng Chen, and Yonghui Wu. Direct speech-to-speech translation with a
model. In Gernot Kubin and Zdravko Kacic, editors, Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019, pages 1123- 127. ISCA, 2019. doi: 2t021443///merppecch2201999919
URL https
[4] Ye Jia, Michelle Tadmor Ramanovich, Quan Wang, and Heiga Zen. CVSS corpus and mas- sively multilingual speech-to-speech translation. In Proceedings of Language Resources and Evaluation Conference (LREC), pages 6691-6703, 2022.
[5] Ann Lee, Peng-Jen Chen, Changhan Wang, Jiatao Gu, Sravya Popuri, Xutai Ma, Adam Polyak, Yossi Adi, Qing He, Yun Tang, Juan Pino, and Wei-Ning Hsu. Direct ppeech-to-speech translation with discrete units. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3327-3339, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 162 aww.eongllllo...o 235
URL htttps:/acclatthot
[6] Ye Jia, Michelle Tadmor Ramanovich, Tal Remez, and Roi Pomerantz. Translatotron 2: High- quality direct speech-to-speedd translation with voice preservation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song. Csaba Szepesvari. Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 10120- 10134. PMLR, 17-23 Jul 2022. URL https //proceedings. mlr press v 62/ jww.ia.aaiai.......
[7] Hirofumi Inaguma, Sravya Popuri, Ilia Kulikov, Peng-Jen Chen, Changhan Wang, Yu-An Chung, Yun Tang, Ann Lee, Shinji Watanabe, and Juan Pino. Unity: Two-pass direct speech-to-spest translation with discrete units, 2022.
[8] Sravya Popuri, Peng-Jen Chen, Changhan Wang, Juan Pino, Yossi Adi, Jiatao Gu, Wei-Ning Hsu, and Ann Lee. Enhanced direct speech- speech-to-speedd to-speech translation using sufevvuee pre-training and data augmentation. In Hanseok Ko and John H. L. Hansen, editors, Interspeech 2022, 23rd Annual Conference of the International Speech Communication Association. Incheon, Korea, 18-22 September 2022, pages 5195- 5199. ISCA, 2022. doi: 10.21 00221457/nterppecch22221111111 URL hol.https: /ooi /10 21437/ httpe//eeereeeeeeceeeeeee 202.eechhh.22211133.3
[9] Rongjie Huang, Jinglin Liu, Huadai Liu. Yi Ren, Lichao Zhang, Jinzheng He, and Zhou Zhao. Transpeech: Speech-to -o-h--tt---eehhhe translation with bilateral perturbation. In The Eleventh International Conference on Learning Representations, 2023. URL URL htttps://openreett
net. Forum? forrmmidd=VVAmFAt.... (=0VAAPAAtC55
[10] Jiatao Gu, James Bradbury, Caiming Xiong, Victor O. K. Li, and Richard Socher. Non- autoregressive neural machine translation. In 6th International Conference on Learning Rep- resentations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018 URL https:/ :tttpp://opperevr /openreviivv ew net/ wtffformm?mmm
B1 8Bt 1Cb

---

[11] Fei Huang, Hao Zhou, Yang Liu, Hang Li, and Minlie Huang. Directed acyclic transformer for -onreetorerressive machine translation. In Proceedings of the 39th International Conference on Machine Learning, ICMI. 2022, 2022.
[12] Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech 2: Fast and high-quality end-to-end text to speech. In International Conference on Learning Representations, 2021. URL pettps://openzeeee
[13] Fei Huang, Pei Ke, and Minlie Huang. Directed acyclic transformer pre-training for high- quality aonaatoregressive text generation. Transactions of the Association for Computational Linguistics, 2023.
[14] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL htt.atsss//ppooeddgs..neeeiioiie wwtp:riippp.ppppppppppppp http://pppppppaeaaaeaaaeeeeeeeee files/paper/2017//5 3 3ffe222555766e1111bb333c3cc42222
[15] Anmol Gulati, James Qin, Chung-Chent Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang. Yonghui Wu, and Ruoming Pang. Conformer: Convolution- augmented Transformer for Speech Recognition. In Proc. Interspeech 2020. pages 5036-5040, 2020. doi: 2222444//nneespeccch2202-2----
[16] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-Xie Attentive language models beyond fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978-2988, Florence. Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/ P19-1285. URL https:
9- 1285
[17] L.R. Rabiner. A tutorial on hidden markov models and selected applications in speech recogni- tion. Proceedings of the IEEE, 77(2):257-286, 1989. doi:
[18] Jungil Kong, Jachyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS 20, Red Hook. NY, USA, 2020. Curran Associates Inc. ISBN 0781713829544
[19] Chenze Shao, Zhengrui Ma, and Yang Feng. Viterbi decoding of directed acyclic transformer for non-autoregressive machine translation. In Findings of EMNLP 2022, 2022.
[20] A. Viterbi. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE Transactions on Information Theory, 13(2):260-26) 1967. doi: 10. 1957.10
[21] Changhan Wang, Anne Wu, and Juan Pino. Covost 2: A massively multilingual speech-to-text translation corpus, 2020.
[22] Christophe Veaux, Junichi Yamagishi, and Kirsten MacDonald. Cstr vetk corpus: English multi-speaker corpus for cstr voice cloning toolkit. 2017.
[23] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA. USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL futtp:/aarxe
org/ abs/ 1412. 6980
[24] Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations, 2019.
[25] Lihua Qian, Hao Zhou, Yu Bao, Mingxuan Wang, Lin Qiu, Weinan Zhang, Yong Yu, and Lei Li. Glancing transformer for -uooeereeere neural machine translation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International
12

---

Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1993-2003, Online, August 2021. Association for Computatiooal Linguistics. doi: 10.18653/11/2021 acl-long. 155. _ htt.t:t/pp:aaaa.
httphhoooooo..oo///...eeeee---- 155
[26] Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Pupers, pages 186-191, Brussels, Belgium, October 2018.
Association for Computational Linguistics. doi: 10. (0.18653/11/1886665
URL https
7/www ac web. www.:/wwwww...ooooooooooo www.nhhhooogyyiiooo 63319
[27] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic
evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. pages 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguisties. doi: 10.3 10.3115/1073383.1000 1073313 www.waclweb.co org/
URL https
[28] xiaobo liang, Lijun Wu, Juntao Li, Yue Wang. Qi Meng, Tao Qin, Wei Chen, Min Zhang, and Tie-Yan Liu. R-drop: Regularized dropout for neural networks. In M. Ranzato, A. Beygelz- imer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Infor- mation Processing Systems, volume 34, pages 10890-10905, Curran Associates, Inc., 2021. URI htt.:ttps..//rrccccngnng.neeerre
[29] Li Wan, Quan Wang, Alan Papir, and Ignacio Lopez Moreno. Generalized end-to-end loss for speaker verification. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), page 4879 4883. IEEE Press, 2018. doi: 10.1100//CSSS..201 84666665. htth:tt:: doi. org 10. 1109 CASSP 2018 8. 8462665
[30] Qingkai Fang, Rong Ye, Lei Li, Yang Feng, and Mingxuan Wang. Stemm: Self-learning with speech-text manifold mixup for speech translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 2022.
[31] Qingkai Fang and Yang Feng. Back translation for speech-to- to-text translation without transcripts. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, 2023.
[32] Qingkai Fang and Yang Feng. Understanding and bridging the modality gap for speech translation. In Proceedings of the 6l Annual Meeting of the Association for Computational Linguistics, 2023.
[33] Yan Zhou, Qingkai Fang, and Yang Feng. CMOT: Cross-modal mixup via optimal transport for speech translation. In Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7873-7887, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. we111166311/2223... aoln/// URL 436
[34] Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura. Speech-to-speech translation between untranseribed unknown languages. In 20/9 IEEE Automatic Speech Recognition and Under- standing Workshop (ASRU), pages 593-600, 2019. doi: 10. 109//SS4666499 1. 220119990000 100990003853.
[35] Chen Zhang, Xu Tan, Yi Ren, Tao Qin, Kejun Zhang, and Tie Yan Liu. Uwspeech: Speech to speech translation for unwritten languages. Proceedings of the AAAI Conference on Artificial Intelligence, (5(11))11319-143333 //ojs. aaai. org/ wnnex.niix.eii php/
May 2021. doi: 10.1609/aai. v35116. 17684. URL https
[36] Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu. Neural discrete representation learning. In I. Guyon, U. Von Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL http:ttpss./poocceeeeeeeeee htwp://weeeeeeioo...e.eeeeeeeeee neur ips wwtp:erww.pp...pp..eeeeeeeeeee filees/ee/// pdff
[37] Ann Lee, Hongyu Gong, Paul-Ambroise Duquenne, Holger Schwenk, Peng-Jen Chen, Chang- han Wang, Sravya Popuri, Yossi Adi, Juan Pino, Jiatao Gu, and Wei-Ning Hsu. Textless speech-to-speech translation real data. In Proceedings of the 2022 Conference of the North
13

---

American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 860-3 872, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. 16253////022..aacllla 2022. maraacl-maiin 63
URL https: http:psss//c..aaaoooooooooooo.
[38] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Trans. Audio, Speech and Lang. Proc., 29: 3451-3460, oct 2021. ISSN 2329-9290. doi: 10. 109/TASLP22011311229999 URL https
7/doi.org// /10 11011119990000 TASLP 2021.312282
[39] Peng-Jen Chen, Kevin Tran, Yilin Yang. Jingfei Du, Justine Kao, Yu-An Chung, Paden Tomasello, Paul-Ambroise Duquenne, Holger Schwenk, Hongyu Gong, Hirofumi Inaguma, Sravya Popuri, Changhan Wang. Juan Miguel Pino, Wei-Ning Hsu, and Ann Lee. Speech-to- speech translation for A real-world unwritten language. CoRR, abs/2211 1.66474, 2022. doi: 10. 48550/arXiv.221 1.06474. URL htLLhttp::// /ww.//doii.. wwt.oo.g//....o.oooooooooo......
48111..485500 40w.0/855////// 30t5///vaavvv..oo 2211 1...1....11 064
[40] Ye Jia, Yifan Ding, Ankur Bapna, Colin Cherry, Yu Zhang, Alexis Conneau, and Nobu Morioka. Leveraging unsupervised and weakly-supervised data to improve direct speech- to-speech translation. In Hanseok Ko and John H. L. Hansen, editors, Interspeech 2022, 23rd the Annual Conference of the International Speech Communication Association, Incheon, Korea,
18-22 September 2022. pages 1721-1725. ISCA, 2022. doi: 10.21437/mmerspeecchhoo w0t.:eeccc.hh..ooo00 2022-1098... httthttp:// doi.oi.... org 10, 21437/144110 htt.33//nnnneeeeeeeeeeeeeee 2022 10938
[41] Qianqian Dong, Fengpeng Yue, Tom Ko, Mingxuan Wang, Qibing Bai, and Yu Zhang. Leveraging pseudo-labeled data to improve direct speech-to-specch translation. In Hanseok
Ko and John H. Hansen, editors, Interspeech 2022, 23rd Annual Conference of the In- ternational Speech Communication Association, Incheon, Korea, 18-22 September 2022, pages 1781 1785. ISCA, 2022. doi: 10.21433/mmerrpeehh2200 10011. URL https
7/doi org 0tt.0.1441///nteeeeeeeeeeeee 2022 10011
[42] Xuan-Phi Nguyen, Sravya Popuri, Changhan Wang, Yun Tang, Ilia Kulikov, and Hongyu Gong, Improving speech-to-speech translation through unlabeled text. arXiv preprint arXiv:2210. 14514, 2022.
[43] Kun Wei, Long Zhou, Ziqiang Zhang, Liping Chen, Shujie Liu, Lei He, Jinyu Li, and Furu Wei. Joint pre-training with speech and bilingual text for direct speech to speech translation. In ICASSP 2023 2023 IEEE International Conference Acoustics. Speech and Signal Processing (ICASSP), pages 1-5, 2023. doi: 10. 100.CCASSP9935. 27..
[44] Ziqiang Zhang, Long Zhou, Chengyi Wang, Sanyuan Chen, Yu Wu, Shujie Liu, Zhuo Chen, Yan- qing Liu, Huaming Wang. Jinyu Li, Lei He, Sheng Zhao, and Furu Wei. Speak foreign languages with your own voice: Cross-lingual neural codec language modeling- CORR, abs/2303.03326 URL https::/ddo
2023. doi: 00448550/axxiw.2333.00 033926.
2303 03926
[45] Shaolei Zhang, Qingkai Fang, Zhuocheng Zhang, Zhengrui Ma, Yan Zhou, Langlin Huang, Mengyu Bu, Shangtong Gui, Yunji Chen, Xilin Chen, and Yang Feng. Bayling: Bridging cross- lingual alignment and instruction following through interactive translation for large language models, 2023.
[46] Jiatao Gu and Xiang Kong. Fully non-autoregresssvv neural machine translation: Tricks of the trade. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 120-133, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/ v1/2021 122202..indings-ze ac URL https: httptaaa/acaaanttolooo.oooo 11
[47] Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference Empirical Methods in Natural Language Processing, pages 1317-1327, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10. 18653/1/
D16- 1139. utthttss: http:s:///acaaaaooooooooooooo
14

---

[48] Chunting Zhou, Jiatao Gu, and Graham Neubig. Understanding knowledge distillation in non- autoregressive machine translation. In International Conference on Learning Representations. 2020. U
wew. net. /er/fo dtt.rummmmdddPyyyAAEE.
[49] Chenze Shao, Xuanfu Wu, and Yang Feng. One reference is not enough: Diverse distillation with reference selection for non-autooregressive translation. In Proceedings of NAACL 2022, 2022.
[50] Raphael Shu, Jason Lee, Hideki Nakayama, and Kyunghyun Cho. variatev non- autoregressive neural machine translation with deterministic inference using a delta pos- terior. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium
Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8846-8853, AAAI Press, 2020. URL
https: //oss aaai. org/ index.nde... whp hhtp:pppp//taaaaaaaaa.. pttp://tt.aaattceee/eeieeeeeeeee http:///...aaaaaaaooooaaa/aaaaaa
6413
[51] Yu Bao, Hao Zhou, Shujian Huang, Dongqi Wang. Lihua Qian, Xinyu Dai, Jiajun Chen, and
Lei Li. latent-glt glat: Glancing latent variables for parallel text generation. ACL. 2022. URL
https://Bat //apiv...iivv..oo org/ abs/ 2204. 02030
[52] Jindřích Libovický and Jindřich Helel. End-to-end non-autoregressive neural machine translation with connectionist temporal classification. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pages 3016-3021, Brussels, Belgium, October- November 2018. Association for Computational Linguistics. doi: 10. 10.18653/v1/D18- 1336. URL 1336
[53] Chitwan Saharia, William Chan, Saurabh Saxena, and Mohammad Norouzi. Non Non-autoregreesive machine translation with latent aligoments. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1098-1 1108, Online, November URL
2020. Association for Computational Linguistics. doi: 10. 10.11865/v1112 wttp:///000000000mmaaaea httts::/acclattholl mmgp:/....oor00000..mmmaoaaaaaaa 88.
[54] Cunxiao Du, Zhaopeng Tu, and Jing Jiang. Order-agnostic cross entropy for non-autoregressive machine translation. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 2849-2859, PMLR, 2021. html
URL
m.r. wt..rprres//vv3//ttt..
[55] Chenze Shao and Yang Feng. Non-monotonic latent alignments for CTC-based non- autoregressive machine translation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances Neural Information Processing Systems, 2022. URL
https:///penreviees wtweeeeiwwweeeeeeeeeeeeeeeee o
[56] Zhengrui Ma, Shaolci Zhang, Shoutao Guo, Chenze Shao, Min Zhang, and Yang Feng. Non- autoregressive streaming transformer for simultaneous translation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.
[57] Chenze Shao, Jinchao Zhang, Yang Feng, Fandong Meng, and Jie Zhou. Minimizing the bag-of- ngrams difference for non-autoregressivee neural machine translation. Proceedings of the AAAI Conference on Artificiall Intelligence, 34(01): Apr. 2020. doi: 10.1609/aaii v34i01. 5351. StRLLhhtps:///omm www.:/...aaaa...ommm htttpp:://oss...aa...r//////eees
[58] Chenze Shao, Yang Feng, Jinchao Zhang, Fandong Meng, and Jie Zhou. Sequence-level training for -ttorreereeee neural machine translation. Computational Linguistics, 47(4):891192 December 2021. doi: 10.1162/col) a 00421. URL http::/pess/acclaatooooooooo////
(7(4))))
cl-48 11-4.29 29
[59] Zhengrui Ma, Chenze Shao, Shangtong Gui, Min Zhang, and Yang Feng. Fuzzy alignments in directed acyclic graph for non-autoregressive machine translation. In The Eleventh Interna- tional Conference on Learning Representations, 2023. URI petttss:/oppnrevis.e hto//pppervvveeee ew. net/
Eorum? ien2iii-ii LSz L
15

---

[60] Shangtong Gui, Chenze Shao, Zhengrui Ma, Xishan Zhang, Yunji Chen, and Yang Feng. Non- autoregressive machine translation with probabilistic context-free grammar. In Advances in Neural Information Processing Systems, 2023.
[61] Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie- Yan Liu. Fastspeech: Fast, robust and controllable text to speech. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Alché-Bur Emily B. Fox, and Roman Gamett, editors. Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8- 14, 2019, Vancouver, BC, Canada, pages 3165 2019. URL wtt.ttps:: httpttttpppp/pppoooeeeeeeeeeeeee neur neur cc pcc/paper/201 9/hash/
[62]
Kainan Peng. Wei Ping, Zhao Song, and Kexin Zhao. Non-autoregressivee neural text-to-speech. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages
7586-7598. PMLR, 2020. DMLLR//2.0.LRLLLCC//////CCCEEEE hhttp:://prceeeinngg..... mms..mmrr.... /tt.E/////.oooood......///////// html
[63] Yoonhyung Lee, Joongho Shin, and Kyomin Jung. Bidirectional variational inference for non-autoregrressive text-to-speech. In International Conference on Learning Representations,
2021. UUR[atttss//ppeereviiie
[64] Xu Tan, Jiawei Chen, Haohe Liu, Jian Cong, Chen Zhang, Yanqing Liu, Xi Wang. Yichong Leng, Yuanhao Yi, Lei He, Frank K. Soong, Tao Qin, Sheng Zhao, and Tie- Yan Liu. Naturalspeech: End-to-end text to speech synthesis with human-level quality. CoRR, abs/220550042222 2022. doi: 20.48550/rXi..2255..omm
URL wttp:tttt://. prstps://dol //or: www.do.......oooooooooooooooeeee .org/1 /00. 188550/arxis3 2205
04421
[65] Chenfeng Miao, Shuang Liang. Minchuan Chen, Jun Ma, Shaojun Wang. and Jing Xiao. Flow- its: non-autoreeressive network for text speech based on flow. In 2020 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2020, Barcelona, Spain, May 4-8, 2020, pages 1229-7211 IEEE, 2020. doi: 10.1109//CSSPP40070 RTA.AASSP44777.2222999...4 2020.9054484. URL https: //o11 org. / /10. 109/ CCASSP 40776 6 2020 9054484
[66] Jaehyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. Glow-tts: A generative flow for text-to-speech via monotonic alignment search. In Proceedings of the 34th International Conference Neural Information Processing Systems, NIPS 20, Red Hook, NY, USA, 2020.
Curran Associates Inc. ISBN
[67] Yi Ren, Jinglin Liu, and Zhou Zhao. Portaspeech: Portable and high-quality generative text-to-speech. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Pro- cessing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 13963- 13974, 2021. URL 748d6b6ed8el 3
wewpeiinnnnneeuiiigg. nww.ips..p.... cC/ wttpcc//ppppeee ppaper/202/haass A.busB-Abbbract..... html
[68] Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail A. Kudi- nov. Grad-tts: diffusion probabilistic model for text-to-speech. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 Proceedings of Machine Learning Research, pages 8608. PMLR, 2021. URL http: httt:://proceeding iins ml r.pprssss press/ v139/popov2la.115 html
[69]
Rongjie Huang, Max W. Y. Lam, Jun Wang, Dan Su, Dong Yu, Yi Ren, and Zhou Zhao. Fastdiff: A fast conditional diffusion model for high-quality speech synthesis. In Lue De Raedt, editor, Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, pages 4157-4163 ijcai.org, 2022, doi: 10.24963/ijcai. 2022/577. URL https:: http:/::///o..oog////./o
.2022/577
16

---

A Details of Baseline Models
In our experiments, implement five baseline systems using fairseq: S2UT, Translatotron, UnitY, Translatotron 2, and TranSpeech. We reproduce TranSpeech with their open-source implementa- tions
In this section, mainly introduce the configurations of the other four baseline systems.
Figure 4 shows the model architectures these models. terms of model architecture, S2UT and Translatotron are single-pass S2ST models while UnitY and Translatotron are two-pass S2ST models. In terms of predicted targets, S2UT and UnitY predict discrete units while Translatotron and Translatotron 2 predict mel- speecrrrroogs.ss Below we describe the details of each model. The detailed hyperparameters can be found in T
HH AHD L OW1
27 38 9 11 30
HiH AHD L OW1
27 38 9 11 3D
Acoustic Decoder
Acoustic Decodar
Linguistic Decoder
Acoustic Decoder
Linguistic Decoder
Acoustic Decoder
Techsto-Sowechh
Testeto-paacch
HHH AHD OW1
HH AHD L OW1
Uiguustiic Decoder
Lingistiic Decoder
Speech
Speech Encoder
Encoddr
Speech
Speech Encoder
Encocer
(a) S2UT
(b) Translatotron
(c) UnitY
(d) Translatotron
Figure 4: Overview of baseline models.
S2UT Our implemented S2UT model includes three parts: a speech encoder, a linguistic decoder, and an acoustic decoder. The speech encoder is the same as DASpeech. The linguistic decoder is appended to the top layer of the speech encoder for multi-task learning, which predicts the target phonemes during training. The acoustic decoder generates the reduced discrete units derived from the 11-th layer of the pretrained mHuBERT mode We do not include other auxiliary tasks and remove CTC decoding in Lee et al. [5] for simplification. The model trained from scratch for 100k steps. We use beam search with a beam size of 10.
The speech encoder and linguistic decoder of Translatotron are the same as S2UT. The acoustic decoder generates mel-spectrograms autoregressively. The pre-net dimension is 32 and the reduction factor of the acoustic decoder is 5. The model is trained from scratch for 100k steps.
Translatotron
UnitY YnitY is a two-pass model that includes four parts: a speech encoder, a linguistic decoder, a text-to-speech encoder, and acoustic decoder. The architecture of the speech encoder, linguistic decoder, and acoustic decoder are the same as S2UT. The additional text-to- -pxeeth encoder is used to bridge the gap in representations between two decoders. We remove R-Drop training for simplification. We first conduct S2TT pretraining and finetune the model for 50k steps. We set the beam size of the first-pass and second-pass decoder to 10 and respectively.
Translatotron
The model architecture of Translatotron is similar to UnitY except that the second decoder generates mel-spectrograms rather than discrete units. The reduction factor of the acoustic decoder is set to 5. We first conduct S2TT pretraining and finetune the model for 50k steps. The beam size is set to 10 for the first-pass decoder.
For all the above models, we save checkpoints every 2000 steps and average the last checkpoints for evaluation, which is the same DASpeech. For S2UT and UnitY, we use the pretrained unit- based HiFi-GAN vocoder to synthesize waveform. For Translatotron and Translatotron 2, we use the same pretrained HiFi-GAN vocoder as DASpeech.
base vp en fr it3
L11
km1000.bin
hi 1a
17

---

Table 5: Hyperparameters DASpeech and baseline models.
Hyperparameters
S2UT
Translatotron
UnitY
Translatotron 2
DASpeech
kernel, encoder type encoder layers encoder emnhed dim encoder Jin, embed dim
(5,5)
(3,5
(3,3) 5)
(3.3)
(3.5)
conformer
conformer
conformer
conformer
conformer
12
12
12
12
Speech Encoder
256
256
256
256
2048
2048
2048
2048
2048
encoder, sttention, heads
4 relative
eacoder www__poss__p____e depthwise conv kernel
enc_
relative
relative
relative
rolative
decnder layers
4
4
decoder, eemeed dim
512
512
512
512
512
Linguistic Decoder
decoder_fie faclcr_fftt_eeeeeeeeeeeee embed dim
2048
2048
2048
2048
2048
decoder ottention heads
label. smoothing x2t weightt
0.1
0.1
0.1
0.1
0.0
8.0
encoder layers encoder, embee dim
2
Text-to-Speech Encoder
encoder_fie ffn. embed dim encnder attention,
2048
512
2048
512
Jayers
6
6
decoder, eembed
512
512
decoder_fie _ffffa_______ embed dim
2048
2048
2048
2048
1024
decoder atteentionn heads
label, seee_shhihhigg
0.1
0.1
Acoustic
n per per_slep
unit dictionary, size
1000
1000
var_ var_pred_ hidden, bid_ree__hideenn dim
256
var pred kernel size
prod, dropout pred_dropout
<2s loss weight
1.0
1.0
1.0
1o-3
1c-3
Ir lc_schedullr warmup updates warmup ir optimizer dropout tokens weight_ deeay
inverse sqit
inverse_see sqir
inverse sqrr
inverse sqrt
inverse. sqit
4000
4000
4000
4000
4000
le:7
le:7
le-7
le:7
le:7
Training
Adam
Adam
Adam
Adam
Adam
0.1
0.1
0.1
40Kx4
40kx4
40kx4
40kxx
40&x8
0.0
0.0
clip pomm
1.0
1.0
1.0
1.0
1.0
max, update
B Detailed Results CVSS-C X---C--BB Datasets
Table 6 summarizes the detailed results of each language pair on CVSS- test sets of the multilin- gual X En S2ST models.
Table 6: Results on CVSS-C test sets of the multilingual X En S2ST models.
High
Mid
Models
Avg.
Fr
Es
Fa
Ru
S2UT 5
5.15
19.65
13.35
15.37
18.58
1.43
14.47
7.94
0.93
6.42
UnitY
8.15
27.27
20.81
24.22
27.58
3.63
21.68
10.86
4.16
8.56
Translatotron
6
8.74
28.04
21.54
25.34
28.77
4.23
23.66
13.41
4.49
9.54
DASpeech
Lookahead
7.42
25.43
17.87
22.58
25.49
3.01
20.80
12,96
2.86
7.90
(A 0.5)
Joint-Viterbi
Joint- Viterbi
7.43
25.39
18.36
22.33
25.10
2.81
20.76
12.94
3.05
7.89
Low
Models
Ar
Lv
Cy
Ta
S2UT S
4.67
0.52
0.36
0.14
0.56
0.39
0.73
1.28
0.66
0.17
0.20
0.38
UnitY Translatotron
3.79
1.50
1.74
0.10
0.15
4.58
1.12
1.35
1.37
1.50
0.10
0.33
DASpeech 0.5)
Lookahead
9.04
1.75
0.04
0.08
0.64
1.43
1.20
1.33
0.70
0.09
0.29
Joint-Viterbi
9.43
1.66
0.48
1.48
1.30
0.85
0.09
0.32
18

---

C Effects of the Graph Size
In this section, we investigate how the graph size affects the performance. vary the size factor from 0.25 to 1.5, and measure the translation quality of both the S2TT DA Transformer model and DASpeecc on the CVSS-C Fr->En test set. As shown in Figures and 6 the performance of S2TT TA-Trassformer keeps increasing as the graph size gets larger, which is consistent with the observations in machine translation [9]. However, DASpeech performs best at A 0.5 and shows performance drop at larger X. We speculate that this is because larger graph size makes and-to-end training more challenging. We will investigate this issue the future.
we observe that
54.50
54.24
25.
53.89
24.96
24,80
53.35
24. 6.
BLEU
53.50
ASR-aIEU
24.41
24:37
53.00
52.71
52,90
24.06
Lookahead Joint-Viterbi
Lookahead
Viteriiterbi
Szee Factor A
Size Factor
Figure 5: Phoneme-level BLEU scores of the S2TT DA- DA-Transformer under different size fac- tor A.
Figure ASR-BLEU scores of DASpeech under different size factor A.
D Speedup Under Batch Decoding
As Gu and Kong [46] pointed out, the speed benefits of non-autoregressive models may degrade during batch decoding. To better understand this problem, we evaluate the speedup ratio under different decoding batch sizes. As shown in Figure 7 the speedup ratio keeps dropping the decoding batch size increases. Nevertheless, DASpeech (A 0.5 with Joint- Viterbi decoding) still achieves more than 6x speedup with decoding batch size of 64 and maintains comparable performance with Translatotron 2.
b-
b=-32
bw
25.20
Transiatotron
25.00
24,80
-SR-BLEU
!
24.60
24.40
DASpeech (A=0 5) Lookahead
24.20
DASpeech (A-0 Joint-Viterbi
DASpeach (A-1
24.00
DASpeach (111 Joint-Vitarbl
10
12
14
16
18
Speedup Ratio
Figure 7: Speedup ratio compared to S2UT baseline (not shown in the figure) and ASR-BLEU score on the CVSS-C Fn test set with different batch decoding sizes (b E {1,8, 16,32, 32, 64)).
19

---

E Best-Path Training
pest-patt-training selects the most probable path Å = (a....., aM) and takes the hidden states path Á as imput to the acoustic decoder. Formally, given the target phoneme sequence Y, the most probable path A arg max () via Viterbi algorithm [20]. Specifically, oi( to denote the probability of the most probable path so far (ā,, .... @z) with 0; j that generates (y),- -.:: yt). Considering the definition of αj
find
have 61 (1) P an and (1 j L) O.
For > 1, we can sequentially calculate 8,(-) from its previous step 6.-1(-) due to the Markov property:
8τ( = mmax(6:-11 Ex. P...):
(16)
(,()) = arg max(6 Eg P Plax); k-j
(17)
where (b(() stores x,- of the most probable path so far (ai,..., 0; = j). After M iterations, we can obtain the most probable path by backtracking from äM = L:
à = (
(18)
Finally, we select the hidden states on the most probable path, i.e., Z; Va. as the input sequence of the acoustic decoder.
20


Reviews:
[{"review_id": "hx70zmc61D", "reviewer": "Reviewer_47bw", "summary": "The paper introduces DASpeech, a directed acyclic transformer model for fast and high-quality speech-to-speech translation (S2ST). DASpeech employs a two-pass architecture, using a linguistic decoder based on DA-Transformer and an acoustic decoder based on FastSpeech 2. The proposed model captures the complex multimodal distribution of the target speech and achieves comparable or better performance than Translatotron 2, with significant speedup compared to autoregressive models. DASpeech also demonstrates the ability to preserve the source speaker's voice during translation.\n", "questions": "Could the authors elaborate on any specific differences between DASpeech and the DA Translation particularly?", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "By incorporating the DA-Transformer and FastSpeech 2, DASpeech leverages the strengths of these models to capture both linguistic and acoustic diversity, resulting in high-quality translations.", "weaknesses": "1. The novelty is not enough, the method of this article seems to just learn from the DAT method of NMT and use it in ST \n2. While the paper addresses the challenges of high decoding latency in S2ST models, it would be beneficial to provide more detailed analysis or experimentation on the trade-off between translation quality and decoding speed in DASpeech.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for providing your valuable and constructive feedback! Please find below the responses to each comment.\n\n1. **Re: The novelty is not enough, the method of this article seems to just learn from the DAT method of NMT and use it in ST (Specific differences between DASpeech and the DA-Transformer particularly)**\n\nThank you for pointing that out. In our opinion, applying the DA-Transformer to **speech-to-text translation** is straightforward as the prediction target remains the target text, and only the text encoder needs to be replaced with a speech encoder. **However, its application to speech-to-speech translation (S2ST) is non-trivial.** For the two-pass S2ST model, predicting the target text is just the first step; it also requires an acoustic decoder to further synthesize the target speech based on the output hidden states from the linguistic decoder. During the training process, **we need to find the decoder's hidden state sequence corresponding to the ground truth target text.** For autoregressive models like UnitY and Translatotron 2, teacher forcing training naturally addresses this problem. However, for the DA-Transformer, which utilizes a Directed Acyclic Graph (DAG) to model multiple possible translations simultaneously, the decoder's output consists of representations for all nodes in the DAG. To find the hidden states corresponding to the ground truth text, we proposed the **\"expect-path training\"** method (Section 3.2). This method considers all possible paths in the DAG and calculates the posterior probabilities $P_\\theta(a_i|X,Y)$ through dynamic programming. Based on these posterior probabilities, we are able to compute the **expected hidden states** corresponding to each word in the ground truth text. This enables us to jointly train the DA-Transformer and the Text-to-Speech (TTS) model for the S2ST task.\n\n**As Reviewers nvMe and WJ3s mentioned, while the novelty of our paper builds upon prior works, effectively integrating them to address a new and challenging task is also considered novel. Note that our approach is not limited to a specific TTS model**. On the contrary, our focus is on efficiently integrating the S2TT DA-Transformer and the TTS model, enabling end-to-end training for the S2ST task. As a result, theoretically, our method can also combine the S2TT DA-Transformer with other TTS models that are more powerful than FastSpeech 2, allowing for the modeling of more diverse and complex target speech. We leave this for future research. We hope that our work can serve as a strong baseline for non-autoregressive speech-to-speech translation in the future.\n\n2. **Re: More detailed analysis or experimentation on the trade-off between translation quality and decoding speed in DASpeech.**\n\nThank you for your advice. Following your suggestion, we have included several additional experiments concerning decoding efficiency of DASpeech. These experiments provide further insights into the trade-off between translation quality and decoding speed of DASpeech. **You can refer to the \"Global Response\" for more details.**"}, {"review_id": "oWhh7nqQ7v", "reviewer": "Reviewer_nvMe", "summary": "This paper proposes a method for fast and high quality speech to speech translation.\nThe paper adopts the two-pass decoding framework.\nThe first pass decoder adapts prior work on non autoregressive decoding for machine translation.\nThe second pass decoder reuses fastspeech 2 except that the input are the hidden states obtained by the first pass decoder.\nThe first pass and second pass components are pretrained on the speech to text translation task and the synthesis task respectively.\nResults show competitive performance compared to the state of the art, with substantial inference speedups.\nThe model is also shown to be able to preserve voice when trained on a corpus with the same voice on the source and target sides.", "questions": "* 227: \"We implement the following baseline systems.\" Would you be able to explain this in more details. Does reimplement mean reusing existing results, using existing implementation and retraining models on different datasets, etc.\n* 235:  \"We remove the R-Drop training [28] for simplification.\" It would be good to justify this choice, especially if it leads to suboptimal results for the corresponding baseline (and also given that there is an existing implementation for that work).\n* 245: \"an autoregressive S2TT model.\" Would it be possible to add details on that S2TT model? Is is a transformer? A particular existing model (if so, add citation, etc.)?\n", "limitations": "The limitations and broader impacts section is very effective in addressing limitations of the work and potential risks.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* the approach is effective: competitive wrt SoTA with much faster inference speed\n* the model can preserve the voice, which is an important feature towards authentic communication\n* the comparison to previous methods is very thorough as many previous methods are reimplemented/compared to", "weaknesses": "There is no specific weakness other than most of the novelty comes from the prior work (DA approach, 2-pass decoding framework, fastspeech 2) and this work very effectively combines this prior work to solve a new task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate your positive evaluation of our work. Below, we have provided the responses to each of your valuable comments.\n\n1. **Re: 227: \"We implement the following baseline systems.\" Would you be able to explain this in more details. Does reimplement mean reusing existing results, using existing implementation and retraining models on different datasets, etc.**\n\nSorry for the misunderstanding. **\"Re-implementation\" refers to \"using existing implementation and retraining models on different datasets\".** Specifically, for S2UT, Translatotron, UnitY, and Translatotron 2, we used the implementations available in the **fairseq** repository without any modifications. For TranSpeech, we utilized the official implementation provided by the authors. Therefore, we can ensure the correctness of our baseline systems' implementations.\n\nRegarding model sizes and training hyperparameters of baseline systems, we strived to match them as closely as possible to the original papers while maintaining consistency among the systems to ensure fairness in the comparisons (See more details in Appendix A). Based on the results of our re-implementations, the performance in translation quality and decoding speed of each system aligns closely with the conclusions reported in the previous papers.\n\n2. **Re: 235: \"We remove the R-Drop training [28] for simplification.\" It would be good to justify this choice, especially if it leads to suboptimal results for the corresponding baseline (and also given that there is an existing implementation for that work).**\n\nR-Drop is a general regularization method for neural network, which can reduce the inconsistency in model predictions between training and inference by dropout, thereby improving the model's performance. UnitY's training incorporates the R-Drop technique, which has been proven to effectively enhance translation quality. However, this training trick has not been widely adopted by other systems, as seen in recent works like TranSpeech, which did not use it. Using R-Drop training exclusively for UnitY would lead to an unfair comparison with other systems. **We aim for the comparisons between different models to primarily depend on the modeling approach (e.g., 1-pass vs. 2-pass model structures, choosing units vs. mel-spectrograms as prediction targets), rather than being influenced by specific training tricks.** Additionally, R-Drop training involves duplicating the input, which increases the training cost. Therefore, in our experiments, we have removed the R-Drop training from UnitY. In the future, we plan to investigate the effects of R-Drop training on our DASpeech.\n\n3. **Re: 245: \"an autoregressive S2TT model.\" Would it be possible to add details on that S2TT model? Is it a transformer? A particular existing model (if so, add citation, etc.)?**\n\nThank you for pointing out this issue. The speech encoder of the autoregressive S2TT model is the same as that used in DASpeech (subsampler + Conformer encoder). The decoder is a conventional 4-layer transformer decoder. We will provide a more detailed description in the final version."}, {"review_id": "iHKvaypHfj", "reviewer": "Reviewer_RzGP", "summary": "The paper describes a non-autoregressive direct Speech-to-Speech Translation (S2ST) model called DASpeech that achieves both high-quality translations and fast decoding speeds. DASpeech uses a two-pass architecture where a linguistic decoder generates the target text, and an acoustic decoder generates the target speech based on the hidden states of the linguistic decoder. The linguistic decoder used is the decoder of DA-Transformer, and the acoustic decoder used is FastSpeech 2. DASpeech achieves much better performance than non-autoregressive baseline and 18.53x speedup compared to autoregressive baseline models and preserves the speaker's voice of the source speech during translation.", "questions": "n/a", "limitations": "See weakness", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper proposes a non-autoregressive direct S2ST model that achieves both high-quality translations and fast decoding speeds. The experiments show that DASpeech performs better than the baseline NAT model and is comparable to the state-of-the-art AT model in terms of translation quality.", "weaknesses": "However, there are some issues in the paper that need to be addressed.\n\nFirstly, the novelty of the paper is not significant. The paper is based on the DA-Transformer model heavily borrowed from the reference [11] (i.e., NAT in text-to-text translation), and there is no breakthrough in terms of technology. Additionally, it is not clear that the proposed model can solve some unique problems in speech-to-speech translation scenario.\n\nSecondly, some of the contributions mentioned in the paper are not well-supported by the experiments. For example, the paper claims that DASpeech can achieve comparable or better performance than SOTA, but the experimental results show that it performs worse. \n\nThirdly, the efficiency comparison is not in-depth enough. Although efficiency is one of the major contributions of the paper, there is a lack of detailed analysis and tables to describe the efficiency. Moreover, since the paper is a NAT model, it is not fair to compare its speed with the AT model directly. It would be more reasonable to compare the efficiency with the NAT baseline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful feedback! Please find below the responses to each comment.\n\n1. **Re: The novelty of the paper is not significant. It is not clear that the proposed model can solve some unique problems in speech-to-speech translation scenario.**\n\nThank you for pointing that out. In our opinion, applying the DA-Transformer to **speech-to-text translation** is straightforward as the prediction target remains the target text, and only the text encoder needs to be replaced with a speech encoder. **However, its application to speech-to-speech translation (S2ST) is non-trivial.** For the two-pass S2ST model, predicting the target text is just the first step; it also requires an acoustic decoder to further synthesize the target speech based on the output hidden states from the linguistic decoder. During the training process, **we need to find the decoder's hidden state sequence corresponding to the ground truth target text.** For auto-regressive models like UnitY and Translatotron 2, teacher forcing training naturally addresses this problem. However, for the DA-Transformer, which utilizes a Directed Acyclic Graph (DAG) to model multiple possible translations simultaneously, the decoder's output consists of representations for all nodes in the DAG. To find the hidden states corresponding to the ground truth text, we proposed the **\"expect-path training\"** method (Section 3.2). This method considers all possible paths in the DAG and calculates the posterior probabilities $P_\\theta(a_i|X,Y)$ through dynamic programming. Based on these posterior probabilities, we are able to compute the **expected hidden states** corresponding to each word in the ground truth text. This enables us to jointly train the DA-Transformer and the Text-to-Speech (TTS) model for the S2ST task.\n\n**As Reviewers nvMe and WJ3s mentioned, while the novelty of our paper builds upon prior works, effectively integrating them to address a new and challenging task is also considered novel. Note that our approach is not limited to a specific TTS model.** On the contrary, our focus is on efficiently integrating the S2TT DA-Transformer and the TTS model, enabling end-to-end training for the S2ST task. As a result, theoretically, our method can also combine the S2TT DA-Transformer with other TTS models that are more powerful than FastSpeech 2, allowing for the modeling of more diverse and complex target speech. We leave this for future research. We hope that our work can serve as a strong baseline for non-autoregressive speech-to-speech translation in the future.\n\n2. **Re: The paper claims that DASpeech can achieve comparable or better performance than SOTA, but the experimental results show that it performs worse.**\n\nWe are sorry for the lack of precision in our claim. In fact, we have indeed achieved comparable or even better results than the state-of-the-art (SOTA) on the CVSS Fr-En dataset, as shown in Table 1. However, DASpeech performs slightly worse than SOTA in the multilingual scenario. Nevertheless, our results also significantly outperform the baseline model S2UT. We will modify our statement in the final version to ensure strict consistency with the experimental results.\n\n3. **Re: Detailed analysis and tables to describe the efficiency**\n\nThank you for your advice. Following your suggestion, we have included several additional experiments concerning decoding efficiency of DASpeech. These experiments provide further insights into the trade-off between translation quality and decoding speed of DASpeech. **You can refer to the \"Global Response\" for more details.**\n\n4. **Re: It is not fair to compare its speed with the AT model directly. It would be more reasonable to compare the efficiency with the NAT baseline.**\n\nThank you for your suggestion. Firstly, we acknowledge that NAT models have an inherent advantage in decoding speed compared to AT models. However, the parallel decoding may lead to a decline in translation quality to some extent. Therefore, we believe that comparing NAT to AT models in both translation quality and decoding speed allows us to highlight the trade-offs between different models. In the field of text-to-text translation, previous studies have also compared NAT models with autoregressive Transformers to assess decoding speed. Thus, **we consider our comparison of DASpeech with the autoregressive model S2UT to be reasonable.**\n\nFurthermore, concerning the comparison with NAT models, DASpeech also exhibits significantly improved decoding efficiency compared to TranSpeech. For instance, taking DASpeech ($\\lambda=0.5$, lookahead decoding) as an example, on the CVSS-C Fr-En dataset, it achieves a decoding speed improvement of 1.49x compared to TranSpeech with 5-iterations decoding, while simultaneously improving translation quality by 8.33 ASR-BLEU. **Compared to TranSpeech with 15-iterations decoding, DASpeech achieves a decoding speed improvement of 5.53x and enhances translation quality by 5.66 ASR-BLEU.** We believe these comparative evaluations contribute to a comprehensive understanding of the strengths and limitations of different models, and they highlight the benefits of our proposed DASpeech. "}, {"review_id": "LZDFxlSoZA", "reviewer": "Reviewer_PNQE", "summary": "DASpeech is a non-autoregressive direct speech-to-speech translation (S2ST) model. The model adopts a two-pass architecture, where a linguistic decoder first generates the target text, and an acoustic decoder then generates the target speech based on the hidden states of the linguistic decoder. The linguistic decoder is based on the DA-Transformer model, which models translations with a directed acyclic graph (DAG). The acoustic decoder is based on the FastSpeech 2 model.\n\nExperiments on the CVSS benchmark demonstrate that DASpeech can achieve comparable or even better performance than the state-of-the-art S2ST model Translatotron 2, while preserving up to 18.53x speedup compared to the autoregressive baseline model. Compared to the previous non-autoregressive S2ST model, DASpeech does not rely on knowledge distillation and iterative decoding, achieving significant improvements in both translation quality and decoding speed. Furthermore, DASpeech shows the ability to preserve the speaker's voice of the source speech during translation.", "questions": "1. The proposed method only evaluate for Fr-En languages. Can you evaluate the proposed method with more languages?\n2. What is this model offer other then speed-up? The results is comparable to T2. Can the translation results improve?\n3. Can you add ablation analysis of each component? ", "limitations": ".", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A new direct Speech to Speech translation model that use Conformer Encoder, Non-autoregressive Linguistic Decoder, Directed\nAcyclic Graph and Non-autoregressive Acoustic Decoder.\n2. Efficient architecture for Speech to Speech translation - up to 18.53 speedup compared to the autoregressive baseline model.\n3. Get comparable results to state of the art models in terms of translation quality (BLEU)\n\nA new direct Speech to Speech translation model:\nThis model is a neural network that can translate speech from one language to another directly, without the need for an intermediate step of transcribing the speech to text. This makes it more efficient and faster than traditional speech translation systems, which require two separate models: one for speech recognition and one for machine translation. The model uses a Conformer encoder, which is a type of neural network that has been shown to be effective for speech recognition tasks. The encoder converts the input speech signal into a sequence of hidden representations. These representations are then decoded by a non-autoregressive linguistic decoder, which generates the translation sentence one word at a time. The model also uses a directed acyclic graph (DAG) and a non-autoregressive acoustic decoder. The DAG is a data structure that allows the model to predict multiple words at a time, which can further improve the efficiency of the translation process. The non-autoregressive acoustic decoder generates the output speech signal one frame at a time, which helps to preserve the naturalness of the audio.\n\nEfficient architecture for Speech to Speech translation:\nThe model's architecture is designed to be efficient, both in terms of memory and speed. The Conformer encoder is a relatively lightweight model, which helps to reduce the memory requirements of the system. The non-autoregressive linguistic decoder and acoustic decoder are also designed to be efficient, which helps to speed up the translation process. The model has been shown to be up to 18.53 times faster than the autoregressive baseline model. This makes it a promising candidate for real-time speech translation applications.\n\nComparable results to state of the art models in terms of translation quality (BLEU):\nThe model has also been shown to achieve comparable results to state-of-the-art models in terms of translation quality (BLEU). \n", "weaknesses": ".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for providing your valuable and constructive feedback! Below, you will find our responses to each comment.\n\n1. **Re: Can you evaluate the proposed method with more languages?**\n\nThank you for your suggestions. Based on your advice, we conducted additional experiments on the CVSS-C De-En dataset. **Please refer to the \"Global Response\" for details.** The experiments confirmed that DASpeech has also outperformed the state-of-the-art performance in the De-En direction, confirming the effectiveness of DASpeech.\n\n2. **Re: What is this model offer other then speed-up? The results is comparable to T2. Can the translation results improve?**\n\nFrom the current results, DASpeech offers not only a significant speed advantage over Translatotron 2 but also demonstrates superior modeling capabilities for complex target voices. In Table 1's experimental results, when the target voice contains only a single speaker (CVSS-C dataset), DASpeech's translation quality slightly falls behind Translatotron 2 (25.03 vs. 25.21). However, when the target voice includes multiple speakers (CVSS-T dataset), DASpeech exhibits a 0.87 ASR-BLEU improvement over Translatotron 2. This improvement is attributed to DASpeech's acoustic decoder's ability to better model the multi-modal characteristics of the speech. Audio samples provided in our paper also reveal that DASpeech produces better audio quality than Translatotron 2 on the CVSS-T dataset.\n\nFurthermore, **we believe DASpeech has even greater potential for future development.** On one hand, recent research [1] indicates that further fine-tuning the DA-Transformer with a fuzzy alignment objective can enhance translation quality, which can be beneficial for DASpeech as well. On the other hand, the training of DASpeech does not rely on a specific TTS structure, allowing us to use more powerful TTS models than FastSpeech 2, enabling us to model more diverse and complex voices. We intend to explore these directions in our future research.\n\n[1] Ma et al., 2023. Fuzzy Alignments in Directed Acyclic Graph for Non-Autoregressive Machine Translation. ICLR 2023.\n\n3. **Re: Can you add ablation analysis of each component?**\n\nThank you for your suggestions. We added the following two ablation experiments:\n\n  - Removing the glancing strategy during the finetuning stage.\n  - Removing TTS pretraining.\n\nThe results on the CVSS-C Fr-En test set are shown in the table below. All results are based on the Lookahead decoding strategy. We observed that these two factors have a minor impact on the results. However, they still contribute to a slight improvement.\n\n| Models                              | ASR-BLEU |\n| ----------------------------------- | -------- |\n| DASpeech ($\\lambda=0.5$, Lookahead) | 24.71    |\n| w/o glancing                        | 24.68    |\n| w/o TTS pretraining                 | 24.63    |\n\nIn addition to this, the training of DASpeech involves two crucial hyperparameters: **the weight of the TTS loss, denoted as $\\mu$, and the target-side input length coefficient for the linguistic decoder, denoted as $\\lambda$.** The impact of these two hyperparameters on the CVSS-C Fr-En dataset is illustrated in the table below. All results are based on the Lookahead decoding strategy.\n\n| $\\mu$                              | ASR-BLEU |\n| ----------------------------------- | -------- |\n| 1  | 23.80    |\n| 2  | 24.12    |\n| 5  | **24.71**    |\n| 10  |  collapse |\n\n| $\\lambda$                     | ASR-BLEU |\n| ----------------------------------- | -------- |\n| 0.25  | 24.08    |\n| 0.5  | **24.71**    |\n| 1.0  | 24.41    |\n| 1.5 |  24.37 |\n\nRegarding the expect-path training method we proposed, we have also conducted a comparison with another strategy, best-path training, as detailed in Section 4.3 of the paper. **Apart from this, if you would like to see results for other configurations, please let us know. We would be more than happy to continue supplementing experiments during the reviewer-author discussion phase.**"}, {"review_id": "1a51auO6kR", "reviewer": "Reviewer_WJ3s", "summary": "This paper presents a non-autoregressive direct speech-to-speech translation model based on Directed Acyclic Transformer. Experiments are conducted on CVSS-C and CVSS-T datasets, demonstrating translation performance comparable to the SOTA (Translatotron 2) but more than 16x speed up. It also demonstrates capacity of voice preservation during translation.", "questions": "1) Voice Preservation evaluation -- from the audio samples, it can been heard clearly as the unit-based approaches don't preserve voice, but DASpeech and Translatotron 1 & 2 can (as stated by the paper). However, looking at the numbers in Table 4, the similarities from Translatotron 1 & 2 are very close to unit-based approach (with DASpeech signicantly higher) -- any explaination? ", "limitations": "Yes.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1) The proposed approach is sound\n2) Although the proposed approach is largely a combination of existing works (Directed Acyclic Transformer for NMT and Translatotron 2 for S2ST), its effective adoption in S2ST is novel.\n3) The experimental results on translation quality, speed, and voice preservation are all promising.\n4) Audio samples accompanying the paper sound consistent with the quantitive evaluation conducted in the paper.", "weaknesses": "1) The main results (Table 1) is conducted on English-French translation, which is an easy case as the order of the two languages are very close. It would be helpful to include more results on languages with significant difference on ordering, such as English-German. Compared to autoregressive model, one potential disadvantage of non-autogressive model is it may not handle reordering well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are deeply grateful for your positive evaluation of our work. Your valuable comments have been carefully considered, and we have provided responses to each of them below.\n\n1. **Re: It would be helpful to include more results on languages with significant differences in ordering, such as English-German.**\n\nThank you for your suggestions. Based on your advice, we conducted additional experiments on the CVSS-C De-En dataset. **Please refer to the \"Global Response\" for details.** The experiments confirmed that DASpeech has also outperformed the state-of-the-art performance in the De-En direction, confirming the effectiveness of DASpeech.\n\n2. **Re: Explaination about the voice preservation evaluation**\n\nThank you for bringing up this issue. We have also observed that Translatotron 1/2 exhibit only a slight improvement in speaker similarity compared to unit-based models. From the audio samples, it is evident that while Translatotron1/2 possess some voice cloning capability, the generated audio quality is not very high and often contains significant background noise. Since we extracted speaker embeddings using a speaker verification model trained on real audio, **we suspect that the presence of noise in the generated audio may negatively impact the calculation of speaker similarity due to distribution shift between training and decoding of the speaker verification model.** In contrast, DASpeech generates audio that is more natural, fluent, and with reduced noise. Additionally, it exhibits stronger voice cloning capabilities. As a result, it significantly outperforms other models in terms of speaker similarity. "}]

Author rebuttal:
We sincerely appreciate the valuable comments from all the reviewers, which have provided us with many insightful suggestions. **We are greatly encouraged by the fact that most reviewers have found our work to be sound and effective, with a novel application in the speech-to-speech translation (S2ST) task.** To address the concerns raised by the reviewers, we have added some new experiments and responded individually to the comments of each reviewer. We greatly look forward to your feedback. **Should you have any further questions, please feel free to continue sharing your comments.** We are enthusiastic about continuing to address your inquiries during the reviewer-author discussion period.

### **New Experiments Related to the Decoding Speed**

Specifically, following the suggestions of Reviewers 47bw and RzGP, **we have conducted several additional analysis experiments related to the decoding speed of DASpeech.** These experiments include:
1. **Trade-off between translation quality and decoding speed (Please refer to Figure 1 in the additional PDF provided in our "Global Response"):** By adjusting the hyperparameter $\lambda$ of DASpeech, the translation quality and decoding latency will change. Specifically, as $\lambda$ increases, the decoding latency of the model will increase, and it achieves the best translation quality when $\lambda=0.5$. Compared to previous autoregressive models (e.g. UnitY and Translatotron 2), DASpeech achieves a significantly higher speedup while maintaining comparable translation quality. When compared to the previous non-autoregressive model TranSpeech (which can balance translation quality and decoding latency by adjusting the number of iterations), DASpeech shows significant improvements in both translation quality and decoding speed. In conclusion, compared to existing works, **DASpeech achieves the best trade-off between translation quality and decoding latency.**
2. **Translation latency of different models across different source speech lengths (Please refer to Figure 2 in the additional PDF provided in our "Global Response"):** We have conducted a study on the translation latency of different models for speech inputs of different lengths. The results indicate that the translation latency of autoregressive models (S2UT, Translatotron, UnitY, Translatotron 2) significantly increases with the length of the source speech. On the other hand, the translation latency of non-autoregressive models (TranSpeech, DASpeech) are hardly affected by the source speech length. **When translating longer speech inputs, DASpeech's decoding speed can reach more than 20 times that of S2UT.**
3. **Speedup ratio of DASpeech under batch decoding (Please refer to Figure 5 in Appendix D provided in our original submitted supplementary materials):** Previous work pointed out that the speed benefits of non-autoregressive models may degrade during batch decoding. To better understand this problem, we evaluate the speedup ratio under different decoding batch sizes. **We find that DASpeech ($\lambda=0.5$ with Joint-Viterbi decoding) still achieves more than 6x speedup with a decoding batch size of 64.** Please refer to Appendix D for further details and information on this analysis.

### **New Experiments on the CVSS-C De-En Dataset**

In addition, based on the suggestions from Reviewer PNQE and WJ3s, **we conducted additional experiments on the CVSS-C De-En dataset.** This language direction poses greater difficulty due to significant word order differences between English and German, compounded by a relatively small training set of only 127K data. The ASR-BLEU scores on the CVSS-C De-En test set are presented in the table below. We observed that, possibly due to the aforementioned challenges, single-pass models (S2UT, Translatotron, TranSpeech) performed poorly in this language direction, whereas 2-pass models showed obviously better performance. Notably, **our DASpeech outperformed the state-of-the-art autoregressive models in the De-En direction**, confirming the effectiveness of DASpeech.



| Models                                    | ASR-BLEU (CVSS-C De-En) |
| ---------------------------------------- | ----- |
| S2UT                                     | 2.99  |
| Translatotron                            | 1.97  |
| UnitY                                    | 15.33 |
| Translatotron 2                          | 15.80 |
| TranSpeech (iter=5)                      | 1.90  |
| TranSpeech (iter=15, b=15, NPD)          | 2.28  |
| DASpeech ($\lambda=0.5$, Lookahead)      | 15.97 |
| DASpeech ($\lambda=0.5$, Joint-Viterbi) | **16.14** |




Meta-review: Most of the reviewers felt that the results looked decent. A couple of questions were raised about testing on more languages by PNQE and WJ3s, which the authors offerred during the rebuttal and the new results seemed to enforce the quality of the results (at least according to reviewer WJ3s). The authors were asked to validate their claim about SOTA results on some of the data by RzGP, which I think the authors clarified, appropriately.

47bw, RzGP raised their opinion that the method was not novel and included stacking two prior works. The authors rebuttal, emphasized the novelty being in -- the "expect-path training" method (Section 3.2). To me that seems like a fair argument, and actually quite novel -- the DA Transformer uses alignments for translation, but this method goes one step further and uses the uncertainty in the transition matrix for translation and manages to connect it to the speech generation part quite effectively.
Decision: Accept (poster)