Title: DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech Translation
Venue: NeurIPS 2023
Accepted: yes
Keywords: speech-to-speech translation

non-autoregressive translation

speech translation

directed acyclic transformer
Abstract: Direct speech-to-speech translation (S2ST) translates speech from one language into another using a single model. However, due to the presence of linguistic and acoustic diversity, the target speech follows a complex multimodal distribution, posing challenges to achieving both high-quality translations and fast decoding speeds for S2ST models. In this paper, we propose DASpeech, a non-autoregressive direct S2ST model which realizes both fast and high-quality S2ST. To better capture the complex distribution of the target speech, DASpeech adopts the two-pass architecture to decompose the generation process into two steps, where a linguistic decoder first generates the target text, and an acoustic decoder then generates the target speech based on the hidden states of the linguistic decoder. Specifically, we use the decoder of DA-Transformer as the linguistic decoder, and use FastSpeech 2 as the acoustic decoder. DA-Transformer models translations with a directed acyclic graph (DAG). To consider all potential paths in the DAG during training, we calculate the expected hidden states for each target token via dynamic programming, and feed them into the acoustic decoder to predict the target mel-spectrogram. During inference, we select the most probable path and take hidden states on that path as input to the acoustic decoder. Experiments on the CVSS Fr$\rightarrow$En benchmark demonstrate that DASpeech can achieve comparable or even better performance than the state-of-the-art S2ST model Translatotron 2, while preserving up to 18.53$\times$ speedup compared to the autoregressive baseline. Compared with the previous non-autoregressive S2ST model, DASpeech does not rely on knowledge distillation and iterative decoding, achieving significant improvements in both translation quality and decoding speed. Furthermore, DASpeech shows the ability to preserve the speaker's voice of the source speech during translation.

DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech Translation

Abstract

Direct speech-to-speech translation (S2ST) translates speech from one language into another using a single model. However, due to the presence of linguistic and acoustic diversity, the target speech follows a complex multimodal distribution, posing challenges to achieving both high-quality translations and fast decoding speeds for S2ST models. In this paper, we propose DASpeech, a non-autoregressive direct S2ST model which realizes both fast and high-quality S2ST. To better capture the complex distribution of the target speech, DASpeech adopts the two-pass architecture to decompose the generation process into two steps, where a linguistic decoder first generates the target text, and an acoustic decoder then generates the target speech based on the hidden states of the linguistic decoder. Specifically, we use the decoder of DA-Transformer as the linguistic decoder, and use FastSpeech 2 as the acoustic decoder. DA-Transformer models translations with a directed acyclic graph (DAG). To consider all potential paths in the DAG during training, we calculate the expected hidden states for each target token via dynamic programming, and feed them into the acoustic decoder to predict the target mel-spectrogram. During inference, we select the most probable path and take hidden states on that path as input to the acoustic decoder. Experiments on the CVSS Fr → En benchmark demonstrate that DASpeech can achieve comparable or even better performance than the state-of-the-art S2ST model Translatotron 2, while preserving up to 18.53 × speedup compared to the autoregressive baseline. Compared with the previous non-autoregressive S2ST model, DASpeech does not rely on knowledge distillation and iterative decoding, achieving significant improvements in both translation quality and decoding speed. Furthermore, DASpeech shows the ability to preserve the speaker's voice of the source speech during translation. 23

1 Introduction

Direct speech-to-speech translation (S2ST) directly translates speech of the source language into the target language, which can break the communication barriers between different language groups and has broad application prospects. Traditional S2ST usually consists of cascaded automatic speech recognition (ASR), machine translation (MT), and text-to-speech (TTS) models [1, 2]. In contrast, direct S2ST achieves source-to-target speech conversion with a unified model [3], which can (1) avoid error propagation across sub-models [4]; (2) reduce the decoding latency [5]; and (3) preserve

non-linguistic information (e.g., the speaker's voice) during translation [6]. Recent works show that direct S2ST can achieve comparable or even better performance than cascaded systems [7, 8].

Despite the theoretical advantages of direct S2ST, it is still very challenging to train a direct S2ST model in practice. Due to the linguistic diversity during translation, as well as the diverse acoustic variations (e.g., duration, pitch, energy, etc.), the target speech follows a complex multimodal distribution. To address this issue, Jia et al. [6], Inaguma et al. [7] propose the two-pass architecture, which first generates the target text with a linguistic decoder, and then uses an acoustic decoder to generate the target speech based on the hidden states of the linguistic decoder. The two-pass architecture decomposes the generation process into two steps: content translation and speech synthesis, making it easier to model the complex distribution of the target speech and achieving state-of-the-art performance among direct S2ST models.

Although the two-pass architecture achieves better translation quality, two passes of autoregressive decoding also incur high decoding latency. To reduce the decoding latency, Huang et al. [9] recently proposes non-autoregressive (NAR) S2ST that generates target speech in parallel. However, due to the conditional independence assumption of NAR models, it becomes more difficult to capture the multimodal distribution of the target speech compared with autoregressive models 4 . Therefore, the trade-off between translation quality and decoding speed of S2ST remains a pressing issue.

In this paper, we introduce an S2ST model with both high-quality translations and fast decoding speeds: DASpeech, a non-autoregressive two-pass direct S2ST model. Like previous two-pass models, DASpeech includes a speech encoder, a linguistic decoder, and an acoustic decoder. Specifically, the linguistic decoder uses the structure of DA-Transformer [11] decoder, which models translations via a directed acyclic graph (DAG). The acoustic decoder adopts the design of FastSpeech 2 [12], which takes the hidden states of the linguistic decoder as input and generates the target mel-spectrogram. During training, we consider all possible paths in the DAG by calculating the expected hidden state for each target token via dynamic programming, which are fed to the acoustic decoder to predict the target mel-spectrogram. During inference, we first find the most probable path in DAG and take hidden states on that path as input to the acoustic decoder. Due to the task decomposition of two-pass architecture, as well as the ability of DA-Transformer and FastSpeech 2 themselves to model linguistic diversity and acoustic diversity, DASpeech is able to capture the multimodal distribution of the target speech. Experiments on the CVSS Fr → En benchmark show that: (1) DASpeech achieves comparable or even better performance than the state-of-the-art S2ST model Translatotron 2, while maintaining up to 18.53 × speedup compared to the autoregressive model. (2) Compared with the previous NAR S2ST model TranSpeech [9], DASpeech no longer relies on knowledge distillation and iterative decoding, achieving significant advantages in both translation quality and decoding speed. (3) When training on speech-to-speech translation pairs of the same speaker, DASpeech emerges with the ability to preserve the source speaker's voice during translation.

2 Background

2.1 Directed Acyclic Transformer

Directed Acyclic Transformer (DA-Transformer) [11, 13] is proposed for non-autoregressive machine translation (NAT), which achieves comparable results to autoregressive Transformer [14] without relying on knowledge distillation. DA-Transformer consists of a Transformer encoder and an NAT decoder. The hidden states of the last decoder layer are organized as a directed acyclic graph (DAG). The hidden states correspond to vertices of the DAG, and there are unidirectional edges that connect vertices with small indices to those with large indices. DA-Transformer successfully alleviates the linguistic multi-modality problem since DAG can capture multiple translations simultaneously by assigning different translations to different paths in DAG.

Formally, given a source sequence X = ( x 1 , ..., x N ) and a target sequence Y = ( y 1 , ..., y M ) , the encoder takes X as input and the decoder takes learnable positional embeddings G = ( g 1 , ..., g L ) as input. Here L is the graph size, which is set to λ times the source length, i.e., L = λ · N , and λ is a hyperparameter. DA-Transformer models the translation probability P θ ( Y | X ) by marginalizing all

possible paths in DAG:

where A = ( a 1 , ..., a M ) is a path represented by a sequence of vertex indexes with 1 = a 1 < · · · < a M = L , and Γ contains all paths with the same length as the target sequence Y . The probability of path A is defined as:

where E ∈ R L × L is the transition probability matrix . We apply lower triangular masking on E to allow only forward transitions. With the selected path A , all target tokens are predicted in parallel:

where P ∈ R L ×| V | is the prediction probability matrix , and V indicates the vocabulary. Finally, we train the DA-Transformer by minimizing the negative log-likelihood loss:

which can be calculated with dynamic programming.

2.2 FastSpeech 2

FastSpeech 2 [12] is a non-autoregressive text-to-speech (TTS) model that generates mel-spectrograms from input phoneme sequences in parallel. It is composed of three stacked modules: encoder, variance adaptor, and mel-spectrogram decoder. The encoder and mel-spectrogram decoder consist of several feed-forward Transformer blocks, each containing a self-attention layer followed by a 1D-convolutional layer. The variance adaptor contains three variance predictors including duration predictor, pitch predictor, and energy predictor, which are used to reduce the information gap between input phoneme sequences and output mel-spectrograms. During training, the ground truth duration, pitch and energy are used to train these variance predictors and also as conditional inputs to generate the mel-spectrogram. During inference, we use the predicted values of these variance predictors. The introduction of variation information greatly alleviates the acoustic multi-modality problem, which leads to better voice quality. The training objective of FastSpeech 2 consists of four terms:

where L L1 measures the L1 distance between the predicted and ground truth mel-spectrograms, L dur , L pitch and L energy compute the mean square error (MSE) loss between predictions and ground truth for duration, pitch, and energy, respectively.

3 DASpeech

In this section, we introduce DASpeech, a non-autoregressive two-pass direct S2ST model that generates target phonemes and target mel-spectrograms successively. Formally, the source speech sequence is denoted as X = ( x 1 , ..., x N ) , where N is the number of frames in the source speech. The sequences of target phoneme and target mel-spectrogram are represented by Y = ( y 1 , ..., y M ) and S = ( s 1 , ..., s T ) , respectively. DASpeech first generates Y from X with a speech-to-text translation (S2TT) 5 DA-Transformer. Subsequently, it generates S with a FastSpeech 2-style decoder conditioned on the last-layer hidden states of the DA-Transformer. We first overview the model architecture of DASpeech in Section 3.1. In Section 3.2, we introduce our proposed training techniques that leverage pretrained S2TT DA-Transformer and FastSpeech 2 models and finetune the entire model for S2ST end-to-end. Finally, we present several decoding algorithms for DASpeech in Section 3.3.

<latexi sh

1_b

64="vY

R

5

K7+qTPNu8

Ef

DWC

AI

>

B

n

c

V

S

/

9L

0

G

g

p

wm

Q

H

y

2

M

X

d

O

F

j

o

z

k

Z

<latexi sh

1_b

64="vY

R

5

K7+qTPNu8

Ef

DWC

AI

>

B

n

c

V

S

/

9L

0

G

g

p

wm

Q

H

y

2

M

X

d

O

F

j

o

z

k

Z

<latexi sh

1_b

64="vY

R

5

K7+qTPNu8

Ef

DWC

AI

>

B

n

c

V

S

/

9L

0

G

g

p

wm

Q

H

y

2

M

X

d

O

F

j

o

z

k

Z

<latexi sh

1_b

64="vY

R

5

K7+qTPNu8

Ef

DWC

AI

>

B

n

c

V

S

/

9L

0

G

g

p

wm

Q

H

y

2

M

X

d

O

F

j

o

z

k

Z

<latexi sh

1_b

64="

w

UOVPBQM

W/

Yk

9p

N

>A

n

S8

EJ3

r

q

L

0

F

7

o

y2m

Z

+

D

I

G

u

j

HC

z

f

v

g

5

d

T

R

X

<latexi sh

1_b

64="

w

UOVPBQM

W/

Yk

9p

N

>A

n

S8

EJ3

r

q

L

0

F

7

o

y2m

Z

+

D

I

G

u

j

HC

z

f

v

g

5

d

T

R

X

<latexi sh

1_b

64="

w

UOVPBQM

W/

Yk

9p

N

>A

n

S8

EJ3

r

q

L

0

F

7

o

y2m

Z

+

D

I

G

u

j

HC

z

f

v

g

5

d

T

R

X

<latexi sh

1_b

64="

w

UOVPBQM

W/

Yk

9p

N

>A

n

S8

EJ3

r

q

L

0

F

7

o

y2m

Z

+

D

I

G

u

j

HC

z

f

v

g

5

d

T

R

X

<latexi sh

1_b

64="y

0qm7W

E59MB

XFPVowD

L

ZU

>A

n

c

NS8

J3

O

Kk

R

j

rG

/Y

2

d

u

+

z

v

T

H

Ip

Q

C

f

g

<latexi sh

1_b

64="y

0qm7W

E59MB

XFPVowD

L

ZU

>A

n

c

NS8

J3

O

Kk

R

j

rG

/Y

2

d

u

+

z

v

T

H

Ip

Q

C

f

g

<latexi sh

1_b

64="y

0qm7W

E59MB

XFPVowD

L

ZU

>A

n

c

NS8

J3

O

Kk

R

j

rG

/Y

2

d

u

+

z

v

T

H

Ip

Q

C

f

g

<latexi sh

1_b

64="y

0qm7W

E59MB

XFPVowD

L

ZU

>A

n

c

NS8

J3

O

Kk

R

j

rG

/Y

2

d

u

+

z

v

T

H

Ip

Q

C

f

g

<latexi sh

1_b

64="UQ8R

P

2N

0

WZInL

J

c

w

f

>A

B

V

S

E

3

r

q/

9

mKoM

F

7

y

p

+

O

D

G

u

j

Y

HC

z

v

g

X

k

T

d

5

<latexi sh

1_b

64="UQ8R

P

2N

0

WZInL

J

c

w

f

>A

B

V

S

E

3

r

q/

9

mKoM

F

7

y

p

+

O

D

G

u

j

Y

HC

z

v

g

X

k

T

d

5

<latexi sh

1_b

64="UQ8R

P

2N

0

WZInL

J

c

w

f

>A

B

V

S

E

3

r

q/

9

mKoM

F

7

y

p

+

O

D

G

u

j

Y

HC

z

v

g

X

k

T

d

5

<latexi sh

1_b

64="UQ8R

P

2N

0

WZInL

J

c

w

f

>A

B

V

S

E

3

r

q/

9

mKoM

F

7

y

p

+

O

D

G

u

j

Y

HC

z

v

g

X

k

T

d

5

<latexi sh

1_b

64="zE

c

Q

w

N

WdZDmK

G

>A

B

n

V

S8

3Ur q/

9L

0

M

F

7

o

2k

p

I

P

j

X

Y

u

+O

gfH

T

v

5

R

<latexi sh

1_b

64="zE

c

Q

w

N

WdZDmK

G

>A

B

n

V

S8

3Ur q/

9L

0

M

F

7

o

2k

p

I

P

j

X

Y

u

+O

gfH

T

v

5

R

<latexi sh

1_b

64="zE

c

Q

w

N

WdZDmK

G

>A

B

n

V

S8

3Ur q/

9L

0

M

F

7

o

2k

p

I

P

j

X

Y

u

+O

gfH

T

v

5

R

<latexi sh

1_b

64="zE

c

Q

w

N

WdZDmK

G

>A

B

n

V

S8

3Ur q/

9L

0

M

F

7

o

2k

p

I

P

j

X

Y

u

+O

gfH

T

v

5

R

<latexi sh

1_b

64="

2

MgRSJq

U/O

oBGZ3

w

u

>A

n

c

V

N

8

E

r

9L

0

P

K

D

T

p

m7

QvH

y5

+

f

dW

j

YI

C

k

z

F

X

<latexi sh

1_b

64="

2

MgRSJq

U/O

oBGZ3

w

u

>A

n

c

V

N

8

E

r

9L

0

P

K

D

T

p

m7

QvH

y5

+

f

dW

j

YI

C

k

z

F

X

<latexi sh

1_b

64="

2

MgRSJq

U/O

oBGZ3

w

u

>A

n

c

V

N

8

E

r

9L

0

P

K

D

T

p

m7

QvH

y5

+

f

dW

j

YI

C

k

z

F

X

<latexi sh

1_b

64="

2

MgRSJq

U/O

oBGZ3

w

u

>A

n

c

V

N

8

E

r

9L

0

P

K

D

T

p

m7

QvH

y5

+

f

dW

j

YI

C

k

z

F

X

<latexi sh

1_b

64="fHYLD/

d

nuRA

z

E

ywI

>

B

c

V

NS8

q

9

0

GP

K

g

m7

Q

Cj

k

X

W

M

Z

+O

2

o

T

v

F

5

p

<latexi sh

1_b

64="fHYLD/

d

nuRA

z

E

ywI

>

B

c

V

NS8

q

9

0

GP

K

g

m7

Q

Cj

k

X

W

M

Z

+O

2

o

T

v

F

5

p

<latexi sh

1_b

64="fHYLD/

d

nuRA

z

E

ywI

>

B

c

V

NS8

q

9

0

GP

K

g

m7

Q

Cj

k

X

W

M

Z

+O

2

o

T

v

F

5

p

<latexi sh

1_b

64="fHYLD/

d

nuRA

z

E

ywI

>

B

c

V

NS8

q

9

0

GP

K

g

m7

Q

Cj

k

X

W

M

Z

+O

2

o

T

v

F

5

p

<latexi sh

1_b

64="HVY

/

KzOMy wgdQk

F

>A

B

n

c

NS

E

3Ur

9L

0m

o

7

W

2

pZ

I

PCj

X

u

f

G

D

T

v

5

R

<latexi sh

1_b

64="HVY

/

KzOMy wgdQk

F

>A

B

n

c

NS

E

3Ur

9L

0m

o

7

W

2

pZ

I

PCj

X

u

f

G

D

T

v

5

R

<latexi sh

1_b

64="HVY

/

KzOMy wgdQk

F

>A

B

n

c

NS

E

3Ur

9L

0m

o

7

W

2

pZ

I

PCj

X

u

f

G

D

T

v

5

R

<latexi sh

1_b

64="HVY

/

KzOMy wgdQk

F

>A

B

n

c

NS

E

3Ur

9L

0m

o

7

W

2

pZ

I

PCj

X

u

f

G

D

T

v

5

R

<latexi sh

1_b

64="5

y3o

u

GPU

CO

q

+

V

F

M

>A

B

n

c

NS8

EJ

r

/

9L

0

R

K

g

D

p

wm7

QvH

T

2

X

W

d

Y

j

z

I

f

Z

k

<latexi sh

1_b

64="5

y3o

u

GPU

CO

q

+

V

F

M

>A

B

n

c

NS8

EJ

r

/

9L

0

R

K

g

D

p

wm7

QvH

T

2

X

W

d

Y

j

z

I

f

Z

k

<latexi sh

1_b

64="5

y3o

u

GPU

CO

q

+

V

F

M

>A

B

n

c

NS8

EJ

r

/

9L

0

R

K

g

D

p

wm7

QvH

T

2

X

W

d

Y

j

z

I

f

Z

k

<latexi sh

1_b

64="5

y3o

u

GPU

CO

q

+

V

F

M

>A

B

n

c

NS8

EJ

r

/

9L

0

R

K

g

D

p

wm7

QvH

T

2

X

W

d

Y

j

z

I

f

Z

k

<latexi sh

1_b

64="+c

p

OK

z

2

F

doEjLIVSY

>A

B

n

N

8

9

0mk

M

7Q

Wy

Z

D

G

u

w

HC

P

f

v

g

X

R

T

5

<latexi sh

1_b

64="+c

p

OK

z

2

F

doEjLIVSY

>A

B

n

N

8

9

0mk

M

7Q

Wy

Z

D

G

u

w

HC

P

f

v

g

X

R

T

5

<latexi sh

1_b

64="+c

p

OK

z

2

F

doEjLIVSY

>A

B

n

N

8

9

0mk

M

7Q

Wy

Z

D

G

u

w

HC

P

f

v

g

X

R

T

5

<latexi sh

1_b

64="+c

p

OK

z

2

F

doEjLIVSY

>A

B

n

N

8

9

0mk

M

7Q

Wy

Z

D

G

u

w

HC

P

f

v

g

X

R

T

5

<latexi sh

1_b

64="kc/f8yXw+

vQ

IR5

2ZO7JgLF0

>A

B

n

V

NS

E

3Ur

q

9

GP

K

D

T

m

H

u

M

<latexi sh

1_b

64="kc/f8yXw+

vQ

IR5

2ZO7JgLF0

>A

B

n

V

NS

E

3Ur

q

9

GP

K

D

T

m

H

u

M

<latexi sh

1_b

64="kc/f8yXw+

vQ

IR5

2ZO7JgLF0

>A

B

n

V

NS

E

3Ur

q

9

GP

K

D

T

m

H

u

M

<latexi sh

1_b

64="kc/f8yXw+

vQ

IR5

2ZO7JgLF0

>A

B

n

V

NS

E

3Ur

q

9

GP

K

D

T

m

H

u

M

dW

o

j

C

Y

p

z

dW

o

j

C

Y

p

z

dW

o

j

C

Y

p

z

dW

o

j

C

Y

p

z

<latexi sh

1_b

64="Z0zSEqu cM

oNHWCQ

2

Ap

8

>

B

n

V

/

9L

PR

K

D

T

wm7

G

g

v

y5

+

f

d

j

YI

O

F

k

X

<latexi sh

1_b

64="Z0zSEqu cM

oNHWCQ

2

Ap

8

>

B

n

V

/

9L

PR

K

D

T

wm7

G

g

v

y5

+

f

d

j

YI

O

F

k

X

<latexi sh

1_b

64="Z0zSEqu cM

oNHWCQ

2

Ap

8

>

B

n

V

/

9L

PR

K

D

T

wm7

G

g

v

y5

+

f

d

j

YI

O

F

k

X

<latexi sh

1_b

64="Z0zSEqu cM

oNHWCQ

2

Ap

8

>

B

n

V

/

9L

PR

K

D

T

wm7

G

g

v

y5

+

f

d

j

YI

O

F

k

X

<latexi sh

1_b

64="WOdA5

y9

8F

q

CU

3

ZjV

>

B

n

c

NS

EJ

/

0

PR

K

g

D

wm7

I

Q

kzX

M

Y

u

+

2

fH

G

o

T

v

p

<latexi sh

1_b

64="WOdA5

y9

8F

q

CU

3

ZjV

>

B

n

c

NS

EJ

/

0

PR

K

g

D

wm7

I

Q

kzX

M

Y

u

+

2

fH

G

o

T

v

p

<latexi sh

1_b

64="WOdA5

y9

8F

q

CU

3

ZjV

>

B

n

c

NS

EJ

/

0

PR

K

g

D

wm7

I

Q

kzX

M

Y

u

+

2

fH

G

o

T

v

p

<latexi sh

1_b

64="WOdA5

y9

8F

q

CU

3

ZjV

>

B

n

c

NS

EJ

/

0

PR

K

g

D

wm7

I

Q

kzX

M

Y

u

+

2

fH

G

o

T

v

p

<latexi sh

1_b

64="

dWpwvMqyTQ

A

fLZ

j

U

>

B

n

c

V

NS8

EJ3

r

/

9

0mko

F

7

2

I

PC

zX

Y

u

+O

g

H

G

D

5

K

R

<latexi sh

1_b

64="

dWpwvMqyTQ

A

fLZ

j

U

>

B

n

c

V

NS8

EJ3

r

/

9

0mko

F

7

2

I

PC

zX

Y

u

+O

g

H

G

D

5

K

R

<latexi sh

1_b

64="

dWpwvMqyTQ

A

fLZ

j

U

>

B

n

c

V

NS8

EJ3

r

/

9

0mko

F

7

2

I

PC

zX

Y

u

+O

g

H

G

D

5

K

R

<latexi sh

1_b

64="

dWpwvMqyTQ

A

fLZ

j

U

>

B

n

c

V

NS8

EJ3

r

/

9

0mko

F

7

2

I

PC

zX

Y

u

+O

g

H

G

D

5

K

R

<latexi sh

1_b

64="pf

2wRT

D

G

nU

8

J

o

>

B

c

V

NS

E

3

q/

9L

P

K

7

gQvH

y5

+

u

M

dW

Z

j

YI

O

F

k

z

X

<latexi sh

1_b

64="pf

2wRT

D

G

nU

8

J

o

>

B

c

V

NS

E

3

q/

9L

P

K

7

gQvH

y5

+

u

M

dW

Z

j

YI

O

F

k

z

X

<latexi sh

1_b

64="pf

2wRT

D

G

nU

8

J

o

>

B

c

V

NS

E

3

q/

9L

P

K

7

gQvH

y5

+

u

M

dW

Z

j

YI

O

F

k

z

X

<latexi sh

1_b

64="pf

2wRT

D

G

nU

8

J

o

>

B

c

V

NS

E

3

q/

9L

P

K

7

gQvH

y5

+

u

M

dW

Z

j

YI

O

F

k

z

X

<latexi sh

1_b

64="Mc0nVz

97Zj8

oXCp

PGq5Jr

>A

B

NS

E

3U

/

L

R

K

g

D

wm

I

Q

k

W

Y

u

+O

2

d

fH

T

v

y

F

<latexi sh

1_b

64="Mc0nVz

97Zj8

oXCp

PGq5Jr

>A

B

NS

E

3U

/

L

R

K

g

D

wm

I

Q

k

W

Y

u

+O

2

d

fH

T

v

y

F

<latexi sh

1_b

64="Mc0nVz

97Zj8

oXCp

PGq5Jr

>A

B

NS

E

3U

/

L

R

K

g

D

wm

I

Q

k

W

Y

u

+O

2

d

fH

T

v

y

F

<latexi sh

1_b

64="Mc0nVz

97Zj8

oXCp

PGq5Jr

>A

B

NS

E

3U

/

L

R

K

g

D

wm

I

Q

k

W

Y

u

+O

2

d

fH

T

v

y

F

<latexi sh

1_b

64="vHdyW8AOz uR

9c

+B

w5KY

>

n

V

NS

EJ3Ur q/

L

0

P

g

D

T

m7

G

Q

2

M

f

F

o

Z

k

j

C

I

p

X

<latexi sh

1_b

64="vHdyW8AOz uR

9c

+B

w5KY

>

n

V

NS

EJ3Ur q/

L

0

P

g

D

T

m7

G

Q

2

M

f

F

o

Z

k

j

C

I

p

X

<latexi sh

1_b

64="vHdyW8AOz uR

9c

+B

w5KY

>

n

V

NS

EJ3Ur q/

L

0

P

g

D

T

m7

G

Q

2

M

f

F

o

Z

k

j

C

I

p

X

<latexi sh

1_b

64="vHdyW8AOz uR

9c

+B

w5KY

>

n

V

NS

EJ3Ur q/

L

0

P

g

D

T

m7

G

Q

2

M

f

F

o

Z

k

j

C

I

p

X

<latexi sh

1_b

64="ocyY

rP

wXQ

H

UO

C

>A

B

n

V

NS8

EJ3

q/

9L

0

M

F

7

W

2m

TpZ

I

j

kz

G

u

D

v

K

f

d

+

5

R

g

<latexi sh

1_b

64="ocyY

rP

wXQ

H

UO

C

>A

B

n

V

NS8

EJ3

q/

9L

0

M

F

7

W

2m

TpZ

I

j

kz

G

u

D

v

K

f

d

+

5

R

g

<latexi sh

1_b

64="ocyY

rP

wXQ

H

UO

C

>A

B

n

V

NS8

EJ3

q/

9L

0

M

F

7

W

2m

TpZ

I

j

kz

G

u

D

v

K

f

d

+

5

R

g

<latexi sh

1_b

64="ocyY

rP

wXQ

H

UO

C

>A

B

n

V

NS8

EJ3

q/

9L

0

M

F

7

W

2m

TpZ

I

j

kz

G

u

D

v

K

f

d

+

5

R

g

<latexi sh

1_b

64="ojzQr m8

Xw

DL

Vn pZ

fk

>A

B

c

NS

EJ3U

q/

9

0

M

F

7

Wy2

+

O

I

G

u

Y

HC

P

v

K

g

R

T

d

5

<latexi sh

1_b

64="ojzQr m8

Xw

DL

Vn pZ

fk

>A

B

c

NS

EJ3U

q/

9

0

M

F

7

Wy2

+

O

I

G

u

Y

HC

P

v

K

g

R

T

d

5

<latexi sh

1_b

64="ojzQr m8

Xw

DL

Vn pZ

fk

>A

B

c

NS

EJ3U

q/

9

0

M

F

7

Wy2

+

O

I

G

u

Y

HC

P

v

K

g

R

T

d

5

<latexi sh

1_b

64="ojzQr m8

Xw

DL

Vn pZ

fk

>A

B

c

NS

EJ3U

q/

9

0

M

F

7

Wy2

+

O

I

G

u

Y

HC

P

v

K

g

R

T

d

5

<latexi sh

1_b

64="7qB/kyI

L

gm

d

0O

fRKCHv3

U

>A

n

c

V

NS8

EJ

r

9

M

F

Q

oW

2

TpZ

P

j

zX

w

G

u

D

+

5

Y

<latexi sh

1_b

64="7qB/kyI

L

gm

d

0O

fRKCHv3

U

>A

n

c

V

NS8

EJ

r

9

M

F

Q

oW

2

TpZ

P

j

zX

w

G

u

D

+

5

Y

<latexi sh

1_b

64="7qB/kyI

L

gm

d

0O

fRKCHv3

U

>A

n

c

V

NS8

EJ

r

9

M

F

Q

oW

2

TpZ

P

j

zX

w

G

u

D

+

5

Y

<latexi sh

1_b

64="7qB/kyI

L

gm

d

0O

fRKCHv3

U

>A

n

c

V

NS8

EJ

r

9

M

F

Q

oW

2

TpZ

P

j

zX

w

G

u

D

+

5

Y

<latexi sh

1_b

64="V5

B

Pn

+

Mw

YR

G

/

k8

>A

c

NS

EJ3Ur

q

9L

0

F

7Q

oWy2m pZ

O

D

I

u

j

HC

z

f

v

K

g

X

T

d

<latexi sh

1_b

64="V5

B

Pn

+

Mw

YR

G

/

k8

>A

c

NS

EJ3Ur

q

9L

0

F

7Q

oWy2m pZ

O

D

I

u

j

HC

z

f

v

K

g

X

T

d

<latexi sh

1_b

64="V5

B

Pn

+

Mw

YR

G

/

k8

>A

c

NS

EJ3Ur

q

9L

0

F

7Q

oWy2m pZ

O

D

I

u

j

HC

z

f

v

K

g

X

T

d

<latexi sh

1_b

64="V5

B

Pn

+

Mw

YR

G

/

k8

>A

c

NS

EJ3Ur

q

9L

0

F

7Q

oWy2m pZ

O

D

I

u

j

HC

z

f

v

K

g

X

T

d

Conformer

Encoder

Non-autoregressive Linguistic Decoder

(Self-Attn + Cross-Attn + FFN)

Attn

g

1

g

2

g

3

g

4

g

5

g

6

N

×

g

7

g

8

Positional

Embeddings

Directed

Expect-Path

Training

Inference

Linear

+

Softmax

Target

Phoneme

v

1

v

2

v

3

v

4

v

5

v

6

v

7

v

8

z

1

z

2

z

3

z

4

z

5

Non-autoregressive Acoustic Decoder

Speech

Encoder

3.1 Model Architecture

As shown in Figure 1, DASpeech consists of three parts: a speech encoder, a non-autoregressive linguistic decoder, and a non-autoregressive acoustic decoder. Below are the details of each part.

Speech Encoder Since our model takes speech features as input, we replace the Transformer encoder in the original DA-Transformer with a speech encoder. The speech encoder contains a subsampler followed by several Conformer blocks [15]. Specifically, the subsampler consists of two 1D-convolutional layers which shrink the length of input sequences by a factor of 4. Conformer combines multi-head attention modules and convolutional layers together to capture both global and local features. We use relative positional encoding [16] in the multi-head attention module.

Non-autoregressive Linguistic Decoder The linguistic decoder is identical to the decoder of DATransformer, which generates the target phoneme sequence from the source speech in parallel. Each decoder layer comprises a self-attention layer, a cross-attention layer, and a feed-forward layer. The decoder takes learnable positional embeddings as input, and the last-layer hidden states are organized as a DAG to model translations, as we described in Section 2.1.

Non-autoregressive Acoustic Decoder The acoustic decoder adopts the design of FastSpeech 2, which generates target mel-spectrograms from the last-layer hidden states of DA-Transformer in parallel. The model architecture is the same as that introduced in Section 2.2, except that the embedding matrix of the input phonemes is removed, since the input has changed from the phoneme sequence to the hidden state sequence.

3.2 Training

DASpeech has the advantage of easily utilizing pretrained S2TT DA-Transformer and FastSpeech 2 models. We use the pretrained S2TT DA-Transformer to initialize the speech encoder and the non-autoregressive linguistic decoder, and use the pretrained FastSpeech 2 model to initialize the non-autoregressive acoustic decoder. Finally, the entire model is finetuned for direct S2ST. This pretraining-finetuning pipeline simplifies model training and enables the use of additional S2TT and TTS data. However, end-to-end finetuning presents a major challenge due to the length discrepancy between the output from the linguistic decoder and the input to the acoustic decoder. Specifically, the hidden state sequence output by the linguistic decoder has a length of L = λ · N , while the input sequence required by the acoustic decoder should have a length of M , which is the length of the ground truth phoneme sequence. Therefore, it is necessary to determine how to obtain the input

sequence of acoustic decoder Z = ( z 1 , ..., z M ) from the last-layer hidden states of linguistic decoder V = ( v 1 , ..., v L ) . The following introduces our proposed approach: Expect-Path Training .

Expect-Path Training Intuitively, the i -th input element z i should be the hidden state of the vertex responsible for generating y i . However, since there may be multiple vertices capable of generating each y i due to numerous possible paths, we would like to consider all potential paths. To address this issue, we define z i as the expected hidden state under the posterior distribution P θ ( a i | X,Y ) :

where P θ ( a i = j | X,Y ) refers to the probability of vertex j being the i -th vertex on path A , which means that y i is generated by vertex j . We can compute P θ ( a i = j | X,Y ) as follows:

where ✶ ( a i = j ) is an indicator function to indicate whether the i -th vertex of path A is vertex j . To calculate ∑ A ∈ Γ ✶ ( a i = j ) · P θ ( Y, A | X ) and ∑ A ∈ Γ P θ ( Y, A | X ) in Equation (9), we employ the forward-backward algorithm [17], which involves two passes of dynamic programming.

Forward Algorithm The forward probability is defined as α i ( j ) = P θ ( y 1 , ..., y i , a i = j | X ) , which is the probability of generating the partial target sequence ( y 1 , ..., y i ) and ending in vertex j at the i -th step. By definition, we have α 1 (1) = P 1 ,y 1 and α 1 (1 < j ≤ L ) = 0 . Due to the Markov property, we can sequentially calculate α i ( · ) from its previous step α i -1 ( · ) as follows:

Backward Algorithm The backward probability is defined as β i ( j ) = P θ ( y i +1 , ..., y M | a i = j, X ) , which is the probability of starting from vertex j at the i -th step and generating the rest of the target sequence ( y i +1 , ..., y M ) . By definition, we have β M ( L ) = 1 and β M (1 ≤ j < L ) = 0 . Similar to the forward algorithm, we can sequentially calculate β i ( j ) from its next step β i +1 ( j ) as follows:

Recalling Equation (9), the denominator is the sum of the probabilities of all valid paths, which is equal to α M ( L ) . The numerator is the sum of the probabilities of all paths with a i = j , which is equal to α i ( j ) · β i ( j ) . Therefore, the Equation (6) can be calculated as:

The time complexity of the forward-backward algorithm is O ( ML 2 ) . Finally, the training objective of DASpeech is as follows:

where µ is the weight of TTS loss. The definitions of L DAT and L TTS are the same as those in Equations (4) and (5).

3.3 Inference

During inference, we perform two-pass parallel decoding. First, we find the most probable path A ∗ in DAG with one of the decoding strategies proposed for DA-Transformer (see details below). We then

feed the last-layer hidden states on path A ∗ to the non-autoregressive acoustic decoder to generate the mel-spectrogram. Finally, the predicted mel-spectrogram will be converted into waveform using a pretrained HiFi-GAN vocoder [18]. Since both DAG and TTS decoding are fully parallel, DASpeech achieves significant improvements in decoding efficiency compared to previous two-pass models which rely on two passes of autoregressive decoding. Considering the trade-off between translation quality and decoding efficiency, we use the following two decoding strategies for DA-Transformer in our experiments: Lookahead and Joint-Viterbi .

Lookahead Lookahead decoding sequentially chooses a i and y i in a greedy way. At each decoding step, it jointly considers the transition probability and the prediction probability:

Joint-Viterbi Joint-Viterbi decoding [19] finds the global joint optimal solution of the translation and decoding path via Viterbi decoding [20]:

After Viterbi decoding, we first decide the target length M and obtain the optimal path by backtracking from a ∗ M = L . More details can be found in the original paper.

4 Experiments

4.1 Experimental Setup

Dataset We conduct experiments on the CVSS dataset [4], a large-scale S2ST corpus containing speech-to-speech translation pairs from 21 languages to English. It is extended from the CoVoST 2 [21] S2TT corpus by synthesizing the target text into speech with state-of-the-art TTS models. It includes two versions: CVSS-C and CVSS-T. For CVSS-C, all target speeches are in a single speaker's voice. For CVSS-T, the target speeches are in voices transferred from the corresponding source speeches. We evaluate the models on the CVSS-C French → English (Fr → En) and CVSS-T French → English (Fr → En) datasets. We also conduct a multilingual experiment by combining all 21 language directions in CVSS-C together to train a single many-to-English S2ST model.

Pre-processing We convert the source speech to 16000Hz and generate target speech with 22050Hz. We compute the 80-dimensional mel-filterbank features for the source speech, and transform the target waveform into mel-spectrograms following Ren et al. [12]. We apply utterance-level and global-level cepstral mean-variance normalization for source speech and target speech, respectively. We follow Ren et al. [12] to extract the duration, pitch, and energy information of the target speech.

Model Configurations The speech encoder, linguistic decoder, and acoustic decoder contain 12 Conformer layers, 4 Transformer decoder layers, and 8 feed-forward Transformer blocks, respectively. The detailed configurations can be found in Table 5 in Appendix A. For model regularization, we set dropout to 0.1 and weight decay to 0.01, and no label smoothing is used. We use the HiFi-GAN vocoder pretrained on the VCTK dataset 6 [22] to convert the mel-spectrogram into waveform.

Training DASpeech follows the pretraining-finetuning pipeline. During pretraining, the speech encoder and the linguistic decoder are trained on the S2TT task for 100k updates with a batch of 320k audio frames. The learning rate warms up to 5e-4 within 10k steps. The acoustic decoder is pretrained on the TTS task for 100k updates with a batch size of 512. The learning rate warms up to 5e-4 within 4k steps. During finetuning, we train the entire model for 50k updates with a batch of 320k audio frames. The learning rate warms up to 1e-3 within 4k steps. We use Adam optimizer [23] for both pretraining and finetuning. For the weight of TTS loss µ , we experiment with µ ∈ { 1 . 0 , 2 . 0 , 5 . 0 , 10 . 0 } and choose µ = 5 . 0 according to results on the dev set. We implement our model with the open-source toolkit fairseq [24]. All models are trained on 4 RTX 3090 GPUs.

In the multilingual experiment, the presence of languages with limited data or substantial interlingual variations makes the mapping from source speech to target phonemes particularly challenging. To address this, we adopt a two-stage pretraining strategy. Initially, we pretrain the speech encoder and

the linguistic decoder using the speech-to-subword task, followed by pretraining on the speech-tophoneme task. In the second stage of pretraining, the embedding and output projection matrices of the decoder are replaced and trained from scratch to accommodate changes in the vocabulary. We employ this pretraining strategy for DASpeech, UnitY and Translatotron 2 in the multilingual experiment. We learn the subword vocabulary with a size of 6K using the SentencePiece toolkit.

We also adopt the glancing strategy [25] during training, which shows effectiveness in alleviating the multi-modality problem for NAT. It first assigns target tokens to appropriate vertices following the most probable path ˆ A = arg max A ∈ Γ P θ ( Y, A | X ) , and then masks some tokens. We linearly anneal the unmasking ratio τ from 0.5 to 0.1 during pretraining and fix τ to 0.1 during finetuning.

Evaluation During finetuning, we save checkpoints every 2000 steps and average the last 5 checkpoints for evaluation. We use the open-source ASR-BLEU toolkit 7 to evaluate the translation quality. The translated speech is first transcribed into text using a pretrained ASR model. SacreBLEU [26] is then used to compute the BLEU score [27] and the statistical significance of translation results. The decoding speedup is measured on the test set using 1 RTX 3090 GPU with a batch size of 1.

Baseline Systems We implement the following baseline systems for comparison. More details about the model architectures and hyperparameters can be found in Appendix A.

S2UT [5] Speech-to-unit translation (S2UT) model generates discrete units corresponding to the target speech with a sequence-to-sequence model. We introduce the auxiliary task of predicting target phonemes to help the model converge.

Translatotron [3] Translatotron generates the target mel-spectrogram with a sequence-to-sequence model. We also introduce the auxiliary task of predicting the target phonemes.

UnitY [7] UnitY is a two-pass model which generates target phonemes and discrete units successively 8 . We remove the R-Drop training [28] for simplification. We pretrain the speech encoder and first-pass decoder on the S2TT task.

Translatotron 2 [6] Translatotron 2 is a two-pass model which generates target phonemes and mel-spectrograms successively. We enhance Translatotron 2 by replacing LSTM with Transformer, and introducing an additional encoder between two decoders following Inaguma et al. [7]. The speech encoder and first-pass decoder are pretrained on the S2TT task.

TranSpeech [9] TranSpeech is the first non-autoregressive S2ST model that generates target discrete units in parallel. To alleviate the acoustic multi-modality problem, TranSpeech introduces bilateral perturbation (BiP) to disentangle the acoustic variations from the discrete units. We re-implement TranSpeech following the configurations in the original paper.

S2T + FastSpeech 2 The cascaded system that combines an autoregressive S2TT model and FastSpeech 2. The S2T model contains 12 Conformer layers and 4 Transformer decoder layers, which is also used in UnitY and Translatotron 2 pretraining.

DAT + FastSpeech 2 The cascaded system that combines the S2TT DA-Transformer model and FastSpeech 2. Both models are used in DASpeech pretraining.

4.2 Main Results

Table 1 summarizes the results on the CVSS-C Fr → En and CVSS-T Fr → En datasets. (1) Compared with previous autoregressive models, DASpeech ( D1-D4 ) obviously surpasses single-pass models ( A1 , A2 ) and achieves comparable or even better performance than two-pass models ( B1 , B2 ), while preserving up to 18.53 times decoding speedup compared to S2UT. (2) Compared with the previous NAR model TranSpeech ( C1-C4 ), DASpeech does not rely on knowledge distillation and iterative decoding, achieving significant advantages in both translation quality and decoding speedup. (3) DASpeech obviously outperforms the corresponding cascaded systems ( D1-D4 vs. E2-E5 ), demonstrating the effectiveness of our expect-path training approach. We also find that the cascaded model prefers larger graph size ( λ = 1 . 0 is better) while DASpeech prefers smaller graph size ( λ = 0 . 5 is better). We think the reason is that a larger graph size can improve S2TT performance, but it also makes end-to-end training more challenging. We further study the effects of the graph size in Appendix C. (4) On the CVSS-T dataset, which includes target speeches from various speakers, we observe a performance degradation in Translatotron and Translatotron 2 as the target mel-spectrogram becomes more difficult to predict. In contrast, DASpeech still performs well since its acoustic decoder explicitly incorporates variation information to alleviate the acoustic multi-modality, demonstrating the potential of DASpeech in handling complex and diverse target speeches.

Table 2 shows the results on CVSS-C dataset of the multilingual X → En S2ST model. We report the average ASR-BLEU scores on all languages, as well as the average scores on high/middle/lowresource languages 9 . We find that DASpeech still obviously outperforms S2UT but performs slightly worse than Translatotron 2 and UnitY in the multilingual setting, with an average gap of about 1.3 ASR-BLEU compared to Translatotron 2. However, DASpeech has about 13 times decoding speedup compared to Translatotron 2, achieving a better quality-speed trade-off.

8 Note that the original UnitY uses subwords instead of phonemes. Here we use phonemes just for consistency with other systems.

[0, 300)

[300, 600)

[600, 900)

[900, 1200)

Length of Source Speech (#Frames)

0

100

200

300

400

500

600

700

800

Decoding Time (ms/sample)

S2UT

Translatotron

UnitY

Translatotron 2

TranSpeech (iter=5)

TranSpeech (iter=15 + b=15 + NPD)

DASpeech (

=0.5, Lookahead)

DASpeech (

=0.5, Joint-Viterbi)

1

2

4

6

8

10

12

14

16

18

20

Speedup Ratio

14.00

16.00

18.00

20.00

22.00

24.00

26.00

ASR-BLEU

#iter=3

#iter=4

#iter=5

#iter=10

#iter=15

#iter=15 + b=15 + NPD

=1.5

=1.0

=0.5

=0.25

=1.5

=1.0

=0.5

=0.25

DASpeech (Lookahead)

DASpeech (Joint-Viterbi)

TranSpeech

S2UT

Translatotron

UnitY

Translatotron 2

4.3 Alternative Training Approach: Best-Path Training

In addition to the expect-path training approach proposed in Section 3.2, we also experiment with a simpler approach: BestPath Training . The core idea is to select the most probable path ˆ A = arg max A ∈ Γ P θ ( Y, A | X ) via Viterbi algorithm [20], and take the hidden states on path ˆ A = (ˆ a 1 , ..., ˆ a M ) as input to the acoustic decoder, i.e., z i = v ˆ a i . As shown in Table 3, bestpath training also performs well but is inferior to expect-path training. We attribute this to the fact that when using best-path training, only hidden states on the most probable path participate in TTS training, which may result in insufficient training for the remaining hidden states. In contrast, all hidden states participate in TTS training with our expect-path training, which achieves

better performance. The time complexity of Viterbi algorithm used in the best-path training is also O ( ML 2 ) . More details about the best-path training can be found in Appendix E.

4.4 Analysis of Decoding Speed

In this section, we provide more detailed analysis of decoding speed. Figure 2 shows the translation latency of different models for speech inputs of different lengths. The results indicate that the translation latency of autoregressive models (S2UT, Translatotron, UnitY, and Translatotron 2) significantly increases with the length of the source speech. In contrast, the translation latency of non-autoregressive models (TranSpeech and DASpeech) are hardly affected by the source speech length. When translating longer speech inputs, DASpeech's decoding speed can reach more than 20 times that of S2UT. Furthermore, we illustrate the quality-speed trade-off of different models in Figure 3. By adjusting the hyperparameter λ , the translation quality and decoding latency will change. Specifically, as λ increases, the decoding latency of the model will increase, and it achieves the best translation quality when λ = 0 . 5 . It is evident that DASpeech achieves the best trade-off between translation quality and decoding latency among all models. We further study the speedup under batch decoding in Appendix D.

4.5 Voice Preservation

In this section, we investigate the voice preservation ability of direct S2ST models on the CVSS-T Fr → En dataset, where target speeches are in voices transferred from source speeches. Specifically, we use a pretrained speaker verification model 10 [29] to extract the speaker embedding of the source

speech and generated target speech. We define the cosine similarity between source and target speaker embeddings as speaker similarity , and report the average speaker similarity on the test set in Table 4. We find that: (1) unit-based S2ST model can not preserve the speaker's voice since discrete units contain little speaker information; and (2) DASpeech can better preserve the speaker's voice than Translatotron and Translatotron 2, since its acoustic decoder explicitly introduces variation information of the target speech, allowing the model to learn more complex target distribution.

5 Related Work

Direct Speech-to-Speech Translation Speech-to-speech translation (S2ST) extends speech-to-text translation [30-33] which further synthesizes the target speech. Translatotron [3] is the first S2ST model that directly generates target mel-spectrograms from the source speech. Since continuous speech features contain a lot of variance information that makes training challenging, Tjandra et al. [34], Zhang et al. [35] use the discrete tokens derived from a VQ-V AE model [36] as the target. Lee et al. [5, 37] extend this research line by leveraging discrete units derived from the pretrained HuBERT model [38] as the target. To further reduce the learning difficulty, Inaguma et al. [7], Jia et al. [6], Chen et al. [39] introduce a two-pass architecture that generates target text and target speech successively. To address the data scarcity issue, some techniques like pretraining and data augmentation are used to enhance S2ST [8, 40-44]. Huang et al. [9] proposes the first non-autoregressive S2ST model which achieves faster decoding speed. Our DASpeech extends this line of research and achieves better translation quality and faster decoding speed.

Non-autoregressive Machine Translation Machine translation based on autoregressive decoding usually has a high decoding latency [45]. Gu et al. [10] first proposes NAT for faster decoding speed. To alleviate the multi-modality problem in NAT, many approaches have been proposed [46] like knowledge distillation [47-49], latent-variable models [50, 51], learning latent alignments [52-56], sequence-level training [57, 58], and curriculum learning [25]. Recently, Huang et al. [11] introduce DA-Transformer, which models different translations with DAG to alleviate the multi-modality problem, achieving competitive results with autoregressive models. Ma et al. [59], Gui et al. [60] further enhance DA-Transformer with fuzzy alignment and probabilistic context-free grammar.

Non-autoregressive Text-to-Speech Ren et al. [61], Peng et al. [62] first propose non-autoregressive TTS that generates mel-spectrograms in parallel. FastSpeech 2 [12] explicitly models variance information to alleviate the issue of acoustic multi-modality. Many subsequent works enhance non-autoregressive TTS with more powerful generative models like variational auto-encoder (VAE) [63, 64], normalizing flows [65-67], and denoising diffusion probabilistic models (DDPM) [68, 69]. DASpeech adopts the design of FastSpeech 2 for training stability and good voice quality.

6 Conclusion

In this paper, we introduce DASpeech, a non-autoregressive two-pass direct S2ST model. DASpeech is built upon DA-Transformer and FastSpeech 2, and we propose an expect-path training approach to train the model end-to-end. DASpeech achieves comparable or even better performance than the state-of-the-art S2ST model Translatotron 2, while maintaining up to 18.53 × speedup compared to the autoregressive model. DASpeech also significantly outperforms previous non-autoregressive model in both translation quality and decoding speed. In the future, we will investigate how to enhance DASpeech using techniques like pretraining and data augmentation.

7 Limitations & Broader Impacts

Limitations Although DASpeech achieves impressive performance in both translation quality and decoding speed, it still has some limitations: (1) the translation quality of DASpeech still lags behind Translatotron 2 in the multilingual setting; (2) the training cost of DASpeech is higher than Translatotron 2 (96 vs. 18 GPU hours) since it requires dynamic programming during training; and (3) the outputs of DASpeech are not always reliable, especially for some low-resource languages.

Broader Impacts In our experiments, we find that DASpeech emerges with the ability to maintain the speaker identity during translation. It raises potential risks in terms of model misuse, such as mimicking a particular speaker or voice identification spoofing.

References

A. Lavie, A. Waibel, L. Levin, M. Finke, D. Gates, M. Gavalda, T. Zeppenfeld, and Puming Zhan. Janus-iii: speech-to-speech translation in multiple languages. In 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing , volume 1, pages 99-102 vol.1, 1997. doi: 10.1109/ICASSP.1997.599557.

S. Nakamura, K. Markov, H. Nakaiwa, G. Kikui, H. Kawai, T. Jitsuhiro, J.-S. Zhang, H. Yamamoto, E. Sumita, and S. Yamamoto. The atr multilingual speech-to-speech translation system. IEEE Transactions on Audio, Speech, and Language Processing , 14(2):365-376, 2006. doi: 10.1109/TSA.2005.860774.

Ye Jia, Ron J. Weiss, Fadi Biadsy, Wolfgang Macherey, Melvin Johnson, Zhifeng Chen, and Yonghui Wu. Direct speech-to-speech translation with a sequence-to-sequence model. In Gernot Kubin and Zdravko Kacic, editors, Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019 , pages 1123-1127. ISCA, 2019. doi: 10.21437/Interspeech.2019-1951. URL https: //doi.org/10.21437/Interspeech.2019-1951 .

Ye Jia, Michelle Tadmor Ramanovich, Quan Wang, and Heiga Zen. CVSS corpus and massively multilingual speech-to-speech translation. In Proceedings of Language Resources and Evaluation Conference (LREC) , pages 6691-6703, 2022.

Ann Lee, Peng-Jen Chen, Changhan Wang, Jiatao Gu, Sravya Popuri, Xutai Ma, Adam Polyak, Yossi Adi, Qing He, Yun Tang, Juan Pino, and Wei-Ning Hsu. Direct speech-to-speech translation with discrete units. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 3327-3339, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.235. URL https://aclanthology.org/2022.acl-long.235 .

Ye Jia, Michelle Tadmor Ramanovich, Tal Remez, and Roi Pomerantz. Translatotron 2: Highquality direct speech-to-speech translation with voice preservation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 10120-10134. PMLR, 17-23 Jul 2022. URL https: //proceedings.mlr.press/v162/jia22b.html .

Hirofumi Inaguma, Sravya Popuri, Ilia Kulikov, Peng-Jen Chen, Changhan Wang, Yu-An Chung, Yun Tang, Ann Lee, Shinji Watanabe, and Juan Pino. Unity: Two-pass direct speech-to-speech translation with discrete units, 2022.

Sravya Popuri, Peng-Jen Chen, Changhan Wang, Juan Pino, Yossi Adi, Jiatao Gu, Wei-Ning Hsu, and Ann Lee. Enhanced direct speech-to-speech translation using self-supervised pre-training and data augmentation. In Hanseok Ko and John H. L. Hansen, editors, Interspeech 2022, 23rd Annual Conference of the International Speech Communication Association, Incheon, Korea, 18-22 September 2022 , pages 5195-5199. ISCA, 2022. doi: 10.21437/Interspeech.2022-11032. URL https://doi.org/10.21437/Interspeech.2022-11032 .

Rongjie Huang, Jinglin Liu, Huadai Liu, Yi Ren, Lichao Zhang, Jinzheng He, and Zhou Zhao. Transpeech: Speech-to-speech translation with bilateral perturbation. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview. net/forum?id=UVAmFAtC5ye .

Jiatao Gu, James Bradbury, Caiming Xiong, Victor O. K. Li, and Richard Socher. Nonautoregressive neural machine translation. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net, 2018. URL https://openreview.net/forum?id= B1l8BtlCb .

Fei Huang, Hao Zhou, Yang Liu, Hang Li, and Minlie Huang. Directed acyclic transformer for non-autoregressive machine translation. In Proceedings of the 39th International Conference on Machine Learning, ICML 2022 , 2022.

Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech 2: Fast and high-quality end-to-end text to speech. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=piLPYqxtWuA .

Fei Huang, Pei Ke, and Minlie Huang. Directed acyclic transformer pre-training for highquality non-autoregressive text generation. Transactions of the Association for Computational Linguistics , 2023.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/file/ 3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf .

Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. Conformer: Convolutionaugmented Transformer for Speech Recognition. In Proc. Interspeech 2020 , pages 5036-5040, 2020. doi: 10.21437/Interspeech.2020-3015.

Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 2978-2988, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/ P19-1285. URL https://aclanthology.org/P19-1285 .

L.R. Rabiner. A tutorial on hidden markov models and selected applications in speech recognition. Proceedings of the IEEE , 77(2):257-286, 1989. doi: 10.1109/5.18626.

Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. In Proceedings of the 34th International Conference on Neural Information Processing Systems , NIPS'20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.

Chenze Shao, Zhengrui Ma, and Yang Feng. Viterbi decoding of directed acyclic transformer for non-autoregressive machine translation. In Findings of EMNLP 2022 , 2022.

A. Viterbi. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE Transactions on Information Theory , 13(2):260-269, 1967. doi: 10.1109/TIT. 1967.1054010.

Changhan Wang, Anne Wu, and Juan Pino. Covost 2: A massively multilingual speech-to-text translation corpus, 2020.

Christophe Veaux, Junichi Yamagishi, and Kirsten MacDonald. Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit. 2017.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. URL http://arxiv.org/abs/1412.6980 .

Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations , 2019.

Lihua Qian, Hao Zhou, Yu Bao, Mingxuan Wang, Lin Qiu, Weinan Zhang, Yong Yu, and Lei Li. Glancing transformer for non-autoregressive neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International

Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 1993-2003, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021. acl-long.155. URL https://aclanthology.org/2021.acl-long.155 .

Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers , pages 186-191, Brussels, Belgium, October 2018. Association for Computational Linguistics. doi: 10.18653/v1/W18-6319. URL https: //www.aclweb.org/anthology/W18-6319 .

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics , pages 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 10.3115/1073083.1073135. URL https: //www.aclweb.org/anthology/P02-1040 .

xiaobo liang, Lijun Wu, Juntao Li, Yue Wang, Qi Meng, Tao Qin, Wei Chen, Min Zhang, and Tie-Yan Liu. R-drop: Regularized dropout for neural networks. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 10890-10905. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/file/ 5a66b9200f29ac3fa0ae244cc2a51b39-Paper.pdf .

Li Wan, Quan Wang, Alan Papir, and Ignacio Lopez Moreno. Generalized end-to-end loss for speaker verification. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , page 4879-4883. IEEE Press, 2018. doi: 10.1109/ICASSP.2018.8462665. URL https://doi.org/10.1109/ICASSP.2018.8462665 .

Qingkai Fang, Rong Ye, Lei Li, Yang Feng, and Mingxuan Wang. Stemm: Self-learning with speech-text manifold mixup for speech translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics , 2022.

Qingkai Fang and Yang Feng. Back translation for speech-to-text translation without transcripts. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics , 2023.

Qingkai Fang and Yang Feng. Understanding and bridging the modality gap for speech translation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics , 2023.

Yan Zhou, Qingkai Fang, and Yang Feng. CMOT: Cross-modal mixup via optimal transport for speech translation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 7873-7887, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.436. URL https://aclanthology.org/2023.acl-long.436 .

Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura. Speech-to-speech translation between untranscribed unknown languages. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) , pages 593-600, 2019. doi: 10.1109/ASRU46091.2019.9003853.

Chen Zhang, Xu Tan, Yi Ren, Tao Qin, Kejun Zhang, and Tie-Yan Liu. Uwspeech: Speech to speech translation for unwritten languages. Proceedings of the AAAI Conference on Artificial Intelligence , 35(16):14319-14327, May 2021. doi: 10.1609/aaai.v35i16.17684. URL https: //ojs.aaai.org/index.php/AAAI/article/view/17684 .

Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu. Neural discrete representation learning. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/ paper/2017/file/7a98af17e63a0ac09ce2e96d03992fbc-Paper.pdf .

Ann Lee, Hongyu Gong, Paul-Ambroise Duquenne, Holger Schwenk, Peng-Jen Chen, Changhan Wang, Sravya Popuri, Yossi Adi, Juan Pino, Jiatao Gu, and Wei-Ning Hsu. Textless speech-to-speech translation on real data. In Proceedings of the 2022 Conference of the North

American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 860-872, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.naacl-main.63. URL https://aclanthology.org/ 2022.naacl-main.63 .

Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Trans. Audio, Speech and Lang. Proc. , 29: 3451-3460, oct 2021. ISSN 2329-9290. doi: 10.1109/TASLP.2021.3122291. URL https: //doi.org/10.1109/TASLP.2021.3122291 .

Peng-Jen Chen, Kevin Tran, Yilin Yang, Jingfei Du, Justine Kao, Yu-An Chung, Paden Tomasello, Paul-Ambroise Duquenne, Holger Schwenk, Hongyu Gong, Hirofumi Inaguma, Sravya Popuri, Changhan Wang, Juan Miguel Pino, Wei-Ning Hsu, and Ann Lee. Speech-tospeech translation for A real-world unwritten language. CoRR , abs/2211.06474, 2022. doi: 10. 48550/arXiv.2211.06474. URL https://doi.org/10.48550/arXiv.2211.06474 .

Ye Jia, Yifan Ding, Ankur Bapna, Colin Cherry, Yu Zhang, Alexis Conneau, and Nobu Morioka. Leveraging unsupervised and weakly-supervised data to improve direct speechto-speech translation. In Hanseok Ko and John H. L. Hansen, editors, Interspeech 2022, 23rd Annual Conference of the International Speech Communication Association, Incheon, Korea, 18-22 September 2022 , pages 1721-1725. ISCA, 2022. doi: 10.21437/Interspeech.2022-10938. URL https://doi.org/10.21437/Interspeech.2022-10938 .

Qianqian Dong, Fengpeng Yue, Tom Ko, Mingxuan Wang, Qibing Bai, and Yu Zhang. Leveraging pseudo-labeled data to improve direct speech-to-speech translation. In Hanseok Ko and John H. L. Hansen, editors, Interspeech 2022, 23rd Annual Conference of the International Speech Communication Association, Incheon, Korea, 18-22 September 2022 , pages 1781-1785. ISCA, 2022. doi: 10.21437/Interspeech.2022-10011. URL https: //doi.org/10.21437/Interspeech.2022-10011 .

Xuan-Phi Nguyen, Sravya Popuri, Changhan Wang, Yun Tang, Ilia Kulikov, and Hongyu Gong. Improving speech-to-speech translation through unlabeled text. arXiv preprint arXiv:2210.14514 , 2022.

Kun Wei, Long Zhou, Ziqiang Zhang, Liping Chen, Shujie Liu, Lei He, Jinyu Li, and Furu Wei. Joint pre-training with speech and bilingual text for direct speech to speech translation. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 1-5, 2023. doi: 10.1109/ICASSP49357.2023.10095616.

Ziqiang Zhang, Long Zhou, Chengyi Wang, Sanyuan Chen, Yu Wu, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei. Speak foreign languages with your own voice: Cross-lingual neural codec language modeling. CoRR , abs/2303.03926, 2023. doi: 10.48550/arXiv.2303.03926. URL https://doi.org/10.48550/arXiv. 2303.03926 .

Shaolei Zhang, Qingkai Fang, Zhuocheng Zhang, Zhengrui Ma, Yan Zhou, Langlin Huang, Mengyu Bu, Shangtong Gui, Yunji Chen, Xilin Chen, and Yang Feng. Bayling: Bridging crosslingual alignment and instruction following through interactive translation for large language models, 2023.

Jiatao Gu and Xiang Kong. Fully non-autoregressive neural machine translation: Tricks of the trade. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , pages 120-133, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/ v1/2021.findings-acl.11. URL https://aclanthology.org/2021.findings-acl. 11 .

Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing , pages 1317-1327, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v1/ D16-1139. URL https://aclanthology.org/D16-1139 .

Chunting Zhou, Jiatao Gu, and Graham Neubig. Understanding knowledge distillation in nonautoregressive machine translation. In International Conference on Learning Representations , 2020. URL https://openreview.net/forum?id=BygFVAEKDH .

Chenze Shao, Xuanfu Wu, and Yang Feng. One reference is not enough: Diverse distillation with reference selection for non-autoregressive translation. In Proceedings of NAACL 2022 , 2022.

Raphael Shu, Jason Lee, Hideki Nakayama, and Kyunghyun Cho. Latent-variable nonautoregressive neural machine translation with deterministic inference using a delta posterior. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020 , pages 8846-8853. AAAI Press, 2020. URL https://ojs.aaai.org/index.php/AAAI/article/view/6413 .

Yu Bao, Hao Zhou, Shujian Huang, Dongqi Wang, Lihua Qian, Xinyu Dai, Jiajun Chen, and Lei Li. latent-glat: Glancing at latent variables for parallel text generation. In ACL . 2022. URL https://arxiv.org/abs/2204.02030 .

Jindˇ rich Libovický and Jindˇ rich Helcl. End-to-end non-autoregressive neural machine translation with connectionist temporal classification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 3016-3021, Brussels, Belgium, OctoberNovember 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1336. URL https://aclanthology.org/D18-1336 .

Chitwan Saharia, William Chan, Saurabh Saxena, and Mohammad Norouzi. Non-autoregressive machine translation with latent alignments. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 1098-1108, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.83. URL https://aclanthology.org/2020.emnlp-main.83 .

Cunxiao Du, Zhaopeng Tu, and Jing Jiang. Order-agnostic cross entropy for non-autoregressive machine translation. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 2849-2859. PMLR, 2021. URL http://proceedings.mlr.press/v139/du21c.html .

Chenze Shao and Yang Feng. Non-monotonic latent alignments for CTC-based nonautoregressive machine translation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=Qvh0SAPrYzH .

Zhengrui Ma, Shaolei Zhang, Shoutao Guo, Chenze Shao, Min Zhang, and Yang Feng. Nonautoregressive streaming transformer for simultaneous translation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , 2023.

Chenze Shao, Jinchao Zhang, Yang Feng, Fandong Meng, and Jie Zhou. Minimizing the bag-ofngrams difference for non-autoregressive neural machine translation. Proceedings of the AAAI Conference on Artificial Intelligence , 34(01):198-205, Apr. 2020. doi: 10.1609/aaai.v34i01. 5351. URL https://ojs.aaai.org/index.php/AAAI/article/view/5351 .

Chenze Shao, Yang Feng, Jinchao Zhang, Fandong Meng, and Jie Zhou. Sequence-level training for non-autoregressive neural machine translation. Computational Linguistics , 47(4):891-925, December 2021. doi: 10.1162/coli_a_00421. URL https://aclanthology.org/2021. cl-4.29 .

Zhengrui Ma, Chenze Shao, Shangtong Gui, Min Zhang, and Yang Feng. Fuzzy alignments in directed acyclic graph for non-autoregressive machine translation. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/ forum?id=LSz-gQyd0zE .

Shangtong Gui, Chenze Shao, Zhengrui Ma, Xishan Zhang, Yunji Chen, and Yang Feng. Nonautoregressive machine translation with probabilistic context-free grammar. In Advances in Neural Information Processing Systems , 2023.

Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech: Fast, robust and controllable text to speech. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 3165-3174, 2019. URL https://proceedings.neurips.cc/paper/2019/hash/ f63f65b503e22cb970527f23c9ad7db1-Abstract.html .

Kainan Peng, Wei Ping, Zhao Song, and Kexin Zhao. Non-autoregressive neural text-to-speech. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 7586-7598. PMLR, 2020. URL http://proceedings.mlr.press/v119/peng20a. html .

Yoonhyung Lee, Joongbo Shin, and Kyomin Jung. Bidirectional variational inference for non-autoregressive text-to-speech. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=o3iritJHLfO .

Xu Tan, Jiawei Chen, Haohe Liu, Jian Cong, Chen Zhang, Yanqing Liu, Xi Wang, Yichong Leng, Yuanhao Yi, Lei He, Frank K. Soong, Tao Qin, Sheng Zhao, and Tie-Yan Liu. Naturalspeech: End-to-end text to speech synthesis with human-level quality. CoRR , abs/2205.04421, 2022. doi: 10.48550/arXiv.2205.04421. URL https://doi.org/10.48550/arXiv.2205. 04421 .

Chenfeng Miao, Shuang Liang, Minchuan Chen, Jun Ma, Shaojun Wang, and Jing Xiao. Flowtts: A non-autoregressive network for text to speech based on flow. In 2020 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2020, Barcelona, Spain, May 4-8, 2020 , pages 7209-7213. IEEE, 2020. doi: 10.1109/ICASSP40776.2020.9054484. URL https://doi.org/10.1109/ICASSP40776.2020.9054484 .

Jaehyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. Glow-tts: A generative flow for text-to-speech via monotonic alignment search. In Proceedings of the 34th International Conference on Neural Information Processing Systems , NIPS'20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.

Yi Ren, Jinglin Liu, and Zhou Zhao. Portaspeech: Portable and high-quality generative text-to-speech. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 1396313974, 2021. URL https://proceedings.neurips.cc/paper/2021/hash/ 748d6b6ed8e13f857ceaa6cfbdca14b8-Abstract.html .

Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail A. Kudinov. Grad-tts: A diffusion probabilistic model for text-to-speech. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 8599-8608. PMLR, 2021. URL http://proceedings.mlr.press/ v139/popov21a.html .

Rongjie Huang, Max W. Y. Lam, Jun Wang, Dan Su, Dong Yu, Yi Ren, and Zhou Zhao. Fastdiff: A fast conditional diffusion model for high-quality speech synthesis. In Luc De Raedt, editor, Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022 , pages 4157-4163. ijcai.org, 2022. doi: 10.24963/ijcai. 2022/577. URL https://doi.org/10.24963/ijcai.2022/577 .

A Details of Baseline Models

In our experiments, we implement five baseline systems using fairseq : S2UT, Translatotron, UnitY, Translatotron 2, and TranSpeech. We reproduce TranSpeech with their open-source implementations 11 . In this section, we mainly introduce the configurations of the other four baseline systems.

Figure 4 shows the model architectures of these models. In terms of model architecture, S2UT and Translatotron are single-pass S2ST models while UnitY and Translatotron 2 are two-pass S2ST models. In terms of predicted targets, S2UT and UnitY predict discrete units while Translatotron and Translatotron 2 predict mel-spectrograms. Below we describe the details of each model. The detailed hyperparameters can be found in Table 5.

Speech

Encoder

Linguistic

Decoder

Acoustic

Decoder

HH AH0 L OW1

27 38 9 11 30

(a) S2UT

Speech

Encoder

Linguistic

Decoder

Acoustic

Decoder

HH AH0 L OW1

(b) Translatotron

Speech

Encoder

Linguistic

Decoder

Acoustic

Decoder

HH AH0 L OW1

27 38 9 11 30

(c) UnitY

Text-to-Speech

Encoder

Speech

Encoder

Linguistic

Decoder

Acoustic

Decoder

HH AH0 L OW1

Text-to-Speech

Encoder

S2UT Our implemented S2UT model includes three parts: a speech encoder, a linguistic decoder, and an acoustic decoder. The speech encoder is the same as DASpeech. The linguistic decoder is appended to the top layer of the speech encoder for multi-task learning, which predicts the target phonemes during training. The acoustic decoder generates the reduced discrete units derived from the 11-th layer of the pretrained mHuBERT model 12 . We do not include other auxiliary tasks and remove CTC decoding in Lee et al. [5] for simplification. The model is trained from scratch for 100k steps. We use beam search with a beam size of 10.

Translatotron The speech encoder and linguistic decoder of Translatotron are the same as S2UT. The acoustic decoder generates mel-spectrograms autoregressively. The pre-net dimension is 32 and the reduction factor of the acoustic decoder is 5. The model is trained from scratch for 100k steps.

UnitY UnitY is a two-pass model that includes four parts: a speech encoder, a linguistic decoder, a text-to-speech encoder, and an acoustic decoder. The architecture of the speech encoder, linguistic decoder, and acoustic decoder are the same as S2UT. The additional text-to-speech encoder is used to bridge the gap in representations between two decoders. We remove R-Drop training for simplification. We first conduct S2TT pretraining and finetune the model for 50k steps. We set the beam size of the first-pass and second-pass decoder to 10 and 1, respectively.

Translatotron 2 The model architecture of Translatotron 2 is similar to UnitY except that the second decoder generates mel-spectrograms rather than discrete units. The reduction factor of the acoustic decoder is set to 5. We first conduct S2TT pretraining and finetune the model for 50k steps. The beam size is set to 10 for the first-pass decoder.

For all the above models, we save checkpoints every 2000 steps and average the last 5 checkpoints for evaluation, which is the same as DASpeech. For S2UT and UnitY, we use the pretrained unit-based HiFi-GAN 13 vocoder to synthesize waveform. For Translatotron and Translatotron 2, we use the same pretrained HiFi-GAN vocoder as DASpeech.

B Detailed Results on CVSS-C X → En Datasets

Table 6 summarizes the detailed results of each language pair on CVSS-C test sets of the multilingual X → En S2ST models.

C Effects of the Graph Size

In this section, we investigate how the graph size affects the performance. We vary the size factor λ from 0.25 to 1.5, and measure the translation quality of both the S2TT DA-Transformer model and DASpeech on the CVSS-C Fr → En test set. As shown in Figures 5 and 6, we observe that the performance of S2TT DA-Transformer keeps increasing as the graph size gets larger, which is consistent with the observations in machine translation [11, 59]. However, DASpeech performs best at λ = 0 . 5 and shows a performance drop at larger λ . We speculate that this is because larger graph size makes end-to-end training more challenging. We will investigate this issue in the future.

0.25

0.50

1.00

1.50

Size Factor

52.00

52.50

53.00

53.50

54.00

54.50

BLEU

52.68

52.71

52.90

53.55

53.79

53.89

53.85

54.24

Lookahead

Joint-Viterbi

D Speedup Under Batch Decoding

As Gu and Kong [46] pointed out, the speed benefits of non-autoregressive models may degrade during batch decoding. To better understand this problem, we evaluate the speedup ratio under different decoding batch sizes. As shown in Figure 7, the speedup ratio keeps dropping as the decoding batch size increases. Nevertheless, DASpeech ( λ = 0 . 5 with Joint-Viterbi decoding) still achieves more than 6 × speedup with a decoding batch size of 64 and maintains comparable performance with Translatotron 2.

1

2

4

6

8

10

12

14

16

18

Speedup Ratio

24.00

24.20

24.40

24.60

24.80

25.00

25.20

ASR-BLEU

Translatotron 2

UnitY

DASpeech (

=0.5) + Lookahead

DASpeech (

=0.5) + Joint-Viterbi

DASpeech (

=1.0) + Lookahead

DASpeech (

=1.0) + Joint-Viterbi

b=1

b=8

b=16

b=32

0.25

0.50

1.00

1.50

Size Factor

23.50

24.00

24.50

25.00

25.50

ASR-BLEU

24.08

24.71

24.41

24.37

24.64

25.03

24.80

24.96

Lookahead

Joint-Viterbi

E Best-Path Training

Best-path-training selects the most probable path ˆ A = (ˆ a 1 , ..., ˆ a M ) and takes the hidden states on path ˆ A as input to the acoustic decoder. Formally, given the target phoneme sequence Y , we can find the most probable path ˆ A = arg max A ∈ Γ P θ ( Y, A | X ) via Viterbi algorithm [20]. Specifically, we use δ i ( j ) to denote the probability of the most probable path so far (ˆ a 1 , ..., ˆ a i ) with ˆ a i = j that generates ( y 1 , ..., y i ) . Considering the definition of a 1 = 1 , we have δ 1 (1) = P 1 ,y 1 and δ 1 (1 < j ≤ L ) = 0 . For i > 1 , we can sequentially calculate δ i ( · ) from its previous step δ i -1 ( · ) due to the Markov property:

where ϕ i ( j ) stores ˆ a i -1 of the most probable path so far (ˆ a 1 , ..., ˆ a i -1 , ˆ a i = j ) . After M iterations, we can obtain the most probable path by backtracking from ˆ a M = L :

Finally, we select the hidden states on the most probable path, i.e., z i = v ˆ a i , as the input sequence of the acoustic decoder.

Peer reviews:
comments: SUMMARY:
This paper presents a non-autoregressive direct speech-to-speech translation model based on Directed Acyclic Transformer. Experiments are conducted on CVSS-C and CVSS-T datasets, demonstrating translation performance comparable to the SOTA (Translatotron 2) but more than 16x speed up. It also demonstrates capacity of voice preservation during translation.

STRENGTHS:
1) The proposed approach is sound
2) Although the proposed approach is largely a combination of existing works (Directed Acyclic Transformer for NMT and Translatotron 2 for S2ST), its effective adoption in S2ST is novel.
3) The experimental results on translation quality, speed, and voice preservation are all promising.
4) Audio samples accompanying the paper sound consistent with the quantitive evaluation conducted in the paper.

WEAKNESSES:
1) The main results (Table 1) is conducted on English-French translation, which is an easy case as the order of the two languages are very close. It would be helpful to include more results on languages with significant difference on ordering, such as English-German. Compared to autoregressive model, one potential disadvantage of non-autogressive model is it may not handle reordering well.

QUESTIONS:
1) Voice Preservation evaluation -- from the audio samples, it can been heard clearly as the unit-based approaches don't preserve voice, but DASpeech and Translatotron 1 & 2 can (as stated by the paper). However, looking at the numbers in Table 4, the similarities from Translatotron 1 & 2 are very close to unit-based approach (with DASpeech signicantly higher) -- any explaination?

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nDASpeech is a non-autoregressive direct speech-to-speech translation (S2ST) model. The model adopts a two-pass architecture, where a linguistic decoder first generates the target text, and an acoustic decoder then generates the target speech based on the hidden states of the linguistic decoder. The linguistic decoder is based on the DA-Transformer model, which models translations with a directed acyclic graph (DAG). The acoustic decoder is based on the FastSpeech 2 model.\n\nExperiments on the CVSS benchmark demonstrate that DASpeech can achieve comparable or even better performance than the state-of-the-art S2ST model Translatotron 2, while preserving up to 18.53x speedup compared to the autoregressive baseline model. Compared to the previous non-autoregressive S2ST model, DASpeech does not rely on knowledge distillation and iterative decoding, achieving significant improvements in both translation quality and decoding speed. Furthermore, DASpeech shows the ability to preserve the speaker's voice of the source speech during translation.\n\nSTRENGTHS:\n1. A new direct Speech to Speech translation model that use Conformer Encoder, Non-autoregressive Linguistic Decoder, Directed\nAcyclic Graph and Non-autoregressive Acoustic Decoder.\n2. Efficient architecture for Speech to Speech translation - up to 18.53 speedup compared to the autoregressive baseline model.\n3. Get comparable results to state of the art models in terms of translation quality (BLEU)\n\nA new direct Speech to Speech translation model:\nThis model is a neural network that can translate speech from one language to another directly, without the need for an intermediate step of transcribing the speech to text. This makes it more efficient and faster than traditional speech translation systems, which require two separate models: one for speech recognition and one for machine translation. The model uses a Conformer encoder, which is a type of neural network that has been shown to be effective for speech recognition tasks. The encoder converts the input speech signal into a sequence of hidden representations. These representations are then decoded by a non-autoregressive linguistic decoder, which generates the translation sentence one word at a time. The model also uses a directed acyclic graph (DAG) and a non-autoregressive acoustic decoder. The DAG is a data structure that allows the model to predict multiple words at a time, which can further improve the efficiency of the translation process. The non-autoregressive acoustic decoder generates the output speech signal one frame at a time, which helps to preserve the naturalness of the audio.\n\nEfficient architecture for Speech to Speech translation:\nThe model's architecture is designed to be efficient, both in terms of memory and speed. The Conformer encoder is a relatively lightweight model, which helps to reduce the memory requirements of the system. The non-autoregressive linguistic decoder and acoustic decoder are also designed to be efficient, which helps to speed up the translation process. The model has been shown to be up to 18.53 times faster than the autoregressive baseline model. This makes it a promising candidate for real-time speech translation applications.\n\nComparable results to state of the art models in terms of translation quality (BLEU):\nThe model has also been shown to achieve comparable results to state-of-the-art models in terms of translation quality (BLEU).\n\nWEAKNESSES:\n.\n\nQUESTIONS:\n1. The proposed method only evaluate for Fr-En languages. Can you evaluate the proposed method with more languages?\n2. What is this model offer other then speed-up? The results is comparable to T2. Can the translation results improve?\n3. Can you add ablation analysis of each component?\n\nLIMITATIONS:\n.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper describes a non-autoregressive direct Speech-to-Speech Translation (S2ST) model called DASpeech that achieves both high-quality translations and fast decoding speeds. DASpeech uses a two-pass architecture where a linguistic decoder generates the target text, and an acoustic decoder generates the target speech based on the hidden states of the linguistic decoder. The linguistic decoder used is the decoder of DA-Transformer, and the acoustic decoder used is FastSpeech 2. DASpeech achieves much better performance than non-autoregressive baseline and 18.53x speedup compared to autoregressive baseline models and preserves the speaker's voice of the source speech during translation.\n\nSTRENGTHS:\nThe paper proposes a non-autoregressive direct S2ST model that achieves both high-quality translations and fast decoding speeds. The experiments show that DASpeech performs better than the baseline NAT model and is comparable to the state-of-the-art AT model in terms of translation quality.\n\nWEAKNESSES:\nHowever, there are some issues in the paper that need to be addressed.\n\nFirstly, the novelty of the paper is not significant. The paper is based on the DA-Transformer model heavily borrowed from the reference [11] (i.e., NAT in text-to-text translation), and there is no breakthrough in terms of technology. Additionally, it is not clear that the proposed model can solve some unique problems in speech-to-speech translation scenario.\n\nSecondly, some of the contributions mentioned in the paper are not well-supported by the experiments. For example, the paper claims that DASpeech can achieve comparable or better performance than SOTA, but the experimental results show that it performs worse. \n\nThirdly, the efficiency comparison is not in-depth enough. Although efficiency is one of the major contributions of the paper, there is a lack of detailed analysis and tables to describe the efficiency. Moreover, since the paper is a NAT model, it is not fair to compare its speed with the AT model directly. It would be more reasonable to compare the efficiency with the NAT baseline.\n\nQUESTIONS:\nn/a\n\nLIMITATIONS:\nSee weakness", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method for fast and high quality speech to speech translation.\nThe paper adopts the two-pass decoding framework.\nThe first pass decoder adapts prior work on non autoregressive decoding for machine translation.\nThe second pass decoder reuses fastspeech 2 except that the input are the hidden states obtained by the first pass decoder.\nThe first pass and second pass components are pretrained on the speech to text translation task and the synthesis task respectively.\nResults show competitive performance compared to the state of the art, with substantial inference speedups.\nThe model is also shown to be able to preserve voice when trained on a corpus with the same voice on the source and target sides.\n\nSTRENGTHS:\n* the approach is effective: competitive wrt SoTA with much faster inference speed\n* the model can preserve the voice, which is an important feature towards authentic communication\n* the comparison to previous methods is very thorough as many previous methods are reimplemented/compared to\n\nWEAKNESSES:\nThere is no specific weakness other than most of the novelty comes from the prior work (DA approach, 2-pass decoding framework, fastspeech 2) and this work very effectively combines this prior work to solve a new task.\n\nQUESTIONS:\n* 227: \"We implement the following baseline systems.\" Would you be able to explain this in more details. Does reimplement mean reusing existing results, using existing implementation and retraining models on different datasets, etc.\n* 235:  \"We remove the R-Drop training [28] for simplification.\" It would be good to justify this choice, especially if it leads to suboptimal results for the corresponding baseline (and also given that there is an existing implementation for that work).\n* 245: \"an autoregressive S2TT model.\" Would it be possible to add details on that S2TT model? Is is a transformer? A particular existing model (if so, add citation, etc.)?\n\nLIMITATIONS:\nThe limitations and broader impacts section is very effective in addressing limitations of the work and potential risks.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

comments: SUMMARY:
The paper introduces DASpeech, a directed acyclic transformer model for fast and high-quality speech-to-speech translation (S2ST). DASpeech employs a two-pass architecture, using a linguistic decoder based on DA-Transformer and an acoustic decoder based on FastSpeech 2. The proposed model captures the complex multimodal distribution of the target speech and achieves comparable or better performance than Translatotron 2, with significant speedup compared to autoregressive models. DASpeech also demonstrates the ability to preserve the source speaker's voice during translation.

STRENGTHS:
By incorporating the DA-Transformer and FastSpeech 2, DASpeech leverages the strengths of these models to capture both linguistic and acoustic diversity, resulting in high-quality translations.

WEAKNESSES:
1. The novelty is not enough, the method of this article seems to just learn from the DAT method of NMT and use it in ST 
2. While the paper addresses the challenges of high decoding latency in S2ST models, it would be beneficial to provide more detailed analysis or experimentation on the trade-off between translation quality and decoding speed in DASpeech.

QUESTIONS:
Could the authors elaborate on any specific differences between DASpeech and the DA Translation particularly?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-07