Transformer-based Direct Speech-tospeech Translation with Transcoder
Takatomo Kano!, Sakriani Sakti!,#, and Satoshi Nakamura!,#1. Nara Institute of Science and Technology, Japan
2. RIKEN, Center for Advanced Intelligence Project AIP, JapanCOPYRIGHT Kano@AHC lab, NAIST Japan1

Introduction
ASR
EnglishspeechMT
HelloTTS
Konnichiwa

JapanesespeechThe traditional approach solves the speech-to-speech translation step by step.
However, there are many limitations.
ü Cascade of ASR, MT, and TTS using text.
ü All of which is independently trained and tuned.
COPYRIGHT Kano@AHC lab, NAIST Japan2

Existing work: Multi-task Speech-to-Speech Translation[Ye et al., 2019]
Multi-task modelü The model has three decoders but uses
only one decoder for translation.

Training step

TargetWaveform
Speaker
EmbeddingConcatEncoderEnglishLog-mel spectrogram

Attention

MT Decoder

Attention

ASR Decoder

TargetCharacter

TargetWaveform

Speaker
Embedding

Encoder

ST Decoder[Weiss et al., 2017, Ye et al., 2019]

Inference step

Vocoder

Attention

Multi-task model

ConcatVocoder

Attention

ST Decoder

EnglishLog-mel spectrogramü The model uses one attention module for alignment sourceand target speech.
EnglishCharacterü ASR and MT decoders do not work in the inference step.
ü There is no information sharing from ASR and MT.
ü All modules use RNN.

COPYRIGHT Kano@AHC lab NAIST Japan3

Proposed: Transformer-based Direct Speech-to-speech Translation
with Transcoder
Proposedü The model utilizes all trained modules to translate.
ASR
DecoderMT
Decoder

Transcoder

Attention

Transcoder

Attention

EnglishSpeechTargettext

Attention

EncoderEnglishtranscription

ST
DecoderJapanese
Speechü ASR and MT decoders provide attention alignment necessary in the next block.

ü The model uses a combination of 3 attentions for alignment source to target speech.
ü Transcoder converts different hidden states in speech translation.
ü The model converts ASR and MT hidden states into MT and TTS hidden states.
ü We revamp the RNN-based speech-to-speech translation model by the Transformer.
COPYRIGHT Kano@AHC lab NAIST Japan

4

Experiment Results
BLEU and METEOR scores of speech-to-speech translation
Syntactic similar
Model

Syntactic distant

En to ESJa to Ko

En to Ja

Ja to EnBLEU

METEORBLEU

METEOR

Baseline: Cascade (RNN)

38.9

47.7

38.749.1

32.5

44.2

32.0

43.2

Baseline: Cascade (Transformer)

41.3

52.1

41.0

51.1

34.1

45.2

35.0

45.3

Multi-task (RNN)

38.8

48.2

39.1

49.9

33.2

45.5

34.2

45.0

Multi-task (Transformer)

43.1

58.8

42.5

58.3

36.9

52.6

38.3

48.4

Transcoder(Transformer)

44.0

59.3

42.9

58.8

40.6

56.6

41.0

55.8

üOur proposed approach outperforms Multi-task speech translation.
üThe proposed method with Transformer further improves performances.
COPYRIGHT Kano@AHC lab, NAIST Japan

5

End of my highlight talk.

COPYRIGHT Kano@AHC lab, NAIST Japan

6

Limitations in Existing Approach
Many existing works only perform on syntactically similar language pairs.
English to French translation• Syntactically similar language pairs have
• Similar word ordering(SVO-SVO or SOV-SOV)• A lot of one-to-one monotonic alignments.

Thanksfor

Mercid'être• English-French, English-Spanish, Japanese-Korean• Syntactically distant language pairs havetoday

comingvenu

English to Japanese translation• Different word ordering(SVO-SOV)• A lot of many-to-many alignments.

Thanksforcomingtoday• English-Japanese• Difficult to translate

今⽇はお越しいただき ありがとう ございます

Direct End-to-End Speech Translation for Distant Language Pairs
COPYRIGHT Kano@AHC lab, NAIST Japan7

Existing work: Multi-task Speech-to-Speech Translation[Ye et al., 2019]
The speech encoder states are most important.
ü Source text decoder helps to tune source speech encoder states.
ü Source text helps find the right segment on source speech states.
TargetWaveform

Limitations
ü Use a simple encoder-decoder model totranslate input speech to target speech.
ü It’s challenging to handle difficulttranslation tasks due to simple architecture.
ü All modules use RNN.

Speaker
EmbeddingConcatEncoder

Vocoder

Attention

ST Decoder

Attention

MT Decoder

TargetCharacter

Attention

ASR Decoder

EnglishCharacter

EnglishLog-mel spectrogram
COPYRIGHT Kano@AHC lab, NAIST Japan

8Proposed: Transformer-based Direct Speech-to-speech Translation
with Transcoder

ASR
Decoder

MTDecoder

Transcoder

Attention

Transcoder

Attention

EnglishSpeechTargettext

Attention

EncoderEnglishtranscription

ST
DecoderJapaneseSpeechü ASR and MT decoders provide attention alignment necessary in the next block in the inference step.

ü The model uses a combination of 3 attentions for alignment source speech to target speech.
ü Transcoder converts different hidden states in speech translation.
ü The model converts ASR and MT hidden states and MT and TTS hidden states.
ü We revamp the RNN-based speech-to-speech translation model with the Transformer.
COPYRIGHT Kano@AHC lab, NAIST Japan9

Proposed: Transformer-based Direct Speech-to-speech Translation
with Transcoder
Alignment information is most important for Speech Translation.
ü This model provide attention alignment necessary in the next block .
ü Transcoder converts different hidden states in speech translation.
Limitations
ü The model has deep architecture, and thetraining is difficult.

Transcoder

attention

Encoderattention

ü We need additional training to convert
ASR hidden states into MT hidden states.

ASR
Decoder

TargetLanguage
Decoder

Konnichiawa

EnglishSpeechCOPYRIGHT Kano@AHC lab, NAIST Japan10

Proposed:

Training mechanism for Transcoder

Encoder

SpeechAttention

The difference between ASR and MT.
ü ASR models input acoustic phoneme sequences.
ü MT models input word sequence.
ü Each hidden states have different distributions. English

ASRDecoder

Transcoder

Context
sequence

Transcoder’s trainingü Transcoder gets ASR context sequence and encodes them with Helloconsidering context information.
ü A pre-trained MT encoder provides the target sequence.
COPYRIGHT Kano@AHC lab, NAIST Japan

Transcoder
sequence
Tuning transcoder withNMT encoder sequence

NMT
Encoder
NMT
encodersequence

11

Proposed:

Training mechanism for Transcoder
We extend the architecture and train model step by step.
ü We get an idea from Curriculum Learning. We change the model architecture and task difficulty.
ü During the training, the model becomes deep, and the task becomes difficult at each step.
ü We utilize a pre-trained MT and TTS encoder as a teacher model.
Step1: Pre-train
Step2: Train Transcoder
Step3: Extend MT decoder
attention

ASR
Encoder

ASR
Decoder

Helloattention

TTS
Encoder

TTS
Decoder

ASR
EncoderMT
Encoder

Transcoder

attention

EnglishSpeechTranscoder

attentionMT
Decoder

Attention

attention

MT
Encoder

ASR
Decoder

ASR
Decoder
ASR
Encoder

Step4...

MT
Decoder

Konnichiawa

EnglishSpeechCOPYRIGHT Kano@AHC lab, NAIST Japan

12

Experimental setups

ü Input is Japanese or English natural speech.
ü Target is Japanese, English, Korean, or Spanish
generated speech.
ü We use the ASR to transcribe the generated Mel
sequences and evaluate translation performance withBLEU and METEOR.

COPYRIGHT Kano@AHC lab, NAIST Japan

13

Experimental results

We evaluate ASR and MT performance to compare.
ü RNN and Transformer model.
ü Syntactically similar and distant language.

ü The Transformer model outperformed the RNN model on ASR and text MT tasks.
ü The syntactic similar language translation achieved a higher score than syntactic
distant language.
COPYRIGHT Kano@AHC lab, NAIST Japan14

We evaluate Speech translation performance to compare.
ü Proposed and Multi-task method.
ü RNN and Transformer model.
ü Syntactically similar and distant language.

Experiment Results

BLEU and METEOR scores of speech-to-speech translation
Syntactic similar
Model

Syntactic distant

En to ESJa to Ko

En to Ja

Ja to EnBLEU

METEORBLEU

METEOR

Baseline: Cascade (RNN)

38.9

47.7

38.7

49.1

32.5

44.2

32.0

43.2

Baseline: Cascade (Transformer)

41.3

52.1

41.0

51.1

34.1

45.2

35.0

45.3

Multi-task (RNN)

38.848.2

39.1

49.9

33.2

45.5

34.2

45.0

Multi-task (Transformer)

43.1

58.8

42.5

58.3

36.9

52.6

38.3

48.4

Transcoder(Transformer)

44.0

59.3

42.9

58.8

40.6

56.6

41.0

55.8

üOur proposed approach outperforms Multi-task speech translation.
üThe proposed method with Transformer further improved performances.
COPYRIGHT Kano@AHC lab, NAIST Japan15

Analysis of the proposed and Multi-task methods

ü The proposed method shows monotonic
attention alignments again.
ü The Multi-task method shows week attentionalignments similar to MT but not preciselythe same.

MTü Both MT attention alignments has a similar
direction, as shown in the arrow.

STü Both methods show monotonic attentionalignments at ASR tasks. However, the Multitask model's attention alignment is not clear.

Proposed

ASR

Multi-task

COPYRIGHT Kano@AHC lab, NAIST Japan

16Conclusion• In this research, we propose the Transformer-based speech-to-speechtranslation with a transcoder that can pass the context information foreach process.
• Our proposed model improves BLEU and METEOR scores compared with theMulti-task model on syntactically distant language.

• Our proposed model learns the end-to-end speech translation step bystep.
• ASR and MT decoders support alignment at each step.

• Our proposed model shows the best performance on both syntacticallysimilar and distant language pairs.
COPYRIGHT Kano@AHC lab, NAIST Japan17

Thank you for your listening.

COPYRIGHT Kano@AHC lab, NAIST Japan

18