カリキュラムラーニングを用いた音声翻訳の学習戦略の提案○叶 高朋, Sakriani Sakti, 中村 哲
奈良先端科学技術大学院大学

2019/4/4

Takatomo Kano

1/21

Traditional Speech Translation
ASR
EnglishspeechMTi am very nervous

TTSJapanesespeech

私 は とても 緊張 して います

Traditional approach in speech-to-speech translation systems
 construct automatic speech recognition (ASR), machinetranslation (MT) and text to speech synthesis (TTS) all of which are independently trained and tuned
2019/4/4

Takatomo Kano

2/21

Traditional Speech Translation
ASRMT
ショーを購入したい

I’d like buy shoes
I’d like buy showASR error affect Translation performance.
 all of which are independently trained and tuned
 NMT module difficult handle input with error word.

2019/4/4

Takatomo Kano

3/21

Limitations in TraditionalApproach
1. Basic unit for information sharing is only words at the text level
 Many languages do not have written form
2. Speech acoustics might involve both linguistic and paralinguistic
information
Paralinguistic information:
 is not a factor in written
communication
 cannot even be expressed inwords
ASR have lost all of their paralinguistic information
2019/4/4

Takatomo Kano4/21

Objective of this Research“Direct speech-to-speech translation”

Direct Speech to SpeechTranslation

Input Speech
2019/4/4

Output Speech
Takatomo Kano

5/21

Related Works• Long Duong et al. NAACL 2016 [1]• Title: An Attentional Model for Speech Translation WithoutTranscription• Spanish to English speech-to-text direct translation withattentional encoder decoder networks• Alexandre Berard et al. NIPS workshop 2016 [2]• Title: Listen and Translate: A Proof of Concept for End-to-End
Speech-to-Text Translation• French to English speech-to-text direct translation withattentional encoder decoder networks• Yoshua Bengio et al. ICML 2009[3]• Title: Curriculum Learning• A Leaning strategy, learn from easy data to difficult data.
2019/4/4

Takatomo Kano

6/21

Related Works• End-to-end Speech-to-text translation withattentional modelSpeech
SignalAcoustic feature

Target Word

Attention

100 ~ frameContinues vector sequence
2019/4/4

Bi-directional
LSTM Encoder
Takatomo Kano

LSTM decoder
7/21

Problems• Their works are only applicable for similar syntax
and word order (SVO-SVO) [1,2]• For such languages, only local movements are
sufficient for translation.

Spanish to English translation
attention matrix [1]2019/4/4

French to English translation
attention matrix [2]Takatomo Kano

8/21

Problems• Syntactically distant language pairs (SVO versus SOV)
suffers from long-distance reordering phenomena.
朝食はいくら です

かhow0.003 9E-04

much

2E-04 0.819 0.828 0.033 0.833 #####

is0.003 0.017 0.037 0.005 0.166 #####

the0.01 0.026 0.038 0.024 2E-04 2E-04

breakfast

0.738 0.003 7E-04 ##### ##### 0.004?

0.08 0.122 0.001 0.882 ##### 0.935

0.09 0.057 0.001 #####English to Japanese NMT attention matrix
2019/4/4

Takatomo Kano

Japanese ASR attention matrix
9/21

Proposed method• A first step we focus Speech to Text direct translation
system(ST) on syntactically distant language pairs• Train attentional model on English-Japanese language
pairs with SVO versus SOV word order.
• To guide the encoder-decoder attentional model tolearn this difficult problem, we proposed astructured-based curriculum learning strategy.
2019/4/4

Takatomo Kano

10/21

Curriculum Learning• Curriculum learning [3]• One learning paradigm, is inspired by the learning
processes of humans and animals that learn from easieraspects and gradually increase to more difficult ones.

Sample inputs from Basic Shape(top)and Genome Shape(bottom)
2019/4/4

Takatomo Kano

11/21

Curriculum Learning• Original “Curriculum learning”• The attentional encoder-decoder architecture trained directly forspeech translation tasks using similar but more and more difficult
speech translation data
How to correct easy data in translation
task? Shorter one? Common word? ...

• Our proposed “Structure Based Curriculum learning”• We train the attentional encoder-decoder architecture by startingfrom a simpler task, switch a certain part of the structure ineach training phase, and set it to a more difficult target task.
2019/4/4

Takatomo Kano

12/21

Attention-based ST withCurriculum Learning
Phase 1Bi-LSTM
Encoder

ASRMTAttention

Attention
LSTM
Decoder

Bi-LSTM
Encoder

LSTM
Decoder

As before, we train the attentional-based encoder-decoder
neural network for a standard ASR and MT task
2019/4/4

Takatomo Kano

13/21

Attention-based ST withCurriculum Learning
Phase 2ASR

Attention
Bi-LSTM
Encoder

LSTM
Transcoder

Using NMT encoder
state as a target

MTThe model’s objective now is to predict the word representation(Using the MT encoder’s output)

2019/4/4

Takatomo Kano

14/21

Attention-based ST withCurriculum Learning
Phase 3

ASR + MT

Attention
Bi-LSTM
Encoder

LSTM
Transcoder

LSTM
Decoder

Slow track
We combine the MT attention and decoder modules to perform the speechtranslation task from the source speech sequence to the target word sequence
2019/4/4

Takatomo Kano

15/21

Attention-based ST withCurriculum Learning
Phase 2

ASR + MT

Attention
Bi-LSTM
Encoder

LSTM
DecoderMTMT
Attention

The model now predicts the corresponding word
sequence in the target language given the input speech
2019/4/4

Takatomo Kano

16/21

Attention-based ST with
Curriculum Learning
Phase 1
ASR

Phase 2Phase 3

Slow track

LSTM
DecoerMT

Fast track
Attentional-based neural trained for ASR and text-based MT tasks
and gradually train the network for end-to-end ST tasks.
2019/4/4

Takatomo Kano17/21

Experimental Set-up
System settings• Cor

Data settings

BTEC Para-text

ASR

Input units

23

Train utterance

45,000Hidden units

512

Valid utterance

5,000

LSTM layer depth

2,2Test utterance

500
BTEC Speech

MTSource Vocabulary27293

Target Vocabulary & Output size

33155

Input units & Embed size

128

Hidden units

512

LSTM layer Depth

2,2Train utterance

45,000

Test utterance

500

Speech feature

F-bank 23dim

ASR word error rate

512
OtherWe use Google TTS system to generate BTEC speech

OptimizerAdam

2019/4/4

Takatomo Kano

18/21

Learning loss for each epoch

 Using CL-based proposed method, we can further decrease the loss Specifically, the one that trained with CL type 1 – Slow Track
successfully
outperformed thetext-based MT system.
2019/4/4
Takatomo Kano
19/21

Translation Accuracy
WER 8%

 Best performance was achieved by proposed Slow Track model Surpassed the text-based MT and cascade ASR+MT systems.
2019/4/4

Takatomo Kano

20/21

Conclusion• To using pre-trained model and extend and training deep
network are succeed.
• We achieved English-Japanese end-to-end speech to texttranslation without being affected by ASR error.
• Experimental results demonstrated that the learning modelis stable and its translation quality outperformed thestandard MT system.
• In future works we intent to consider natural speech, andexpand the speech-to-text translation task to a speech-tospeech translation task.
2019/4/4

Takatomo Kano

21/21

Thank you for your listening.

2019/4/4

Takatomo Kano

22/19

Additional page
• “Using Spoken Word Posterior Features in Neural
Machine Translation” (IWSLT18 S.Sakti et al.)Bi-LSTMEncoder

ASRMT

Attention

Attention
LSTM
Decoder

LSTM
Decoder

Weakly End to End
Huge parameters
ASR and NMT have to
use same vocab

Easy to analysis
Easy to training
2019/4/4

Bi-LSTM
Encoder

Takatomo Kano

23/19

Additional page
Similar meaning or usage word
will embedding similar space.
Some time it makes attention
error.

Additional information( pronounce similarity) can fix
above problem and get goodattention result
2019/4/4

Takatomo Kano

24/19