1

Recent Advances in Speech Translation
Dr. Satoshi Nakamura*with Katsuhito Sudo, Sakriani Sakti# ,and Ryo Fukuda, Sashi Novitasari, Tomoya Yanagita,
Kosuke Doi, Yasumasa Kano, Yuka Ko, Yuki Yano, Hirotaka Tokuyama, Yui Oka*Director, Data Science Center,

Professor, Graduate School of Science and Technology,

Nara Institute of Science and Technology(#currently with JAIST)
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 12 2021

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2021 @IGDTUW India

http://www.naist.jp/

2

Talk Outline
Recent advances– Machine translation
Speech translation– Speech translation research history– Simultaneous speech translation

Summary and future directions

無限の可能性、ここが最先端 －Outgrow your limits－

2021/11/12

AIIS 2021 Jakarta, Copyright Satoshi Nakamura, NAIST

http://www.naist.jp/

3

Recent MT progress
Rule-based MT：–

Linguists generate translation rules

Corpus-based MT:–––

Example-Based: Automatic rule extraction from corpus[M.Nagao84, Sato et.al.,89, Sumita et. al., 91 ]Statistical MT: Statistical Modeling of MT based on Noisy Channel Model [P.F.Brown, et.al. 93]
Phrase-base SMT•

P.Koehn, et al., “Statistical Phrase-based Translation”, UNIVERSITY OF SOUTHERN CALIFORNIA MARINA DEL REY INFORMATION SCIENCES INST
2003

Tree-to-string–

Statistical MT based on Tree Structure•

Y.Liu, et al,” Adjoining Tree-to-String Translation”, Proc. ACL 2011

Neural Machine Translation–

Combination of Encoder and Decoder by LSTM•

I.Sutskever, “Sequence to Sequence Learning with Neural Networks”, Proc. NIPS 2014

Attention NMT–

Add Attention to encoder and decoder•

D.Bardanau, et al., “Neural Machine Translation by Jointly Learning to Align and Translate”, Proc. of ICLR 2014

Transformer NMT–

Self attention by multiple heads. Transformer.
•

Ashish Vaswani et al., Attention Is All You Need, Proc. of NIPS 2017.

無限の可能性、ここが最先端 －Outgrow your limits－

2021/11/12

AIIS 2021 Jakarta, Copyright Satoshi Nakamura, NAIST

http://www.naist.jp/

4

Phrase Based Statistical Machine Translation⚫

Divide the sentence into small phrases and translate
SVO: Subject Verb Object

Today I will give a lecture on machine translation .
Today
I will give
今日は、 を行います

Today
今日は、

a lecture onの講義

machine translation
機械翻訳

machine translation
機械翻訳

a lecture onの講義

I will giveを行います

今日は、機械翻訳の講義を行います。
kyowa kikaihonyaku no kogi wo okonaimasu
⚫

.
。

.
。
SOV: Subject Object Verb

Score translations with translation model (TM),reordering model (RM), and language model (LM)

無限の可能性、ここが最先端 －Outgrow your limits－

2021/11/12

AIIS 2021 Jakarta, Copyright Satoshi Nakamura, NAIST

http://www.naist.jp/

5

Statistical MT• Translation model, Language Model, Distortion Model
Source and target
language paralleltext corpus

Target language
text corpus

Parameter estimation

Parameter estimation

Translation model Reordering modelPhrase substitution
Input text(
Source Language)

無限の可能性、ここが最先端 －Outgrow your limits－

Language model
Grammaticalcorrectness

Machine Translation
Decoding

2021/11/12

Translation text
(Target Language)

AIIS 2021 Jakarta, Copyright Satoshi Nakamura, NAIST

http://www.naist.jp/

6

Encoder-decoder Model [Sutskever+ 14]
Memorize input sentence by LSTM recurrent neural network
Generate output sentence by LSTM recurrent neural network

これ
kore

は 機械
翻訳
です
wa kikai honnyaku desu

Memorize
Sentence
Generate MT sentence
looking back the memory

Vector Representation

Decoder

Encoder

Vector Representation

Memorize input
sentence

無限の可能性、ここが最先端 －Outgrow your limits－

2021/11/12

This

is

a

machine translation

AIIS 2021 Jakarta, Copyright Satoshi Nakamura, NAIST

http://www.naist.jp/

7

Attention Mechanism [Bahdanau+ 14]
Better Memorization of Sentence and Looking-back Mechanism– Weighted-sum by the attention

これ
kore

は 機械
翻訳
です
wa kikai honnyaku desu

Looking back byAttention Mechanism

Vector Representation

Vector RepresentationThis

無限の可能性、ここが最先端 －Outgrow your limits－

2021/11/12

isamachine translation

AIIS 2021 Jakarta, Copyright Satoshi Nakamura, NAIST

http://www.naist.jp/

8Transformer: Fully Attention-based NN [Vaswani+ 2017]
No RNNs, stacked NNs Self-attention to extract
context dependent information
Positional encoding
instead of recurrent steps
Multi-head attention toutilize various aspects

無限の可能性、ここが最先端 －Outgrow your limits－

2021/11/12

AIIS 2021 Jakarta, Copyright Satoshi Nakamura, NAIST

http://www.naist.jp/

History of Speech Translation Research in Japan
1986

NAIST

1992
ATR

1999

ATR

2006

Daily
Conversation• Syntactically correct• Clear utterance• Limited domain
Ex. “Conference
Registration”• Standard expression• Unclear utterance• Limited domain
Ex. “Hotel Reservation”

2010

NICT

ATR

Read
Speech

2008

9

Wider andReal Domain+ More languages
for translation•21 multilateral text translation
VoiceTra• Wider and real domain“International Travel”
A-STAR• Realistic expressions• Noisy speech• J-E, J-C speech translation • Multilateral translation for 8 Asian languages• Network-based S2ST

2011

C-STAR• Multilateral translation for 7 world languages

Fundamentals Rule-based• Evaluation Campaign of S2S technologies

Statistical
MachineTranslation

MachineTranslation
無限の可能性、ここが最先端 －Outgrow your limits－

IWSLT

2021/11/12

Simultaneous
SpeechTranslation

NeuralMachineTranslation

AIIS 2021 Jakarta, Copyright Satoshi Nakamura, NAIST

http://www.naist.jp/

10

Simultaneous Incremental Speech Translation
Delay: Reduced
ASR
こんにちは、駅はkonnichiwa ekiwa

MT

どこですか？
dokodesuka

MTMT

Hello,

the station where is it?

TTS

TTS

無限の可能性、ここが最先端 －Outgrow your limits－

TTS

But, this is not easy!
2021/11/12

AIIS 2021 Jakarta, Copyright Satoshi Nakamura, NAIST

http://www.naist.jp/

11

Cascade Simultaneous S2S Translation System

I am a chef and a foodpolicy guy,
but I come from ...

Input English
speech

Output Japanese
speech

私はシェフで、食べ物政策の人…

Incremental

Incremental

Incremental

ASR
(ISR)

MT
(IMT)

TTS
(ITTS)

Intermediate
English transcript

I was a chef and a foodpolicy guy,
but I come from ...

無限の可能性、ここが最先端 －Outgrow your limits－

2021/11/12

私はシェフで、食べ物 Intermediate
Japanese MT result
政策の人…

AIIS 2021 Jakarta, Copyright Satoshi Nakamura, NAIST

http://www.naist.jp/

STACL: Wait-k Algorithm [M.Ma, et al., 2018]

Encoder-decoder NN

ブッシュ

大統領

は

今日

with

Putin

today

.

プーチン

12

と

Wait-3President

Bush

meets

Mingbo Ma, et al., “STACL: Simultaneous Translation with Integrated Anticipation and Controllable Latency”, arXiv:1810.08398v3 [cs.CL] 3 Nov 2018
無限の可能性、ここが最先端 －Outgrow your limits－
AIIS 2021 Jakarta, Copyright Satoshi Nakamura, NAIST
2021/11/12

http://www.naist.jp/

13Translation Timing Control by Syntactic Prediction in SMT (2015,2021)
Syntax Prediction Process

Syntactic Prediction [Oda, et al., 2015]
1. Parse the input as-is– Incremental bottom up parsing– Feature extraction and syntactic prediction

PP
IN

Subject Verb Object : English
Subject Object Verb: Japanese

DT

JJ

CD

NP
NNS

in the next 18 minutes
Input translation unit

Wait MT output when specific labels appear.

Use LSTM and BERT to predict next tag.
[Kano, et al., 2021]

Word:R1=I
POS:R1=NN
Word:R1-2=I,minutes
POS:R1-2=NN,NNS
...

NP
NP

– Control MT output timing according
reordering

2. Extract features

10/16/2016

NN

I

ROOT=PPROOT-L=IN
ROOT-R=NP...

3. Predict the next tag(linear SVM)
VP ... 0.65
NP ... 0.28
4. Append tonil ... 0.04
sequence
...

VP

nil

5. Repeat until nil‘m going to take you on a journey
46

Invited Talk ©Satoshi Nakamura, NAIST

Increment
al parsing
andsyntactic
predictionin the next 18 minutes[NP]predict [VP] (wait)

MT results

18 分 で あ る[NP]i 'm going to take [keep]
i 'm going to take you on a journey [VP end]

[VP](wait) を 行 っ て い ま す[keep]
皆さん を 旅 に お連れ します [VP end]

Oda, Yusuke et al., Syntax-based Simultaneous Translation through Prediction of Unseen Syntactic Constituents, Proc. of ACL-IJCNLP 2015.
Y.Kano, K.Sudo, S.Nakamura, “Simultaneous Neural Machine Translation with Constituent Label Prediction”, Proc. of WMT 2021.
無限の可能性、ここが最先端 －Outgrow your limits－

2021/11/12

AIIS 2021 Jakarta, Copyright Satoshi Nakamura, NAIST

http://www.naist.jp/

14

Video

無限の可能性、ここが最先端 －Outgrow your limits－

2021/11/12

AIIS 2021 Jakarta, Copyright Satoshi Nakamura, NAIST

http://www.naist.jp/

15

Problem in the current Simultaneous Speech Translation System
Source speech spectrogram (English)

Seriousdelay

Target speech spectrogram(Japanese)

無限の可能性、ここが最先端 －Outgrow your limits－

2021/11/12

AIIS 2021 Jakarta, Copyright Satoshi Nakamura, NAIST

http://www.naist.jp/

16

Evaluation: Quality (ASR & MT for TED Talks)
ISR+IMT

ISR
Model
LSTM
Nonincremental Transformer
LSTM(low latency)

LSTM
Incremental(high latency)

Transformer(low latency)

Transformer(high latency)

BLEU- Subjective evaluation
4 (%) Adequacy Fluency

WER (%)
25.46
20.74
31.88
32.43

Gold transcript+MT

15.7

3.41

3.93

Non-incremental
ASR+MT

12.8

3.20

4.01

IMT

5.1

2.80

3.03

(low latency EVS:8.81s)

IMT

32.06

(medium latency
EVS:11.87s)

8.4

2.98

3.54

25.01

IMT

9.4

3.34

3.80

(high latency EVS:16.91s)

R.Fukuda, et al, “SIMULTANEOUS SPEECH-TO-SPEECH TRANSLATION SYSTEM WITH TRANSFORMER-BASED INCREMENTAL ASR, MT,
AND TTS”, Proc. Oriental COCOSDA 2021
無限の可能性、ここが最先端 －Outgrow your limits－

2021/11/12

AIIS 2021 Jakarta, Copyright Satoshi Nakamura, NAIST

http://www.naist.jp/

17

Summary
Remarkable progress– Statistical Models– Recurrent Deep Neural Network– Progress in Speech Translation

Automatic Simultaneous Speech Translation– Data Collection– Automatic Simultaneous Speech Translation for distant language-pairs

Further Research– Higher Quality with Shorter Latency– Evaluation of Simultaneous Speech Interpretation– Context/ Situation dependency– Semantics, Discourse Analysis– Para-linguistics/ Multi-modal

無限の可能性、ここが最先端 －Outgrow your limits－

2021/11/12

AIIS 2021 Jakarta, Copyright Satoshi Nakamura, NAIST

http://www.naist.jp/

18

Thank you for listening

無限の可能性、ここが最先端 －Outgrow your limits－

2021/11/12

AIIS 2021 Jakarta, Copyright Satoshi Nakamura, NAIST

http://www.naist.jp/

19

Human Interpreting[A.Mizuno 2016]
E-J Translation Example(1) The relief workers (2) say (3) they don’t have (4) enough food, water, shelter,and medical supplies (5) to deal with (6) the gigantic wave of refugees (7) who areransacking the countryside (8) in search of the basics (9) to stay alive.
(1) 救援担当者は (9) 生きるための (8) 食料を求めて (7) 村を荒らし回っている (6) 大量の難民達の (5)世話をするための (4) 十分な食料や水，宿泊施設，医療品が (3) 無いと (2) 言っています．

(1) 救援担当者達の (2) 話では (4)食料，水，宿泊施
設，医薬品が， (3) 足りず (6) 大量の難民達の (5)世話が出来ないとのことです．(7) 難民達は今村々
を荒らし回って， (9) 生きるための (8) 食料を求めているのです．

Memory Chunk

Necessary #Chunk＞３！
無限の可能性、ここが最先端 －Outgrow your limits－

2021/11/12

Necessary #Chunk＜３！
AIIS 2021 Jakarta, Copyright Satoshi Nakamura, NAIST

http://www.naist.jp/

20

Recent Progress of ASR
Traditional Technologies– Template Matching, Dynamic Programing [Sakoe 71]– Hidden Markov Modeling, N-Gram Model [Mercer 83, etc]
– Neural Network, TDNN[Waibel 89], LSTM [Hochreiter 97]– Weighted Finite State Transducer [Mohri 2006]– Big Training Data, Data Collection through Trial Service

Deep Learning– DNN-HMM [Hinton 2012]• Estimate State Posterior Probability by DNN– Connectionist Temporal Classification [Graves 2013]• Predict Phoneme Label every frame– Listen, Attend, and Spell [Chan 2016]• CTC+Attention: End-to-end modeling– Transformer ASR• Faster calculation by Multiple-head attention
無限の可能性、ここが最先端 －Outgrow your limits－

2021/11/12

AIIS 2021 Jakarta, Copyright Satoshi Nakamura, NAIST

http://www.naist.jp/

21

Speech Recognition
2014+

1960

1990

Template matching
dynamic time warping

Statistical-based HMM

End-to-end ASR

Speech B

Speech A

無限の可能性、ここが最先端 －Outgrow your limits－

2021/11/12

AIIS 2021 Jakarta, Copyright Satoshi Nakamura, NAIST

http://www.naist.jp/

22

Speech Recognition (2)Speech recognition model (Y = text; X = speech feature)arg max p(Y |X ) = arg max p(X |Y ) p(Y )Conventional speech recognition (statistical-based HMM)o Acoustic model p(X |L)o Lexicon p(L |Y )o Language model p(Y)End-to-end ASRo Directly model p(Y|X) with a single networko Integrate acoustic p(X|L), lexicon p(L|Y), and language model p(Y) models

無限の可能性、ここが最先端 －Outgrow your limits－

2021/11/12

AIIS 2021 Jakarta, Copyright Satoshi Nakamura, NAIST

http://www.naist.jp/

23

Connectionist temporal classification (CTC)[Graves et al. 2014, Miao et al., 2015]

Speech (X) and text (Y) can vary in length
Use bidirectional RNNs to predict frame-based labels(including blanks)
Find speech-text alignments using dynamic
programming𝑇
CTC objective
𝑝 𝑌𝑋 =

CTC conditional
probability

෍𝐴∈𝐴𝑋,𝑌
Marginalizes overthe set of output
candidates

ෑ 𝑝𝑡 (𝑎𝑡 |𝑋)

𝑡=1
Computing theprobability for a singlealignment step-by-step

CTC

c

a

c

_ a

t
a

a

_

t

_

_

StackedRNNs

ASR loss: negative log-likelihood
෍ − log 𝑝(𝑌|𝑋)(𝑋,𝑌)

無限の可能性、ここが最先端 －Outgrow your limits－

2021/11/12

AIIS 2021 Jakarta, Copyright Satoshi Nakamura, NAIST

http://www.naist.jp/

24

Attention-based encoder-decoder[Chorowski et al., 2014; Chan et al., 2015]

Input:
Mel-spectrogram (continuous)
Output: Character / sub-word (discrete)• Encoder: stacked bidirectional RNNso Encode speech features into hidden representation• Decoder: stacked RNNso Decode the encoded representation using the previous output
and attention information• Attention:o Alignments between decoder state - encoder stateo Decoder finds only relevant information from encoder states.
• ASR loss: negative log-likelihood

無限の可能性、ここが最先端 －Outgrow your limits－

2021/11/12

AIIS 2021 Jakarta, Copyright Satoshi Nakamura, NAIST

http://www.naist.jp/

25

Recent Performance
Saon, et al. “English Conversational Telephone Speech Recognition by
Humans and Machines”, INTERSPEECH 2017

[1] R. P. Lippmann, “Speech recognition by machines and humans,”
Speech communication, vol. 22, no. 1, pp. 1–15, 1997.

無限の可能性、ここが最先端 －Outgrow your limits－

2021/11/12

AIIS 2021 Jakarta, Copyright Satoshi Nakamura, NAIST

http://www.naist.jp/

Attention-based encoder-decoder with Transformer

26[Dong et al., 2020]

Replace RNN with Transformer
Encoder and decoder with
– Self-attention to generate inner sequencealignment– Multi-head attention to leverage different
attending representation jointly– Positional encoding to represent positinal
information (Transformer ≠ RNN)

ASR loss: negative log-likelihood[Dong et al., 2020]
無限の可能性、ここが最先端 －Outgrow your limits－

2021/11/12

AIIS 2021 Jakarta, Copyright Satoshi Nakamura, NAIST

http://www.naist.jp/

27

Talk Outline
Recent advances– Machine translation (text-to-text)– Speech recognition (Speech-to-text)– Speech synthesis (Text-to-speech)
Speech translation– Speech translation research history– End-to-end speech translation– Simultaneous speech translation

Summary and future directions
無限の可能性、ここが最先端 －Outgrow your limits－

2021/11/12

AIIS 2021 Jakarta, Copyright Satoshi Nakamura, NAIST

http://www.naist.jp/

28

Recent Speech Synthesis
Formant-based Synthesis, Waveform Concatenation

Statistical Speech Synthesis: HTS– Speech Synthesis by HMM•

Tokuda, et al., “Speech parameter generation algorithms for HMM-based speech synthesis”, ICASSP 2000

WaveNet– Waveform Convolution• van den Oord et al., “WAVENET: A GENERATIVE MODEL FOR RAW AUDIO”, arXiv:1609.03499v2 [cs.SD] 19 Sep 2016

Tacotron (Encoder-decoder DNN TTS+ Griffin Lim)
– End-to-end speech synthesis with character input. Waveform generation by Griffin-Lim• Wang, et al., “TACOTRON: TOWARDS END-TO-END SPEECH SYNTHESIS”, arXiv:1703.10135v2 [cs.CL] 6 Apr 2017

Tacotron2 (Encoder-decoder DNN TTS+ Wavenet)• J. Sheng, et al, “Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions”,

Transformer TTS:
– Multi-head attention• N. Li, S. Liu, Y. Liu, S. Zhao, and M. Liu, “Neural speech synthesis with Transformer network,” in Proc. AAAI Conference on
Artificial Intelligence, vol. 33, no. 01, 2019, pp. 6706–6713• M. Chen, X. Tan, Y. Ren, J. Xu, H. Sun, S. Zhao, and T. Qin,“MultiSpeech: Multi-speaker text to speech with Transformer,”in Proc. INTERSPEECH, 2020, pp. 4024–4028.
無限の可能性、ここが最先端 －Outgrow your limits－

2021/11/12

AIIS 2021 Jakarta, Copyright Satoshi Nakamura, NAIST

http://www.naist.jp/

29

Speech Synthesis
Tacotron

Conditional WaveNet – TTS

HMM[Oord et al., 2016][Zen et al. 2009]
無限の可能性、ここが最先端 －Outgrow your limits－[Wang et al.; 2017]
2021/11/12

AIIS 2021 Jakarta, Copyright Satoshi Nakamura, NAIST

http://www.naist.jp/

Sequence-to-sequence TTS with Transformer

30[Li et al., 2018]

Input: Character / sub-word (discrete)
Output:o Mel-spectrogram (continuous)o End-of-speech flag (binary)Replace RNNs in Tacotron with Transformer
Autoregressive decoding:o Use the previous decoder timestep output to predict the current
timestep output (same as Tacotron)
Loss TTS: L1/L2 norm + cross-entropy for end-of-speech[Li et al., 2018]

無限の可能性、ここが最先端 －Outgrow your limits－

2021/11/12

AIIS 2021 Jakarta, Copyright Satoshi Nakamura, NAIST

http://www.naist.jp/

31

Adjusting Timing with Reordering Probabilities in SMT[Fujita,2013]
First, temporarily choose strings according to method one
Next, if that phrase's right probability exceeds a threshold, actually translate the words inthe cache⚫
⚫

Example (threshold = 0.8):
hello
where
is

the

station“hello”“hello where”“where is”“where is the”
“the station”
phrase exists phrase missing phrase exists phrase missing utterance ends
↓
↓
↓
↓
↓
wait
choose “hello”
waitchoose “where is”
translate↓↓“where isright probability
right probability is the station”
is 0.9 > 0.8
0.6 < 0.8↓↓translate “hello”
do not translate yet
Fujita, et. al., 2013⚫

Threshold 1.0 = traditional, 0.0 = method one

無限の可能性、ここが最先端 －Outgrow your limits－

2021/11/12

AIIS 2021 Jakarta, Copyright Satoshi Nakamura, NAIST

http://www.naist.jp/

Mechanism of speech-to-speech translation system

Multilingual
SpeechRecognition

Japanese「私は学校に行く: Watashi wa Gakko he iku」
Watashi wa
Gakko he iku

Statistical Speech
Recognition

Multilingual
SpeechSynthesis

SpokenLanguageTranslation

EnglishI go to school

I go to school

Statistical MachineTranslation +Multi-engine approach

Corpus-based
Speech Synthesis

CorporaLarge Scale
Japanese Speech
Corpora無限の可能性、ここが最先端－Outgrow your limits－
2021/11/12

Large Scale
Japanese Text
CorporaLarge Scale Parallel
Corpora ofJapanese and English

Large Scale
English Text
Corpora

AIIS 2021 Jakarta, Copyright Satoshi Nakamura, NAIST

Large Scale
English Speech
Corpora32

http://www.naist.jp/

33

Cascade Simultaneous S2S Translation System

I am a chef and a foodpolicy guy,
but I come from ...

Input English
speech

Output Japanese
speech

私はシェフで、食べ物政策の人…

Incremental

Incremental

Incremental

ASR
(ISR)

MT
(IMT)

TTS
(ITTS)

Intermediate
English transcript

I was a chef and a foodpolicy guy,
but I come from ...

無限の可能性、ここが最先端 －Outgrow your limits－

私はシェフで、食べ物 Intermediate
Japanese MT result
政策の人…

AIIS 2021 Jakarta, Copyright Satoshi Nakamura, NAIST
2021/11/12

http://www.naist.jp/

34

Incremental Speech Recognition (ISR) [Novitasari, 2020]

S.Novitasari, A.Tjandra, T.Yanagita, S.Sakti, S.Nakamura, “Incremental Machine Speech Chain Towards Enabling Listening while
Speaking in Real-time”, Proceedings of INTERSPEECH 2020, Oct. 2020
無限の可能性、ここが最先端 －Outgrow your limits－

2021/11/12

AIIS 2021 Jakarta, Copyright Satoshi Nakamura, NAIST

http://www.naist.jp/

35

Incremental Speech Synthesis (iTTS)

S.Novitasari, S.Sakti, S.Nakamura, “Dynamically Adaptive Machine Speech Chain Inference for TTS in Noisy Environment:
Listen and Speak Louder”, Proc. Interspeech 2021, 4124-4128, Aug. 30, 2021
無限の可能性、ここが最先端 －Outgrow your limits－

2021/11/12

AIIS 2021 Jakarta, Copyright Satoshi Nakamura, NAIST

http://www.naist.jp/

36

Overview of our corpus
Over 300 hours* = interpreted by interpreters from all 3 ranks (4h x 3 interpreters)
Others = interpreted by either an S- or A-rank interpreter
About half of the SIs have been transcribed
Direction

Source

2018

2019

2020

Experience

Rank

En → Ja

TED

67+12*

50

50

15 yearsS-rankJp → En

TEDx

12*

40

0

4 years

A-rank

CSJ

33

0

0

1 years

B-rank

JNPC

4

36.5

0

Total

128

126.5

50

Cum.

128

254.5

304.5

Detailed analysis: K.Doi, K.Sudoh, S.Nakamura, “Large-scale English-Japanese Simultanous Interpretation Corpus:
Construction and Analysis with sentence-aligned data, Proc. IWSLT 2021
無限の可能性、ここが最先端 －Outgrow your limits－

2021/11/12

AIIS 2021 Jakarta, Copyright Satoshi Nakamura, NAIST

http://www.naist.jp/

37

Evaluation: Quality (TTS)Model

L2-norm loss

TacotronNonincremental Transformer
Tacotron(low latency)

Tacotron
Incremental(high latency)

Transformer(low latency)

Transformer(high latency)

0.57
0.51
0.77

Tacotron
TTS

Sam e

Transform er TTS

Transformer
ITTS (7 words)

Transform er TTS

Sam e

0.58
Transformer
ITTS (7 words)

0.65

0.57

0%

25%

Transform er
ITTS (5 words)

50%

Same

75%

100%

R.Fukuda, et al, “SIMULTANEOUS SPEECH-TO-SPEECH TRANSLATION SYSTEM WITH TRANSFORMER-BASED INCREMENTAL ASR, MT,
AND TTS”, Proc. Oriental COCOSDA 2021AIIS 2021 Jakarta, Copyright Satoshi Nakamura, NAIST
無限の可能性、ここが最先端 －Outgrow your limits－

2021/11/12

http://www.naist.jp/

38

Evaluation Overview
The cascade system was evaluated using TED Talks data in:– System-level latency• Ear-Voice Span (EVS)– Span between the start of the input & output– Module-level quality• ISR: Word Error Rate (WER)• ISR+IMT: BLEU
• ITTS: L2-norm loss and subjective evaluation (AB preference test)

Three latency regimes: low, medium, high (following IWSLT)

R.Fukuda, et al, “SIMULTANEOUS SPEECH-TO-SPEECH TRANSLATION SYSTEM WITH TRANSFORMER-BASED INCREMENTAL ASR, MT,
AND TTS”, Proc. Oriental COCOSDA 2021AIIS 2021 Jakarta, Copyright Satoshi Nakamura, NAIST
無限の可能性、ここが最先端 －Outgrow your limits－

2021/11/12

http://www.naist.jp/

Problem: Delay (Ear-Voice Span)

39

Delay
ASR
こんにちは、駅はどこですか？
konnichiwa eki wa dokodesuka
MT
Hello, where is the station?
TTS

無限の可能性、ここが最先端 －Outgrow your limits－

2021/11/12

AIIS 2021 Jakarta, Copyright Satoshi Nakamura, NAIST

http://www.naist.jp/

40

Statistical Translation Frameworks
Symbolic Models

Continuous-space (Neural) Models

Phrase-based MT [Koehn+ 03]
he

has

a

has
引いている

he
彼は

a cold
風邪 を

彼

は 風邪

を

he

cold

he
彼は

Encoder-Decoder [Sutskever+ 14]
hasacold 


彼

は

風邪

を

引いて
いる

彼

は

風邪

を

引いて
いる




a cold
風邪 を

has
引いている

引いている

Attentional [Bahdanau+ 14]

Tree-to-String MT [Liu+ 06]

he

S

has

a

cold

VP
NP
PRP

VBZ

DET

NN

he

has

a

cold

彼

は

風邪

を

g1,...,g4
a1

a2

hi-1

引いている

2021/11/12

a4hi
P(ei|F,e1,...,ei-1)

ri-1

無限の可能性、ここが最先端 －Outgrow your limits－a3Intelligent and Invisible Computing

AIIS 2021 Jakarta, Copyright Satoshi Nakamura, NAIST

40

http://www.naist.jp/

41

Evaluation: Latency (TED Talks)
ISR: Incremental Speech Recognition, IMT: Incremental Machine Translation, ITTS: Incremental Text-to-Speech Synthesis

LatencyregimeLow
MediumHigh

ISR delay(sec.)

0.93
0.93

1.30

ISR+IMT
delay(sec.)

4.69
8.43

11.47

Total delay (=EVS)Hyper(sec.)
parameters

8.81

ISR: 64 frames
IMT: k=10
ITTS: 5 words

11.87

ISR: 64 frames
IMT: k=20
ITTS: 5 words

16.91

ISR: 64 frames
IMT: k=30
ITTS: 7 words

R.Fukuda, et al, “SIMULTANEOUS SPEECH-TO-SPEECH TRANSLATION SYSTEM WITH TRANSFORMER-BASED INCREMENTAL ASR, MT,
AND TTS”, Proc. Oriental COCOSDA 2021
無限の可能性、ここが最先端 －Outgrow your limits－

2021/11/12

AIIS 2021 Jakarta, Copyright Satoshi Nakamura, NAIST

http://www.naist.jp/

42

Simultaneous Interpretation
European Commission
Simultaneous interpreting is a mode of interpreting in which the speaker makes aspeech and the interpreter reformulates the speech into a language his audience
understands at the same time (or simultaneously).
The three main actions are also essentially the same as consecutive interpreting.
1) listen actively (understand)2) analyse (structure the message)3) reproduce (communicate)
The difference with consecutive interpreting is that in simultaneous interpreting
all of these things need to happen at the same time (or simultaneously).
In addition to a special way of listening, prioritising information anddistinguishing between primary and secondary information, activating shortterm memory, communicating, etc. ,a good simultaneous interpreter also has to be able to anticipate what the
speaker might say.
無限の可能性、ここが最先端 －Outgrow your limits－

2021/11/12

AIIS 2021 Jakarta, Copyright Satoshi Nakamura, NAIST

http://www.naist.jp/

43Translation Timing Control by Syntactic Prediction in NMT[Kano, et al, 2021]
ICLP: Incremental Constituent Label Prediction (LSTM, BERT)

Y.Kano, K.Sudo, S.Nakamura, “Simultaneous Neural Machine Translation with Constituent Label Prediction”, Proc. of WMT 2021.
無限の可能性、ここが最先端 －Outgrow your limits－

2021/11/12

AIIS 2021 Jakarta, Copyright Satoshi Nakamura, NAIST

http://www.naist.jp/