Simultaneous Speech Translation

Simultaneous SpeechTranslation
Graham NeubigNara Institute of Science and Technology (NAIST)
10/16/2015
Joint Work With:
Satoshi Nakamura, Tomoki Toda, Sakriani Sakti, Tomoki Fujita,
Hiroaki Shimizu, Yusuke Oda, Takashi Mieno, Quoc Truong Do
1

Simultaneous Speech Translation

Background2

Simultaneous Speech Translation

Speech Translation

Source: Microsoft Research
http://research.microsoft.com/en-us/news/features/translator-052714.aspx

Source: NICT
http://www.nict.go.jp/press/2010/06/29-1.html

3
Source: Karlsruhe Institute of Technology
http://isl.anthropomatik.kit.edu/english/1520.php

Simultaneous Speech Translation

Traditional Speech Translation
Divide atsentence boundaries

ASR

こんにちは、駅はどこですか？
MT
Hello, where is the station?
TTS
4Simultaneous Speech Translation

Problem: Delay (Ear-Voice Span)
Delay
ASR
こんにちは、駅はどこですか？
MT
Hello, where is the station?
TTS
5Simultaneous Speech Translation

Speech Translation Example

6Simultaneous Speech Translation

Simultaneous Speech Translation
Delay: ReducedASR
こんにちは、駅はMT

MT

どこですか？

MT

Hello,the station where is it?

TTS

TTS

TTS

But, this is not easy!

7Simultaneous Speech Translation

Professional Simultaneous InterpretationPhoto Credit:
https://www.flickr.com/photos/joi/2027679714
https://www.flickr.com/photos/european_parliament/4268490015

8

Simultaneous Speech Translation

Simultaneous Interpretation Data
[Shimizu+ LREC14]
 Recorded data
－ About 10 Hours of TED Talks(English-Japanese, Japanese-English)
 Simultaneous interpreters－ 3 pros with varying years ofexperience－ Ranked S, A, and B

Experience
15 years
4 years
1 yearFreely available for research purposes:
http://ahclab.naist.jp/resource/stc/Rank
S rankA rankB rank9

Simultaneous Speech Translation

Simultaneous Interpreter Example

10

Simultaneous Speech Translation

Source:So How do SimultaneousInterpreters Do It?

今ご覧いただいたこの映像は今から五年前、日本で世間を賑わせていた裁判員制度が始まる一年前、大学四年生だった私が模擬裁判用の資料として作った物ですTranslation:
Five years ago, as a college senior, I created the video that you justsaw as a reference material for a mock trial, one year before themuch-talked-about jury system commenced in Japan.

Interpretation: Predict NP
You just saw this video clip. Five years ago, at that time in Japan,
the ordinary people's justice system, jury system, was very muchtalked about in Japan, and I created this video as a referencematerial for that.

Segmentation Prediction

Rewording Summarization

11

Simultaneous Speech Translation

Can We Do the Same in
Speech Translation Systems?
Four problems in this talk:
●
●

●

●

Segmentation: When do we start translating?
Prediction: Can we predict things that haven't been
said?
Rewording: Can we reword sentences to be conducive
to simultaneous translation?
Evaluation: How do we decide which results arebetter?
12

Simultaneous Speech Translation

Segmentation

13

Simultaneous Speech Translation

Heuristic Segmentation Strategies
Division on pauses [Fugen+ 07, Bangalore+ 12]

hellowhere is the stationcommano comma
Division on predicted commas [Sridhar+ 13]
Division based on reordering probabilities [Fujita+ 13]

hello → probability of reordering 0.1
where → probability of reordering 0.814

Simultaneous Speech Translation

Optimizing Segmentation Strategies for
Simultaneous Speech Translation[Oda+ ACL14]●●

All previous segmentation strategies were based onheuristics
Don't directly take into account effect on translation
accuracy

What if we could directly optimize sentence
segmentation for translation accuracy?

15

Simultaneous Speech Translation

Training/Testing Framework
Training Corpus
src src src
src src src
src src src

trg trg trg
trg trg trg
trg trg trg

Find segmentation S*
that maximizes MT
accuracy

Segmentation S*
src src src
src src src
src src src

ModelTrain segmentation
model

Testing Corpus

Segmented Test

Translated Test

src src src
src src src
src src src

src src src
src src src
src src src

trg trg trg
trg trg trg
trg trg trg

Segment

Translate16Simultaneous Speech Translation

S* Search Method 1: Greedy Search
I ate lunch but she left

私は昼食を食べたが彼女は帰った

I ate lunch but she left 私 昼食を食べたが彼女は帰った 0.7
I ate lunch but she left 私は食べた ランチ彼女は帰った 0.4
I ate lunch but she left 私は昼食を食べた しかし彼女は帰った 0.6
I ate lunch but she left 私は昼食を食べたが 彼女は帰った 1.00.2
私は食べたが彼女 左I ate lunch but she left
I ate lunch but she left
I ate lunch but she left 私 昼食を食べたが 彼女は帰った 0.9
I ate lunch but she left 私は食べた 昼食だが 彼女は帰った 0.3
I ate lunch but she left 私は昼食を食べたしかし 彼女は帰った 0.6
I ate lunch but she left
0.2
私は昼食を食べたが 彼女 左I ate lunch but she left
17

Train SVM classifier to recover / at test timeSimultaneous Speech Translation

S* Search Method 2:
Grouping by Features●●

Because MT/Evaluation is complicated, there is thepotential to overfit
Solution: group boundaries by features
IPRN

atelunch

VBD

NN

Pronoun + Verb

I

PRN

ate

VBDbutCC

Noun + Conjunction

anDETapple
NN

shePRNleft
VBD

Determiner + NounandCC

anDETorange
NNSearch can be performed using dynamic programming
Features for the model trivial, no learning is needed18

Simultaneous Speech Translation

Results on TED Talks→ 2-3 times faster with no loss in BLEU

19

Simultaneous Speech Translation

Simultaneous Translation Demo●

Greedy+Grouping at 10 words

20

Simultaneous Speech Translation

Future Contributions to Segmentation?
●●●●Speech:Optimized models using acoustic features?
Parsing:
Incorporation with incremental parsing? e.g. [Ryu+ 06]
Machine Learning:
Smarter models: neural networks?
Algorithms:

Integration with incremental decoding? e.g. [Sankaran+ 10]

21

Simultaneous Speech Translation

Prediction

22

Simultaneous Speech Translation

What Kind of Prediction do SimultaneousInterpreters Do? [Wilss 78, Chernov+ 04]●Lexical prediction

サイエンスを正しく楽しく、これを合い言葉にサイエンス CGscience
factual fun
this keyword as science CG
then what I wanted to do is to
クリエーターとして活動しています。
creatorasworkingpromote fun and factual science, that's my keyword. I'm a …
●

Structural prediction

今 ご覧頂いた 映像
now you saw
video
you just saw a video clip

23

Simultaneous Speech Translation

Predicting Sentence-final Verbs[Grissom et al., EMNLP14]
Method for translating from verb-final languages (e.g. German)● Train a classifier to predict the sentence-final verb
● Use reinforcement learning to decide to “wait” “predict” or “commit”●24Simultaneous Speech Translation

Syntax-based Simultaneous Translation through Predictionof Unseen Syntactic Constituents [Oda+ ACL15]
Predict unseen syntax constituents● Translate from correct tree
●Predict

In the next 18 minutes IVPPPS

IN

NP
NP

in

PPNP

DT

JJ

CDNNSNNthenext18 minutesI

今 から 18 分 私NPINinNP

VPPRPDTJJCDNNSthenext18 minutesI(VP)

今 から 18 分 で 私 は (VP)

25

Simultaneous Speech Translation

Why is Syntax Necessary?
●Tree-to-string (T2S) MT framework●●Obtains state-of-the-art results on syntactically distant language
pairs (c.f. phrase-based translation; PBMT)
Possible to use additional syntactic constituents explicitly
S

This is NP

ParseVPNPDT VBZThis●NPMT

これ は NP で す

isAdditional heuristic to wait for more input based on whentranslation requires reordering

26

Simultaneous Speech Translation

Making Training Data for Syntax Prediction●Decompose gold trees in the treebank
1. Select any leaf span in the tree
2. Find the path between
leftmost/rightmost leaves

SNP

3. Delete the outside subtree

VPDT VBZ

4. Replace inside subtrees
with topmost phrase label

DTisaThisNN

NNNN

5. Finally we obtain:nilNP

isapennil
Leaf span

Left syntax

Leaf span

Right syntax

27Simultaneous Speech Translation

Syntax Prediction Process
1. Parse the input as-is

2. Extract features

PPIN

Word:R1=I
POS:R1=NN
Word:R1-2=I,minutes
POS:R1-2=NN,NNS...

NPNPDT

JJ CDNPNNSNN

in the next 18 minutes

I

ROOT=PPROOT-L=IN
ROOT-R=NP...

3. Predict the next tag(linear SVM)
VP ... 0.65
NP ... 0.28
4. Append tonil ... 0.04
sequence
...

VPnil

5. Repeat until nil

Input translation unit
28

Simultaneous Speech Translation

Results: Translation Trade-off (1)Translation Accuracy

BLEU

RIBES

0.15

0.6

0.14

0.58
0.56

0.13

0.54

0.12

0.52
0.11
0.5
0.1
0.48
0.09

0.46

0.08

0.44

0.07
0246

Short

810121416Long

0.42
0246

Short

810121416Long

Mean #words in inputs ∝ Delay

PBMT

Using N-words segmentation (not-optimized)●Short inputs reduce translation accuracies

29Simultaneous Speech Translation

Results: Translation Trade-off (2)Translation Accuracy

BLEU

RIBES

0.15

0.6

0.14

0.58
0.56

0.13

0.54

0.12

0.52
0.11
0.5
0.1
0.48
0.09

0.46

0.08

0.44

0.07
02

4

6

8

Short

PBMT
T2S

10

1214

16

Long

0.42
02

46

8

Short

10121416Long

Mean #words in inputs ∝ DelayLong phrase● Short phrase●... T2S > PBMT... T2S < PBMT

30

Simultaneous Speech Translation

Results: Translation Trade-off (3)Translation Accuracy

BLEU

RIBES

0.15

0.6

0.14

0.58
0.56

0.13

0.54

0.12

0.52
0.11
0.5
0.1
0.48
0.09

0.46

0.08

0.44

0.07
02

46

8

Short

PBMT
T2S
Proposed

10

1214

16

Long

0.42
02

46

Short

810121416

LongMean #words in inputs ∝ Delay

Prevent accuracy decreasing in short phrases● More robustness for reordering
31●

Simultaneous Speech Translation

Future Contributions to Prediction?
●●●Language Modeling:
More sophisticated models for lexical prediction.
Lexical Simplification:
Predict a more general word, then replace it later?
Machine Learning:
End-to-end reinforcement learning of the wholesystem?
Application of neural MT models?

32

Simultaneous Speech Translation

Rewording33

Simultaneous Speech Translation

What Kinds of Rewording
May Be Helpful?
Conjunction Clauses [Shimizu+ 13]
X because Y
Y dakara X
X nazenaraba Y●

Passivization [He+ 15]

●

私は 昨日
安い 本を 買った
I yesterday a cheap book bought
I●bought a cheap book yesterday
yesterday a cheap book was bought by meetc.

34Simultaneous Speech Translation

Constructing a Speech Translation System using
Simultaneous Interpretation Data [Shimizu+ IWSLT13]
 Approach:－ Incorporate simultaneous interpretation data intraining the MT system
Proposed

Traditional
Interpreted

Translated

Training
Input

Interpretationlike results

Translation System
 [Paulik+ 08] use interpretation data, but to improve accuracy
35

Simultaneous Speech Translation

Incorporating Interpretation Data
Interpretation data is small,so use adaptation techniques Tuning (Tu)－ Tune the parameters of the translation systems
to match the interpretation data
 Language Model (LM): Linear Interpolation－ Match the style of simultaneous interpreters
 Translation Model (TM): fill-up [Bisazza+ 11]－ Like the LM, adapt the TM to match interpretation data36

Simultaneous Speech Translation

Experimental Evaluation
Accuracy measured against simultaneousinterpretation reference

PhraseSentence

37

Simultaneous Speech Translation

Examples of Learned Traits

Shortening

Starting sentences with “OK” or “And”(Also done by interpreter in 25% of sentences)38

Simultaneous Speech Translation

Syntax-based Rewriting for Simultaneous
Machine Translation [He+ EMNLP15]●Reword the target language to be closer to source●

Passivizing, changing order of clauses when beneficial

39

Simultaneous Speech Translation

Future Contributions in Rewording?
●●Paraphrasing:More generalized models of structural paraphrasing?
Semantic Similarity:
How can we evaluate semantic similarity between
sentences structurally different from the reference?

40

Simultaneous Speech Translation

Evaluation

41

Simultaneous Speech Translation

Speed vs. Accuracy
Tradeoff between speed and accuracy.
High●Accuracy

do you have a more reasonable hotel ? /Don’t split the sentence
もっと 手頃な ホテルは ありませんかmore cheap hotel
is there
Split the sentenceLowmore / reasonable / is there a hotel ? /Short●DelayLong

Given two systems of different speed and accuracy,
which is better?

42

Simultaneous Speech Translation

Speed or Accuracy? A Study in Evaluation of
Simultaneous Speech Translation Systems[Mieno+ InterSpeech15]

Accuracy

Accuracy• Based on speed and accuracy, determine which system is better

Delay
Delay
Accuracy

HighLow

Delay

43

Simultaneous Speech Translation

How to Create an Evaluation Function?
(Based on Data)
FeaturesAccuracy
Manual
Evaluation
ResultsDelayEvaluation Function

Movie data

Translations with various
delays and accuracies

Machine Learning

Training Data

44

Simultaneous Speech Translation

Manual Evaluation Format• Rank-based evaluation

System A

Output A

System B

Output B

System C

Output C

2

Ranking byevaluators

Input video– Perform comparative evaluation of which output is “better”– Allows for consideration of both speed and accuracy1345

Simultaneous Speech Translation

Evaluation Sheet Example

46

Simultaneous Speech Translation

Learning an Evaluation Function
Define a linear function that takes a video as inputand returns a score

Weight vector

Features useful inevaluation(i.e., delay and accuracy)

Displayedvideo

This function can be learned from ranked datausing “learning to rank”
47

Simultaneous Speech Translation

Experimental Setup• Target video
TED Talks① Realtime trans. isimportant② Often used in MT
evaluation• Gathered dataVideo

20 Types

20-30 Seconds

Delay

7 Types

0,1,2,3,5,7,10Seconds

Accuracy

3 Types

Auto: BLEU/RIBES
Man: Adequacy

Subjects

15Japanese speakers

Modalities

SubtitledDubbed• Translation data
(5 varieties)
English → JapaneseTranslatorInterpreter 1(S Rank)
Interpreter 2(A Rank)
Syntax-based MT
Phrase-based MT
48

Speed or Accuracy? A Study in Evaluation of Simultaneous Speech Translation

Evaluation of Evaluation

Accuracy

Text Subtitles
10.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0

Acc.

Delay+Acc.

Speech
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0None
BLEU+1
RIBES
Adeq.

Acc.

Delay+Acc.

49

Speed or Accuracy? A Study in Evaluation of Simultaneous Speech Translation

Q1: Is Delay Important in S2STranslation?

Accuracy

Text Subtitles
10.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0Acc.

Delay+Acc.

Speech
10.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0None
BLEU+1
RIBES
Adeq.

Acc.

Delay+Acc.

A: Yes! In all cases, the scoring function considering delay
did as good or better than just considering accuracy.
50

Speed or Accuracy? A Study in Evaluation of Simultaneous Speech Translation

Q2: Does Importance Depend on
Modality of Presentation?

Accuracy

Text Subtitles
10.9
0.8
0.7
0.6
0.5
0.4
0.3
0.20.10Avg. +7%

Acc.

Delay+Acc.

Speech
10.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0Avg. +3%
None
BLEU+1RIBES
Adeq.

Acc.

Delay+Acc.

A: Yes! Considering delay was more useful whenpresenting results through subtitles.
Why?: Probably because when watching subtitles, it
is possible to hear the original speech.

51

Speed or Accuracy? A Study in Evaluation of Simultaneous Speech Translation

Learned Evaluation Functions(for Adequacy)
Accuracy Delay

1 point of adequacy =Subtitle Output

1.40-0.059

Speech Output

1.99-0.018

8.0 sec. of delay
28.5 sec. of delay

Subtitle Output

43210246Delay (s)Speech Output

55 Level Adequacy

5 Level Adequacy

58104

32

10246Delay (s)81052

Simultaneous Speech Translation

Future Contributions in Evaluation?
●●●

Adaptation:
A more flexible evaluation measure that generalizes tomany modalities, genres, tasks.
Machine Learning:
Non-linear regression functions?
Speech/UI:Other factors including presentation modality(avatars?), synthesis quality play a large role.

53

Simultaneous Speech Translation

Conclusion

54

Simultaneous Speech Translation

Conclusion●●The problem of high-accuracy simultaneous translation
covers many fields of NLP/Speech: parsing, machinelearning, language modeling, prosody, paraphrasing.
Still a new field, lots of opportunities for interesting
applications of NLP tech!

55