Simple, Lexicalized Choice of Translation Timing for Simultaneous Speech Translation

Simple, Lexicalized Choiceof Translation Timingfor Simultaneous Speech Translation
Tomoki Fujita, Graham Neubig, Sakriani Sakti,
Tomoki Toda, Satoshi Nakamura
Graduate School of Information Science,
Nara Institute of Science and Technology
1Simple, Lexicalized Choice of Translation Timing for Simultaneous Speech Translation

Background2

Simple, Lexicalized Choice of Translation Timing for Simultaneous Speech Translation

Speech Translation Systems●Translate speech from source language to target

ASR
こんにちは、駅はどこですか？
MTHello, where is the station?
TTS
3Simple, Lexicalized Choice of Translation Timing for Simultaneous Speech Translation

Problem: Delay●

Wait for the whole utterance to end before translating
DelayASR
こんにちは、駅はどこですか？
MT
Hello, where is the station?
TTS
4Simple, Lexicalized Choice of Translation Timing for Simultaneous Speech Translation

Solution: Divide into Smaller Chunks●Choose appropriate timing to start translation
Delay: ReducedASR

こんにちは、駅はMT

MT

どこですか？

MT

Hello,the station where is it?

TTS

TTS

TTS
5

Simple, Lexicalized Choice of Translation Timing for Simultaneous Speech Translation

Previous Work: Incremental Dependency
Parsing/Manual Rules [Ryu+ 04]●

Utilize knowledge of English/Japanese to derive rules

subjI went

prepprepto the park with your brotherTranslate after the first prepositional phrase completes!

MTMT
私は公園に行きました

あなたの弟と●- Requires a bilingual linguist to design rules●- Requires an accurate incremental dependency parser 6

Simple, Lexicalized Choice of Translation Timing for Simultaneous Speech Translation

Previous Work: Division on Pauses[Fugen+ 08, Bangalore+ 12]●

Simply divide on short pauses in the utterance

ASR
hello●●where is the station- Cannot capture relationship between languages- Result will greatly change with speech speed,
disfluencies
7Simple, Lexicalized Choice of Translation Timing for Simultaneous Speech Translation

Proposed Method●Utilize the TM directly to choose translation timing+ Can be constructed automatically● + Uses information about the language pair
● + Very simple to implement●●

Specifically:
Choose translation timing at the end of each phrase in
the phrase table● Utilize reordering probabilities to adjust granularity● Adapt the language model to the translation task●

8Simple, Lexicalized Choice of Translation Timing for Simultaneous Speech Translation

Preliminaries

9Simple, Lexicalized Choice of Translation Timing for Simultaneous Speech Translation

Phrase Based Machine Translation●Divide the sentence into small phrases and translate

Today I will give a lecture on machine translation .
Today
I will give
今日は、 を行います

Today
今日は、

a lecture onの講義

machine translation
機械翻訳

machine translation
機械翻訳

a lecture onの講義

I will giveを行います.
。

.
。

今日は、機械翻訳の講義を行います。
●

Score translations with translation model (TM),reordering model (RM), and language model (LM)10Simple, Lexicalized Choice of Translation Timing for Simultaneous Speech Translation

Translation Model Creation●

Perform automatic alignment of bitext●

From aligned text, extract phrases for translation

ホテ 受ルの付
thehotelfrontdesk

ホテル の → hotel
ホテル の → the hotel
受付 → front desk
ホテルの受付 → hotel front desk
ホテルの受付 → the hotel front deskSimple, Lexicalized Choice of Translation Timing for Simultaneous Speech Translation

Lexicalized Reordering Model●●

Probabilistically models reorderings for increased
accuracy of translation
Given current phrase and next phrase:

Monotone:
背 の 高い 男

太郎 を 訪問 した

the tall man

visited Taro

Discontinuous Right:
私 は 太郎 を 訪問したI●Swap:

visited Taro

Discontinuous Left:
背 の 高い 男 を 訪問 した
visitedthe tallman“monotone” + “discontinuous right” = “right probability”

Simple, Lexicalized Choice of Translation Timing for Simultaneous Speech Translation

Proposed Method13Simple, Lexicalized Choice of Translation Timing for Simultaneous Speech Translation

Method One:
Choosing Translation Timing with Phrases●Input words one at a time from ASR●While words exist in phrase table, don't translate yetSimple, Lexicalized Choice of Translation Timing for Simultaneous Speech Translation

Method One:
Choosing Translation Timing with Phrases●

Input words one at a time from ASR●

While words exist in phrase table, don't translate yet

Phrase Table
hello→ こんにちは
where is→ どこですか
where→ どこthe→ そのInput String
hello
whereisthethe station→ 駅station

Simple, Lexicalized Choice of Translation Timing for Simultaneous Speech Translation

Method One:
Choosing Translation Timing with Phrases●

Input words one at a time from ASR●

While words exist in phrase table, don't translate yet

Phrase Table
hello→ こんにちは
where is→ どこですか
where→ どこthe→ そのInput String
hello
whereisthethe station→ 駅

stationSimple, Lexicalized Choice of Translation Timing for Simultaneous Speech Translation

Method One:
Choosing Translation Timing with Phrases●

Input words one at a time from ASR●

While words exist in phrase table, don't translate yet

Phrase Table
hello→ こんにちは
where is→ どこですか
where→ どこthe→ そのInput String
hello
whereisthe“hello”
phrase exists↓waitthe station→ 駅

stationSimple, Lexicalized Choice of Translation Timing for Simultaneous Speech Translation

Method One:
Choosing Translation Timing with Phrases●

Input words one at a time from ASR●

While words exist in phrase table, don't translate yet

Phrase Table
hello→ こんにちは
where is→ どこですか
where→ どこthe→ そのInput String
hello
whereisthe“hello”
phrase exists↓
waitthe station→ 駅station

Simple, Lexicalized Choice of Translation Timing for Simultaneous Speech Translation

Method One:
Choosing Translation Timing with Phrases●

Input words one at a time from ASR●While words exist in phrase table, don't translate yet

Phrase Table
hello→ こんにちはwhere is→ どこですか
where→ どこthe→ そのInput String
hello
whereisthe“hello”“hello where”
phrase exists phrase missing↓↓wait
translate“hello”

the station→ 駅station

Simple, Lexicalized Choice of Translation Timing for Simultaneous Speech Translation

Method One:
Choosing Translation Timing with Phrases●

Input words one at a time from ASR●

While words exist in phrase table, don't translate yet

Phrase Table
hello→ こんにちは
where is→ どこですか
where→ どこthe→ そのInput String
hellowhereis
the“hello”“hello where”
phrase exists phrase missing↓↓
waittranslate“hello”

the station→ 駅

stationSimple, Lexicalized Choice of Translation Timing for Simultaneous Speech Translation

Method One:Choosing Translation Timing with Phrases●

Input words one at a time from ASR●

While words exist in phrase table, don't translate yet

Phrase Table
hello→ こんにちは
where is→ どこですか
where→ どこthe→ そのInput Stringhellowhereis
the“hello”“hello where”
phrase exists phrase missing↓↓
wait
translate“hello”“where is”
phrase exists↓wait

the station→ 駅

stationSimple, Lexicalized Choice of Translation Timing for Simultaneous Speech Translation

Method One:
Choosing Translation Timing with Phrases●

Input words one at a time from ASR●

While words exist in phrase table, don't translate yet

Phrase Table
hello→ こんにちは
where is→ どこですか
where→ どこthe→ そのInput String
hello
whereisthe“hello”“hello where”
phrase exists phrase missing↓↓
wait
translate“hello”“where is”
phrase exists↓
wait

the station→ 駅station

Simple, Lexicalized Choice of Translation Timing for Simultaneous Speech Translation

Method One:
Choosing Translation Timing with Phrases●

Input words one at a time from ASR●While words exist in phrase table, don't translate yet

Phrase Table
hello→ こんにちは
where is→ どこですか
where→ どこthe→ そのInput String
hello
whereisthe“hello”“hello where”
phrase exists phrase missing↓↓
wait
translate“hello”

the station→ 駅

station“where is”“where is the”
phrase exists phrase missing↓↓
wait
translate“where is”

Simple, Lexicalized Choice of Translation Timing for Simultaneous Speech Translation

Method One:
Choosing Translation Timing with Phrases●

Input words one at a time from ASR●

While words exist in phrase table, don't translate yet

Phrase Table
hello→ こんにちは
where is→ どこですか
where→ どこthe→ そのInput String
hello
whereisthe“hello”“hello where”
phrase exists phrase missing↓↓
wait
translate“hello”

the station→ 駅station“where is”“where is the”
phrase exists phrase missing↓↓
wait
translate“where is”

Simple, Lexicalized Choice of Translation Timing for Simultaneous Speech Translation

Method One:
Choosing Translation Timing with Phrases●

Input words one at a time from ASR●

While words exist in phrase table, don't translate yet

Phrase Table
hello→ こんにちは
where is→ どこですかwhere→ どこthe→ そのInput String
hello
whereisthe“hello”“hello where”
phrase exists phrase missing↓↓wait
translate“hello”

the station→ 駅station“where is”“where is the” “the station”
phrase exists phrase missing utterance ends↓↓↓
wait
translatetranslate“where is”“the station”

Simple, Lexicalized Choice of Translation Timing for Simultaneous Speech Translation

Problem with Method One●Has the potential to degrade translation accuracy:

Normal phrase-based translation:
こんにちは
Hello,駅はwhere isどこですかthe station

Translation with early timing:
こんにちはHello,駅はthe station

どこですか

where is itSimple, Lexicalized Choice of Translation Timing for Simultaneous Speech Translation

Method Two:
Adjusting Timing with Reordering Probabilities●●First, temporarily choose strings according to method one

Next, if that phrase's right probability exceeds athreshold, actually translate the words in the cache

Example (threshold = 0.8):
hellowhereisthestation“hello”“hello where”“where is” “where is the”
“the station”
phrase exists phrase missing phrase exists phrase missing utterance ends↓↓↓↓↓waitchoose “hello”
waitchoose “where is”
translate↓↓“where isright probability
right probability is the station”
is 0.9 > 0.8
0.6 < 0.8↓
↓translate “hello”
do not translate yet●Threshold 1.0 = traditional, 0.0 = method one

Simple, Lexicalized Choice of Translation Timing for Simultaneous Speech Translation

Problem with Method Two●

LMs are traditionally trained on sentences
This is not appropriate for translating shorter chunks● e.g.: The translator will try to “finish” sentences●

こんにちはHello .

駅はどこですかWhere is the station ? Where is it .

Simple, Lexicalized Choice of Translation Timing for Simultaneous Speech Translation

Method Three:
Language Model Adaptation●Before learning the language model, split the training
data according to the same criterion
Traditional LM Training

Hello, where is the station.
My name is John.
Chunk
Segmentation

Hello,
where isthe station.
My nameis John.

LM Training

Sentence
Level LM

Proposed Method

LM Training

Chunk
Level LM

Simple, Lexicalized Choice of Translation Timing for Simultaneous Speech Translation

ExperimentsSimple, Lexicalized Choice of Translation Timing for Simultaneous Speech Translation

Experimental Setup●Four Types of Experiments:
Japanese-English BTEC Travel Conversation (ja-en)● Japanese-English BTEC with 11+ Words (ja-en 11+)
● English-Japanese BTEC Travel Conversation (en-ja)● French-English WMT News (fr-en)●●Evaluation Measures:●Accuracy
14-ref BLEU for BTEC, 1-ref BLEU for News– Manually-graded acceptability–●Delay (Seconds)

Simple, Lexicalized Choice of Translation Timing for Simultaneous Speech Translation

Accuracy (BLEU)

Result One:
Comparison Across Settings
807060504030
20
100

t=1.0

t=0.002en-jaja-enja-en (11+)
fr-en468101214

Delay (Seconds)●●

Delay decreases in all settings
Better delay/accuracy tradeoff for long sentences,similar languages

Simple, Lexicalized Choice of Translation Timing for Simultaneous Speech Translation

Result Two:
Compare with Pause-based Segmentation
Accuracy (BLEU)

50
4540
35

Proposed
Pause

30
25
20

1

1.5

2

2.5

3

3.5

44.5

Delay (Seconds)●

In faster settings proposed method best

●In slower settings pause-based method bestSimple, Lexicalized Choice of Translation Timing for Simultaneous Speech Translation

Manual Evaluation●Decrease in manual evaluation as well, but lessobvious than evaluated by BLEU

Simple, Lexicalized Choice of Translation Timing for Simultaneous Speech Translation

ConclusionSimple, Lexicalized Choice of Translation Timing for Simultaneous Speech Translation

Conclusion●

Proposed method for choosing timing in speechtranslation using phrase table and reordering model
Considers reordering tendencies across languages● Simple and language independent● Competitive accuracy●●Future work:
Combination of prosodic, reordering, and syntactic cues
● How do we evaluate translations?
●Simple, Lexicalized Choice of Translation Timing for Simultaneous Speech Translation

Thank You!

Simple, Lexicalized Choice of Translation Timing for Simultaneous Speech Translation

Example:サーフィン に いい 場所 を 教え て ください
please tell me a good surfing place

サーフィン に | いい 場所 を 教え て くださいfor surfing | please tell me a good place