NAIST Simultaneous Speech-to-Text
Translation System for IWSLT 2022
Ryo Fukuda1, Yuka Ko1, Yasumasa Kano1, Kosuke Doi1, Hirotaka Tokuyama1,Sakriani Sakti1,2, Katsuhito Sudoh1, and Satoshi Nakamura1
1Nara Institute of Science and Technology (NAIST), Japan

2Japan Advanced Institute of Science and Technology (JAIST), Japan

Brief Summary
• IWSLT 2022 Simultaneous Speech Translation task
• Track: Text-to-Text, Speech-to-Text
• Language: English → {German, Japanese, Mandarin Chinese}

• We applied Bilingual Prefix Alignment (Kano et al., 2022) to speech-to-text
• Boundary prediction model with adaptive segmentation policies
• Prefix-to-prefix translation model fine-tuned on bilingual prefix pairs

• Our system did not perform as well as
the other teams but did demonstrate
robustness to low latency

(partial results fromIWSLT 2022 findings)

1

Background
Simultaneous Machine Translation (SimulMT) approaches
• Fixed policy
• Wait-k (Ma et al., 2019; Elbayad et al., 2020), …

• Adaptive policy
• RL-based (Gu et al., 2017), Monotonic attention (Arvazhagan et al., 2019; Maet al., 2020), Prefix Alignment (Zhang et al., 2020, Kano et al., 2022), …

• For simultaneous speech translation (SimulST), adaptive
policies can be more effective than fixed policies
• e.g. fixed policies provide output even when speech is paused
Background

Method

Experiments

Conclusions

2

Bilingual Prefix Alignment (BPA) (Kano et al., 2022)
Extract bilingual prefix pairs and use them to(1) train a boundary prediction model and(2) fine-tune a offline translation model
Read source
words

Boundary
Prediction

translation

私はStep 1

I0.9 > 0.5

Step 2

I bought

0.2 < 0.5

Step 3

I bought a0.3 < 0.5

Step 4

I bought a pen

0.7 > 0.5

私はペンを買った

Step 5

I bought a pen .

0.7 > 0.5

私はペンを買った。

The translation process of SimulMT based on Bilingual Prefix Alignment.
Background

Method

Experiments

Conclusions

3

Bilingual Prefix Alignment (BPA) (Kano et al., 2022)
Extract bilingual prefix pairs and use them to(1) train a boundary prediction model and(2) fine-tune a offline translation model
Read source
words

Boundary
Prediction

translation

私はStep 1

I0.9 > 0.5

Step 2

I bought

0.2 < 0.5

Step 3

I bought a0.3 < 0.5

Step 4

I bought a pen

0.7 > 0.5

私はペンを買った

Step 5

I bought a pen .

0.7 > 0.5

私はペンを買った。

The translation process of SimulMT based on Bilingual Prefix Alignment.
Background

Method

Experiments

Conclusions

4

Bilingual Prefix Alignment (BPA) (Kano et al., 2022)
Extract bilingual prefix pairs and use them to(1) train a boundary prediction model and(2) fine-tune a offline translation model
Read source
words

Boundary
Prediction

translation

私はStep 1I

0.9 > 0.5

Step 2

I bought

0.2 < 0.5

Step 3

I bought a0.3 < 0.5

Step 4

I bought a pen

0.7 > 0.5

私はペンを買った

Step 5

I bought a pen .

0.7 > 0.5

私はペンを買った。

The translation process of SimulMT based on Bilingual Prefix Alignment.
Background

Method

Experiments

Conclusions

5

Bilingual Prefix Alignment (BPA) (Kano et al., 2022)
Extract bilingual prefix pairs and use them to(1) train a boundary prediction model and(2) fine-tune a offline translation model
Read source
words

Boundary
Prediction

translation

私はStep 1I

0.9 > 0.5

Step 2

I bought

0.2 < 0.5

Step 3

I bought a0.3 < 0.5

Step 4

I bought a pen

0.7 > 0.5

私はペンを買った

Step 5

I bought a pen .

0.7 > 0.5

私はペンを買った。

The translation process of SimulMT based on Bilingual Prefix Alignment.
Background

Method

Experiments

Conclusions

6

Bilingual Prefix Alignment (BPA) (Kano et al., 2022)
Extract bilingual prefix pairs and use them to(1) train a boundary prediction model and(2) fine-tune a offline translation model
Read source
words

Boundary
Prediction

translation

私はStep 1I

0.9 > 0.5

Step 2

I bought

0.2 < 0.5

Step 3

I bought a

0.3 < 0.5

Step 4

I bought a pen

0.7 > 0.5

私はペンを買った

Step 5

I bought a pen .

0.7 > 0.5

私はペンを買った。

forced output prefix

The translation process of SimulMT based on Bilingual Prefix Alignment.
Background

Method

Experiments

Conclusions

7

BPA - Stage 1. Extracting Prefix Pairs
Prefix
pairs
Full-sentence translation
Output

私はペンを買った。

私は。
NMT

Input

I𝑥!"
Background

Method

Experiments

Conclusions

8

BPA - Stage 1. Extracting Prefix Pairs
Prefixpairs

[I, 私は]

Full-sentence translation
Output

私はペンを買った。

common prefix
私は。
NMT

Input

I𝑥!"
Background

Method

Experiments

Conclusions

9

BPA - Stage 1. Extracting Prefix Pairs
Prefix
pairs

[I, 私は]

Full-sentence translation

私はペンを買った。

ignore the same
Output

Input

私は。

私は買った。

NMT

NMT

I

I bought𝑥!"
Background

Method

𝑥!#
Experiments

Conclusions

10

BPA - Stage 1. Extracting Prefix Pairs
Prefix
pairs

[I, 私は]

Full-sentence translation
Output

Input

私は。

私は買った。

私は買った。

NMT

NMT

NMT

I bought

I bought a𝑥!#𝑥!$I𝑥!"
Background

私はペンを買った。

Method

Experiments

Conclusions

11

BPA - Stage 1. Extracting Prefix Pairs
Prefixpairs

[I, 私は]

[I bought a pen, 私はペンを買っ]

Full-sentence translation
Output

Input

私は。

私は買った。

私は買った。

私はペンを買って。

NMT

NMT

NMT

NMT

I bought

I bought a

I bought a pen

𝑥!#

𝑥!$𝑥!%

I𝑥!"
Background

私はペンを買った。

Method

Experiments

Conclusions

12

BPA - Stage 1. Extracting Prefix Pairs
Prefixpairs

[I, 私は]

[I bought a pen, 私はペンを買っ]

Full-sentence translation
Output

Input

私は。

私は買った。

私は買った。

私はペンを買って。

私はペンを買った。

NMT

NMTNMT

NMT

NMT

I bought

I bought aI bought a pen

I bought a pen .

𝑥!#𝑥!$𝑥!%

𝑥!&I𝑥!"
Background

私はペンを買った。

Method

Experiments

Conclusions

13

BPA - Stage 2. Training of Models
(1) Train a boundary prediction model with prefix-{0,1} label pairs
• [I, 1]
• [I bought, 0]
• [I bought a , 0]
• [I bought a pen , 1]
• [I bought a pen. , 1]

(2) Fine-Tune a offline translation model with prefix pairs
• [I, 私は]
• [I bought a pen, 私はペンを買っ]

Background

Method

Experiments

Conclusions

SimulST based on Bilingual Prefix Alignment
Applying Bilingual Prefix Alignment to SimulST
Read sourcewords

Boundary
Prediction

translation

私は

Step 1

I

0.9 > 0.5

Step 2

I bought

0.2 < 0.5

Step 3

I bought a

0.3 < 0.5

Step 4

I bought a pen

0.7 > 0.5

私はペンを買った

Step 5

I bought a pen .

0.7 > 0.5

私はペンを買った。

• SimulST takes the source language speech as input
• no word unit
• extremely long sequences
Background

MethodExperiments

Conclusions

15

Extracting Speech Prefix Pairs
Prefixpairs

[I, 私は]

[I bought a pen, 私はペンを買っ]

Full-sentence translation
Output

Input

私は。

私は買った。

私は買った。

私はペンを買って。

私はペンを買った。

NMT

NMT

NMT

NMT

NMT

I bought

I bought aI bought a pen

I bought a pen .

𝑥!#

𝑥!$

𝑥!%

𝑥!&

I𝑥!"
Background

私はペンを買った。

Method

Experiments

Conclusions

16

Extracting Speech Prefix Pairs
Prefixpairs

[, 私は]

[

Full-sentence translation
Output

, 私はペンを買っ]

私はペンを買った。

私は。

私は買った。

私は買った。

私はペンを買って。

私はペンを買った。

NMT

NMT

NMT

NMT

NMT

𝑥!'

𝑥!#'

𝑥!$'

𝑥!%'

𝑥!&'Input

Background

Method

Experiments

Conclusions

𝜏: step size (number of frames)

17

Data filtering of Prefix Pairs
Unbalanced prefix pairs were sometimes extracted
• pairs of a long source speech prefix and a short target text prefix

.. frequently appear between distant language pairs such as English and Japanese
• e.g. {English prefix, Japanese prefix} would consist of {S, S}, {SV, S}, {SVO, SOV}
300 f

400 f

500 f!!"#$%&

She took our order, and then went to the couple in the booth next to us,#$!"#$%&and shelowered hervoice ...

▁ 注文
▁ 注文 を
▁ 注文 を 取った あと ▁ 隣 の ブー ス で カップル

#$'$$(%)# ▁ 注文 を 取った あと ▁ 隣 の ブー ス で カップル に会い ました ▁彼女は 自分の 声を あまり にも 軽 蔑 しました...

we removed such unbalanced pairs by the length ratio
• exclude if source prefix length / target prefix length > 𝑚𝑎𝑥𝑟𝑎𝑡𝑖𝑜
Background

Method

Experiments

Conclusions

18

Boundary Predictor
• Training with pairs of a speech prefix and a corresponding binary label
sequence
• {
• {
• {
• {
• {

,

[1..1]}
,
[1..10..0]}
,
[1..10..00..0]}
,
[1..10..00..01..1]}
, [1..10..00..01..11..1] }

• The boundary predictor is trained with weighted cross-entropy loss
• The boundary predictor predicts a boundary in every 𝜏 frames

• WRITE if the proportion of label 1 is larger than or equals to 𝜆!"#$ , otherwise READ

Background

Method

Experiments

Conclusions

19

Experimental setting
• Data: MuST-C v2
• contained about 250k segments for En-De and 330k for En-Ja

• Models
• Speech Translation: 12 encoder + 6 decoder Transformer layers
• Boundary prediction: a 2D-convolution layer, a unidirectional
LSTM layer, and an output linear layer
• decision size: 𝜏 = 100 frames

• Evaluated on MuST-C v2 tst-COMMON using SimulEval
Background

Method

Experiments

Conclusions

20

En-De Results

• a model fine-tuned with prefix pairs (FT) worked better than the non fine-tuned model(w/o FT) in the range of AL ≤ 4,000
• robust to lower latency
• non fine-tuned model worked better than the fine-tuned model in high latency
Background

Method

Experiments

Conclusions

21

En-Ja Results

• non fine-tuned model worked better than wait-k baselines
• large improvements in the low latency ranges
• fine-tuned model were worse than those of wait-k and non fine-tuned model
• unbalanced prefix pairs degraded the performance
Background

Method

Experiments

Conclusions

22

Data filtering for En-Ja

• the model fine-tuned with all prefix pairs (None) preferred too short outputs
• the model fine-tuned with filtered data on maxratio = 20 (filter20) significantly improved
the performance and outperformed non fine-tuned model
Background

Method

Experiments

Conclusions

23

Conclusions
• We described our SimulST systems in English-to-German andEnglish-to-Japanese.
• We used Bilingual Prefix Alignment to
• train boundary predictor that judges when to READ and WRITE and• fine-tune the offline speech translation model.

• Our system achieved some improvements compared to the waitk baselines in every latency regime
• especially, our system was robust to lower latency
• data filtering was important for En-Ja because unbalanced prefix pairs
frequently appeared due to differences in sentence structures
BackgroundMethodExperiments

Conclusions

24