NAIST English-to-JapaneseSimultaneous Translation System

for IWSLT 2021 Simultaneous Text-to-text Task
Ryo Fukuda, Yui Oka, Yasumasa Kano, Yuki Yano,Yuka Ko, Hirotaka Tokuyama, Kosuke Doi,
Sakriani Sakti, Katsuhito Sudoh, Satoshi Nakamura
Nara Institute of Science and Technology (NAIST)Japan
This work is supported by JSPS KAKENHI (Grant number 17H06101).

Another Conclusion
Techniques
Results
Overview wait-k Seq.
KD Chunk shuf. Setup Dev results Summary attempt

Overview of Our Submissions
Latency regimeLow (AL ≤ 8)

Medium (AL ≤ 12)

Base system

wait-k (Ma+ 2019) on Fairseq (Ott+ 2019) & SimulEval (Ma+ 2020)
w/ Transformer-base settings

Data

Training: WMT20 News & IWSLT17 train / Dev: IWSLT17 dev
Shared BPE vocabulary (16,000)

2High (AL ≤ 16)

Latency
hyperparameterk = 10k = 20Additional
training-timefeatureTarget-side
chunk shuffling

Seq. Knowledge
Distillation froman offline model

BLEU/AL (dev)

13.77 / 7.29

15.22 / 11.48

15.57 / 13.70

BLEU/AL (test)

14.41 / 7.21

16.20 / 11.54

16.19 / 13.83

IWSLT 2021 (Simultaneous Translation Task)

k = 30Another Conclusion
Techniques
Results
Overview wait-k Seq.
KD Chunk shuf. Setup Dev results Summary attempt

wait-k prefix-to-prefix translation(Ma+ 2019)3• Wait for k tokens first and thengenerate outputs concurrently withinputs
ブッシュ ⼤統領は
今⽇ プーチン とWait-3

President Bush

meets

with

Putin

today

IWSLT 2021 (Simultaneous Translation Task).

Another Conclusion
Techniques
Results
Overview wait-k Seq.
KD Chunk shuf. Setup Dev results Summary attemptSeq.-level Knowledge Distillation(Kim and Rush, 2016)• Train simultaneous (student) NMT usingoutputs from offline (teacher) NMT
Parallel corpus for training

English(source)

Japanese

Simultaneous NMT model(Transformer; wait-k)(target)Pseudo-parallel corpus

Offline NMT model(Transformer)

English(source)IWSLT 2021 (Simultaneous Translation Task)

Japanese(MT results)

4Another Conclusion
Techniques
Results
Overview wait-k Seq.
KD Chunk shuf. Setup Dev results Summary attempt

Why Seq.-level KD?

5• Colloquial expressions and free (nonliteral) translation in Japanese subtitles• Difficult to generate for NMT…

• Literal translation by NMT• Would be easy to generate for NMT

Parallel corpus for training

English(source)

Japanese

Simultaneous NMT model(Transformer; wait-k)(target)Pseudo-parallel corpus

Offline NMT model(Transformer)

English(source)

Japanese(MT results)

Motivated by recent
non-autoregressive NMT
Mitigate non-parallelism
in the training dataIWSLT 2021 (Simultaneous Translation Task)

Another Conclusion
Techniques
Results
Overview wait-k Seq.
KD Chunk shuf. Setup Dev results Summary attempt

Target-side Chunk Shuffling (Cshuf)• Reorder Japanese text chunks randomly
in the training data
• With a small probability 𝑝! (1 to 3%)• Chunk size k is fixed and set as the sameas latency hyperparameter for wait-k
Japanese sentence in the training data (tokenized)
𝑡! , 𝑡" , ⋯ , 𝑡#𝑡!"# , 𝑡!"$ , ⋯ , 𝑡$!

𝑡$!"# , 𝑡$!"$ , ⋯ , 𝑡%! 𝑡%!"# , 𝑡%!"$ , ⋯ , 𝑡&!

Chunk shuffling𝑡#$! , 𝑡#$" , ⋯ , 𝑡"# 𝑡%!"# , 𝑡%!"$ , ⋯ , 𝑡&!

𝑡! , 𝑡" , ⋯ , 𝑡#

IWSLT 2021 (Simultaneous Translation Task)

𝑡$!"# , 𝑡$!"$ , ⋯ , 𝑡%!

6

Another Conclusion
Techniques
Results
Overview wait-k Seq.
KD Chunk shuf. Setup Dev results Summary attempt

Why Chunk Shuffling?
• Rough simulation of chunk reordering
allowed in Japanese• The order of Japanese chunks (bunsetsu;⽂節) is not so strict• CShuff encourages order-free outputs inthe training, which would help
monotonic translation using wait-k• More linguistically-motivated reordering
would be worth trying in future work…
IWSLT 2021 (Simultaneous Translation Task)

7

Another Conclusion
Techniques
Results
Overview wait-k Seq.
KD Chunk shuf. Setup Dev results Summary attempt

System Setup• wait-k on fairseq & SimulEval (based onthe official baseline)• Transformer-base• BPE-based subwords with voc. size of
16,000, shared btw. En-Ja• Training: 17.9M from WMT20, 223K
from IWSLT17 train• Fine-tuning: IWSLT17 dev (fine-tune)• Dev-test: IWSLT21 dev (dev-test)
IWSLT 2021 (Simultaneous Translation Task)

8

Another Conclusion
Techniques
Results
Overview wait-k Seq.
KD Chunk shuf. Setup Dev results Summary attempt

IWSLT21 dev results: Seq.-level KD• BLEU improvements with small k18
1618

BLEU

14k=1016121214k=101412182020242226 28 30 32
28 3024223226

1610

wait-kwait-k+SKD

86681012

Average Lagging
IWSLT 2021 (Simultaneous Translation Task)

1416

9

Another Conclusion
Techniques
Results
Overview wait-k Seq.
KD Chunk shuf. Setup Dev results Summary attempt

IWSLT21 dev results: CShuf• Worked the best with 𝑝! = 0.02

• Very sensitive with 𝑝! , in output length

Baseline (wait-10)

Target-side ChunkShuffling (Cshuf)

𝑝$BLEU

Length ratio

011.80

1.233

0.01

10.57

1.372

0.0213.77

1.053

0.03

9.87

1.516

IWSLT 2021 (Simultaneous Translation Task)(hyp / ref)10

Another Conclusion
Techniques
Results
Overview wait-k Seq.
KD Chunk shuf. Setup Dev results Summary attempt

Results Summary

11• Mid-latency system worked (relatively)well, but still 2 pts. behind other teams
SystemIWSLT21 dev
BLEU
AL

Offline

16.80

---

wait-10 +Cshuf

13.77

7.29

14.41

7.21

15.22

11.48

16.20

11.54

15.57

13.70

16.19

13.83

(low latency)

wait-20 +Seq.KD(medium latency)

wait-30(high latency)

IWSLT21 test (Official)
BLEU
AL

IWSLT 2021 (Simultaneous Translation Task)

Another Conclusion
Techniques
Results
Overview wait-k Seq.
KD Chunk shuf. Setup Dev results Summary attempt

Another attempt

12• The use of future syntax information• Motivated by Oda et al. (2015) that used
it to determine when to start translationin simultaneous SMT• Next Constituent Label Prediction (NCLP)• 1-lookahead prediction• BERT-based classifier

SBAR

Those of us who … Prediction
Observation

IWSLT 2021 (Simultaneous Translation Task)

Another Conclusion
TechniquesResultsOverview wait-k Seq.
KD Chunk shuf. Setup Dev results Summary attempt

wait-k with NCLP

13

• Enhance inputs with predicted labels• The length of an input is *doubled*Those of us who are underrepresented …
Those
Those PP ofThose PP of NP us
Those PP of NP us SBAR who
Those PP of NP us SBAR who SQ areThose PP of NP us SBAR who SQ are VP under...
IWSLT 2021 (Simultaneous Translation Task)

Another Conclusion
Techniques
Results
Overview wait-k Seq.
KD Chunk shuf. Setup Dev results Summary attempt

Results of wait-k with NCLP• No advantages were observed…
1816wait-k
wait-k+NCLP

BLEU

1432

1214k=10

40

60

64

48

2844

221485652

24

1816161220

103618 20

26 283022 24
32

12k=1060

246810

Average Lagging
IWSLT 2021 (Simultaneous Translation Task)

1214

16

14

Another Conclusion
Techniques
Results
Overview wait-k Seq.
KD Chunk shuf. Setup Dev results Summary attempt

Conclusions• Some improvements in En-Ja
simultaneous translation by:• Sequence-level Knowledge Distillation• Encouraging literal translation• Target-side Chunk Shuffling• Encouraging order-free translation• NCLP did not work by the current way• Further investigation needed
IWSLT 2021 (Simultaneous Translation Task)

15