NAIST Simultaneous Speech Translation System for IWSLT 2023
Ryo Fukuda1, Yuta Nishikawa1, Yasumasa Kano1, Yuka Ko1, Tomoya Yanagita1,11211
1Kosuke Doi , Mana Makinae , Sakriani Sakti , Katsuhito Sudoh , Satoshi Nakamura
1Nara Institute of Science and Technology, Japan 2Japan Advanced Institute of Science and Technology, Japan

Our Systems
n SimulS2T for En-{De,Ja,Zh}

私 は ペン を 買っ

• Model: Multilingual ST with pretrained models

Read• Encoder: HuBERT with Inter-connection• Decoder: mBART50 decoder layersWrite

Multilingual STPronunciation
Estimation(HuBERT + mBART50)• Training: Two-stage fine-tuning

ワタシハペン

1. Multilingual ST corpora
2. Prefix pairs extracted using Prefix Alignment (PA)Acoustic model+ Vocoder

Incremental TTS• Policy: Local Agreement [Liu+2020]

n SimulS2S for En-Ja

Prefix Alignment [Kano+IWSLT2022]

Effective connection between pre-trained Encoder and DecoderØ Aggregate hidden states from intermediate layers of HuBERT→ Input it to the mBART Decoder

Data augmentation for prefix-to-prefix translation
Ø Extract prefix pairs using offline translation→ fine-tune an offline ST model for SimulST

Target Text
𝑦#HuBERTmBARTDecoder

𝐻$Encoder(L)𝐻!

Length Adaptor…

• Pronunciation Estimation• subword to Japanese
pronunciation (katakana)• LSTM with wait-2• Acoustic model + Vocoder• Tacotron2 and neuralvocoderInter-connection [Nishikawa+INTERSPEECH2023]

Encoder(1)Layer Normalization

CNN Feature Extractor

Source Prefix

Prefix Translation (gloss)

I

私は。(I.)

I bought

私は買った。 (I bought)

I bought a

私は一つ買った (I bought one)

Weighted Sum

Prefix pairs

Inter-connection[(“I”, “私は”),(“I bought pens.”, “私はペンを買った。”)]

Source Utterance 𝑋

Submitted S2T systems on MuST-C v2 tst-COMMON.

n Setup: Local Agreement with 𝑛 = 2 (LA-2), 𝑐ℎ𝑢𝑛𝑘 𝑠𝑖𝑧𝑒 = 200~100 ms
Ø PA-fine-tuned model outperformed an offline baseline

Lang pair chunk size BLEU
En-De
950 ms29.98
En-Ja
840 ms15.32
En-Zh
700 ms22.11• The best En-Ja results came from filtering 73% of prefix pairs⇒ Reducing unbalanced pairs in distant languages is important• Filtering did not work well for En-De and En-ZhAL
19641974
1471

En-DeResults with and w/o Inter-connection.

Ø Inter-connection worked for En-De and En-Ja Model⇒ Shared aggregation weights were
helpful across multiple languages

私はペンを買った

I bought a pen 私はペンを買った (I bought a pen)<bos> , 𝑦! , 𝑦"SimulS2T

Offline translation

En-De En-Ja En-Zh Ave.

HuBERT + mBART 30.47 15.71 25.01 23.73

PA(nonfiltered)w/ Inter-connection 30.89 15.89 24.75 23.84
OfflineØ Fixed-size policy can be a good choice in practice.
• Local Agreement was faster than fixed-size policy in non-computation-aware AL• Local Agreement was slower than fixed-size policy in computation-aware AL

En-JaLocal
AgreementPA(73% filtered)

Fixed-sizepolicy

AL (En-De)

OfflineSummary

SimulS2Sn Setup: ASR_BLEU with Whisper-large
Ø SimulS2S resulted in muchworse compared to S2T En-Ja

PA(nonfiltered)

ComputationAware AL (En-De)

Agreement
matters!

Submitted S2S system on MuST-C v2 tst-COMMON.

ASR_BLEU
9.87• BLEU 15.32 → ASR_BLEU 9.87
• S2S had a character error rate of 28.3%• Japanese TTS has difficulty controlling intonation⇒ Significant room for improvement in the TTS

StartOffset

EndOffset

ATD2495

4135

3279

SimulS2S En-Jan SimulS2T results showed• effectiveness of Prefix Alignment
and Inter-connection.
• superiority of fixed policy incomputation-aware latency.

n SimulS2S results showed• promising performance bysimple cascade of SimulS2T
and incremental TTS.