音声翻訳研究の現状と今後
Satoshi Nakamura
WithKatsuhito Sudo and AHC ST-TeamNara Institute of Science and Technology (NAIST), Japan
S-nakamura@is.naist.jp
Part of this work is supported by JSPS KAKENHI (Grant numbers JP21H05054, 21H03500, 17H06101)

My Speech Translation Research
21992 NAIST

1986
ATR

ATR

ReadSpeech

Daily
Conversation• Syntactically correct• Clear utterance• Limited domain
Ex. “ConferenceRegistration”

1999

2006

2011

2010

Wider andReal Domain+ More languages
for translation•21 multilateral text translation• Standard expression• Unclear utterance• Limited domain
Ex. “Hotel Reservation”• Wider and real domain“International Travel”
A-STAR• Realistic expressions• Noisy speech• J-E, J-C speech translation • Multilateral translation for 8 Asian languages• Network-based S2ST

• Multilateral translation for 7 world languages

Rule-based
MachineTranslation

NAIST

2023

NICT

ATR

C-STAR

Approach2008

Simultaneous
Speech-to-speech
Translation@IWSLT

IWSLT• Evaluation Campaign of S2S technologies

Statistical
MachineTranslation

NeuralMachineTranslation

Satoshi Nakamura, AAMT Talk (November 29th, 2023)

2023/11/27

Simultaneous Incremental Speech Translation

3Delay: ReducedASR
こんにちは、駅はkonnichiwa ekiwa

MT

どこですか？
dokodesuka

MTMTHello,the station where is it?

TTSTTSTTS

But, this is Satoshi
notNakamura,easy!
AAMT Talk (November 29th, 2023)

2023/11/27

Challenge in En-Ja Simul MT4

 Long distance reordering

Source:(1) The relief workers (2) say (3) they don’t have (4) enough food, water,
shelter, and medical supplied (5) to deal with (6) the gigantic wave ofrefugees (7) who are ransacking the countryside (8) in search of the basics(9) to stay alive.

Example from Mizuno, A. “Simultaneous Interpreting and Cognitive Constraints,”
Journal of College of Literature, Aoyama Gakuin University, vol. 58, pp. 1–28, 2016

Typical Challenge in En-Ja Simul MT

5Source:(1) The relief workers (2) say (3) they don’t have (4) enough food, water, shelter, andmedical supplied (5) to deal with (6) the gigantic wave of refugees (7) who areransacking the countryside (8) in search of the basics (9) to stay alive.
Target:(1) kyuuen tan’tousha wa (9) ikiru tame no (8) shokuryo o motomete (7) mura o arashi
mawatte iru (6) tairyo no nan-min tachi no (5) sewa o suru tame no (4) juubun na
shokuryo ya mizu, shukuhaku shisetsu, iyakuhin ga (3) nai to (2) itte imasu.

Example from Mizuno, A. “Simultaneous Interpreting and Cognitive Constraints,”
Journal of College of Literature, Aoyama Gakuin University, vol. 58, pp. 1–28, 2016

Interpreters’ monotonic translation

6Source:(1) The relief workers (2) say (3) they don’t have (4) enough food, water, shelter, andmedical supplied (5) to deal with (6) the gigantic wave of refugees (7) who areransacking the countryside (8) in search of the basics (9) to stay alive.
Target:(1) kyuuen tan’tousha tachi no (2) hanashi de wa, (4) shokuryo, mizu,shukuhaku shisetsu, iyakuhin ga (3) tari zu (6) tairyo no nan-min tachi no(5) sewa ga dekinai to no koto desu.
(7) nan-min tachi wa ima muramura o arashi mawatte, (9) ikiru tame no(8) shokuryo o motomete iru no desu.
Example from Mizuno, A. “Simultaneous Interpreting and Cognitive Constraints,”
Journal of College of Literature, Aoyama Gakuin University, vol. 58, pp. 1–28, 2016

Translation vs InterpretationTranslation

Necessary #Chunk＞３！

Interpretation

7

Necessary #Chunk＜３！

Today’s contents
 Background Cascaded Simultaneous Speech-to-speech Translation (IWSLT2022)
 End-to-end ST and incremental TTS Speech-to-speech Translation(IWSLT2023)
 NAIST Simultaneous Speech Interpretation Corpus Recent Progress Paralinguistic Conversion in STST End-to-end Speech-to-speech Translation (TRANSLATOTRON 3)

 Summary8

Cascade Simultaneous S2S Translation System
Input English
speechI am a chef and afood policy guy,
but I come from ...

Incremental
ASR
(ISR)

Intermediate
English transcript

I was a chef and afood policy guy,
but I come from ...

Output Japanese
speechIncremental
MT(IMT)

9

私はシェフで、食べ物政策の人…

Incremental
TTS(ITTS)

私はシェフで、食べ Intermediate
Japanese MT result
物政策の人…

2023/11/27

Translation Timing Control by Syntactic Prediction in SMT
Syntax Prediction Process(2015,2021)
 Syntactic Prediction [Oda, et al., 2015]

1. Parse the input as-is Feature extraction and syntactic predictionINDTJJal parsing
andsyntactic
prediction
MT results Use LSTM and BERT to predict next tag.

[Kano, et al., 2021]

NNSNNI

ROOT=PPROOT-L=IN
ROOT-R=NP...

3. Predict the next tag(linear SVM)
VP ... 0.65
NP ... 0.28
4. Append tonil ... 0.04
sequence
...

VPnil

5. Repeat until nil

Input translation unit

 Wait for MT output when specific labels appear. Increment

reordering

CDNP

in the next 18 minutes

10/16/2016

 Control MT output timing according

Word:R1=I
POS:R1=NN
Word:R1-2=I,minutes
POS:R1-2=NN,NNS...

NPNP

Subject Verb Object : English
Subject Object Verb: Japanese

2. Extract features

PP Incremental bottom up parsing

10‘m going to take you on ajourney

Invited Talk ©Satoshi Nakamura, NAIST

46

in the next 18 minutes[NP]predict [VP] (wait)
i 'm going to take [keep]
i 'm going to take you on a journey [VP end]
18 分 で あ る[NP][VP](wait) を 行 っ て い ま す[keep]
皆さん を 旅 に お連れ します [VP end]

Oda, Yusuke et al., Syntax-based Simultaneous Translation through Prediction of Unseen Syntactic Constituents, Proc. of ACL-IJCNLP 2015.
Y.Kano, K.Sudo, S.Nakamura, “Simultaneous Neural Machine Translation with Constituent Label Prediction”, Proc. of WMT 2021.
2023/11/27
Satoshi Nakamura, AAMT Talk (November 29th, 2023)

Incremental MT: Wait-k Algorithm [M.Ma, et al., 2018]11
Encoder-decoder NN

ブッシュ

大統領

は今日withPutintoday.

プーチンとWait-3

President

Bushmeets

Mingbo Ma, et al., “STACL: Simultaneous Translation with Integrated Anticipation and Controllable Latency”,
arXiv:1810.08398v3 [cs.CL] 3 Nov 2018
Satoshi Nakamura, AAMT Talk (November 29th, 2023)

2023/11/27

NAIST IWSLT 2022 System

Input English
speechIntermediate
English transcript
Incremental
ASR(ISR)

Attention Transfer
ISR (A-LSTM)

12Intermediate
Japanese MT result
Incremental
MT(IMT)

Wait-k IMT

Output Japanese
speechIncremental
TTS(ITTS)

Attention Transfer
ITTS (A-LSTM)

2023/11/27

Incremental MT by Boundary Prediction↑ Forced decoding

13

The other translation Timing Control by Syntactic Prediction[Kano, et al, 2021]

14

ICLP: Incremental Constituent Label Prediction (LSTM, BERT)

Y.Kano, K.Sudo, S.Nakamura, “Simultaneous Neural Machine Translation with Constituent Label Prediction”, Proc. of WMT
2021.
2023/11/27
Satoshi Nakamura, AAMT Talk (November 29th, 2023)

Incremental Speech Recognition (ISR) [Novitasari, 2020]

15

S.Novitasari, A.Tjandra, T.Yanagita, S.Sakti, S.Nakamura, “Incremental Machine Speech Chain Towards Enabling Listening
while Speaking in Real-time”, Proceedings of INTERSPEECH 2020, Oct. 2020
Satoshi Nakamura, AAMT Talk (November 29th, 2023)

2023/11/27

Incremental Speech Synthesis (iTTS)

16

S.Novitasari, S.Sakti, S.Nakamura, “Dynamically Adaptive Machine Speech Chain Inference for TTS in Noisy Environment:
Listen and Speak Louder”, Proc. Interspeech 2021, 4124-4128, Aug. 30, 2021
Satoshi Nakamura, AAMT Talk (November 29th, 2023)

2023/11/27

Video172023/11/27

How it works
18
Source speech spectrogram (English)

delay

Target speech spectrogram(Japanese)

Satoshi Nakamura, AAMT Talk (November 29th, 2023)

Results
 SimulMT model trained using

the prefix pairs outperformed
other methods (En-De)
 The advantage is smaller inEn-Ja; PA failed to induceenough short prefix pairs thathelps SimulMT

19

Today’s contents
 Background Cascaded Simultaneous Speech-to-speech Translation (IWSLT2022)
 End-to-end ST and incremental TTS Speech-to-speech Translation(IWSLT2023)
 NAIST Simultaneous Speech Interpretation Corpus Recent Progress Paralinguistic Conversion in STST End-to-end Speech-to-speech Translation (TRANSLATOTRON 3)

 Summary20

2023 SimulS2S System [Fukuda+ 2023 IWSLT]
Input English
speech

Output Japanese
speechIncremental

IncrementalST(IST)No explicit
Englishtranscriptions

TTS(ITTS)

私はシェフで、食べ物政策の人…

Intermediate Japanese MT
result21

IWSLT NAIST 2023 SYSTEMIWSLT 2022 STST System

Incremental ASR(Transformer)

Incremental MT(Atten. ED+Wait-k)

Incremental TTS(Transformer)

22

IWSLT 2023 STST System

SpeechEncoder(Hubert)Multilingu
al MBART(Wait-k)

embedding
VectorSequence

Incremental TTS(Transformer)

Sub word
SequenceVideo (En-Ja, speech-to-speech)23

System Structure24Fine-tuning by Prefix Alignment [Kano+ 2022 IWSLT]
Full-sentence
Output

私は ペンを買った(1) FT data
[ , 私は][ , 私はペンを買っ]common
target prefix(2) BP data
Part Output
Offline STPart Input

私は私はExtracting
prefix pairs[[[[[, 1..1], 1..10..0], 110..00..0], 1..10..00..01..1], 1..10..00..01..11..1]

25

Local Agreement: Chunk-wise Inference with stable hypothesis
26
Incremental decoding described by Liu et al., (2020)
1. Audio is split to chunks

2. After the system receives a new chunk,it starts with forced-decoding of
the stable hypothesis and thencontinues with beam search

Danni Liu, Gerasimos Spanakis, and Jan Niehues. 2020. LowLatency Sequence-to-Sequence Speech Recognition and Translation
by Partial Hypothesis Selection. In Proc. Interspeech 2020.

Results
Lang pairchunk size

BLEU

AL

En-De

950 ms

29.98

1964

En-Ja

840 ms

15.32

1974

En-Zh

700 ms

22.11

1471

(EN-DE)

27(EN-DE)

Computation-aware AL

IWSLT Evaluation Campaign – SimulST track History 2020: text/speech-to-text, En-De
 2021-2022: text/speech-to-text, En-De/Ja/Zh 2023: speech-to-text/speech, En-De/Ja/Zh We’re planning the next edition (2024)

 Regulations
 Use publicly-available speech/language resources Configure SimulST systems to satisfy given latency limits
 Submit systems in form of Docker images

28

Today’s contents
 Background Cascaded Simultaneous Speech-to-speech Translation (IWSLT2022)
 End-to-end ST and incremental TTS Speech-to-speech Translation(IWSLT2023)
 NAIST Simultaneous Speech Interpretation Corpus Recent Progress Paralinguistic Conversion in STST End-to-end Speech-to-speech Translation (TRANSLATOTRON 3)

 Summary29

NAIST-SIC (Simultaneous Interpretation Corpus)
 A collection of Simultaneous Interpretation
 https://dsc-nlp.naist.jp/data/NAIST-SIC/
 You can (easily) find by searching “NAIST-SIC”
 A part of this corpus (NAIST-SIC 2021) was used for IWSLT SimutaneousTranslation shared task
 It was also presented at IWSLT 2021
 Doi et al., Large-Scale English-Japanese Simultaneous Interpretation Corpus:

Construction and Analyses with Sentence-Aligned Data, Proc. IWSLT 2021.

 An additional release (NAIST-SIC 2022) includes automatic source-target sentencealignment

30

Corpus Development
 We collected SI recordings ~10 years in our government-funded projectand released a part of them: NAIST-SIC 2014: En-Ja/Ja-En
 NAIST-SIC 2021: En-Ja (TED Talks), 57 SI transcripts on 28 talks
 NAIST-SIC 2022: En-Ja (TED Talks), 833 SI transcripts on 831 talks with automatic

sentence alignment
 We’re planning to release remaining portions (esp. Ja-En) by March 2024
 Our recordings reach 340 hours in total

31

Using SI Corpus for SimulST

32
BLEU

 A naïve use of the corpus was not so effective
SI ref.

Offline ref.

Offline finetune

7.8

16.0

 Mixed use of offline translation

SI fine-tune

10.96.3

and SI (Ko+ 2023 IWSLT)

Mixed finetune

9.4

13.3

Proposed

10.3

15.4

Proposed +up-sampling SIdata

12.2

14.2

 Reported in our preprint: NAIST-SIC-Aligned(arXiv: 2304.11766)

 Introducing style tags <off> <si>to distinguish data sources Used to fine-tune pre-trained ST models

Training step

Tgt Ja text (Offline+SI)

Source En Speech
Pretrained STModel
I bought a pen<si>私は、買った。ペンを、<si>…..
<off>私はペンを買いました<off> …
<off> …

Today’s contents
 Background Cascaded Simultaneous Speech-to-speech Translation (IWSLT2022)
 End-to-end ST and incremental TTS Speech-to-speech Translation(IWSLT2023)
 NAIST Simultaneous Speech Interpretation Corpus Recent Progress End-to-end Speech-to-speech Translation (TRANSLATOTRON 3)
 Paralinguistic Conversion in STST Summary33

Speech Translation and its Limitation34
 What is it?
 Translates speech in one language into text/speech in another Limitation
 Unable to consider paralinguistic info
 If the linguistic info is the same, so are the translations.

is lost Could cause misunderstandings between a speaker and a hearer

Paralinguistic Information in S2ST
“It is hot today”

English“今日は熱いです”

ASRMTESET

Source emphasis
information

Target emphasis
information.

(1) Emphasis estimation (ES) systems:(2) Emphasis translation (ET) systems:

35
JapaneseTTS

by speech and word sequenceby mapping emphasis to the target language.

Satoshi Nakamura, AAMT Talk (November 29th, 2023)

2023/11/27

Paralinguistics Conversion II36

 Paraphrase focused speech into focused text
 Mappingintowithin the SL

 Translating the paraphrased text into the TL

 Could return different translations depending on the focused items
N.Suzuki, S.Nakamura, “Representing ‘how you say’ with ‘what you say’:
English corpus of focused speech and text reflecting corresponding implications”, INTERSPEECH2022

Summary Recent Speech-to-speech Translation Systems
 Cascaded Simultaneous Speech-to-speech Translation (IWSLT2022)
 End-to-end ST and incremental TTS Speech-to-speech Translation (IWSLT2023)

 NAIST Simultaneous Speech Interpretation Corpus Recent Progress Paralinguistic Conversion in STST37

Future Directions
 Analysis of the interpretation data More practical and efficient Simul-SST
 Beyond the use of offline translation data
 Effective use of the interpretation data

 Sophisticated quality and latency assessment for Simul-SST
 Applicable to SI and human interpretation

 Pretrained LLM and fine-tuning Summarization for text and output speech Paralinguistic/nonlinguistic information in Simul-SST38

39

Thank you for your attention!