CJNLP 2023

NLP/SLP research activity in
NAIST AHC Lab.
Katsuhito Sudoh
Associate Professor
Nara Institute of Science and Technology (NAIST), Japan
Part of this work is supported by JSPS KAKENHI (Grant numbers JP21H05054, 21H03500, 17H06101)

Simultaneous speech translationand its evaluation
Y. Kano+, “Simultaneous Neural Machine Translation with Prefix Alignment,” IWSLT 2022
R. Fukuda+, “NAIST Simultaneous Speech-to-speech Translation System for IWSLT 2023,” IWSLT 2023
Y. Kano+, “Average Token Delay: A Latency Metric for Simultaneous Translation,” Interspeech 2023
CJNLP 2023 (November 11th, 2023)

CJNLP 2023 (November 11th, 2023)

IMTdelay
ITTS delay (start speaking)

Our SimulS2S System [Fukuda+ 2023 IWSLT]

d a food policy guye (sec)Smallatency
ISR
IMT
ITTS

MediumLatency
ISR
IMT
ITTS

High
Latency
ISR
IMT
ITTS

0

but i come from a whole family teacher

Input Englishspeech

12| i was a chef

3

4

6

78

but i come from a whole family teacher| watashi ha shefu de tabemono seisaku no hito

Incremental

ISR delay| watashi ha shefu de ari tabe mono seisaku no otoko
Output Japanese| watashi wa shefu de ari tabemono seisaku no ospeech
IMTdelay

5and a food policy guy
| i was a chef

ISR delay

No explicit
Englishtranscriptions| i was a chef

ISR delay

ST(IST)and a food policy guy

12| tabemono seisaku no hito deshita ga watashi wa kyoushi no kazoku z

Incremental

family of teachers

ITT
| ryouri nin de ari syoku seisaku no o

ITTS delay (start speaking)but i come from a whole family teacher


TTS(ITTS)
IMT delay| watashi ha shefu de ari tabe mono seisaku no otoko deshita ga watashi ha kazoku no kyous| ryouri nin de ari syoku se| watashi wa shefu de ari tabemono seisaku no otoko deshita ga watashi wa kazoku noIMTdelay

ITT

ITTS delay (start speaking)and a food policy guy

11

deshita ga watashi ha kyoushi no kazoku zeiin kara kimashita 


| watashi ha shefu de|but i come from a whole

10

ITTS delay (start speaking)

IMTdelay

and a food policy guy

9

私はシェフで、⾷べ物
ITTS delay (start speaking)but i come from a whole family of teachers
| ryouri nin de ari syoku seisaku no otoko deshita ga kyoushi no ikka
政策の⼈…

Intermediate Japanese MT result| ryouri nin de ari syoku seisaku no otoko deshita ga kyoushi no i

IMT delay

ITTS delay (start speaking)CJNLP 2023 (November 11th, 2023)

Prefix Alignment [Kano+ 2022 IWSLT]• Induce prefix-to-prefix translation pairs from parallel corpora
Sentence pair

I bought a pen.
僕はペンを買った。

Full-sentence
MT (pretrained)

Source prefix

Prefix translationFull-sent. MT (from n-best) Matched prefixI

僕は僕はペンを買いました。 僕はI bought

僕は買った

僕はペンを買った。

I bought a

僕は買った

僕はペンを買った。

I bought a pen 僕はペンを買った

僕はAnペンを買った。
induced target prefix僕はペンを買った
is used as a

I bought a pen. 僕はペンを買った。 僕はペンを買った。
僕はペンを買った。
fixed (forced-decoded)
target prefix
CJNLP 2023 (November 11th, 2023)

Results• SimulMT model trainedusing the prefix pairs
outperformed othermethods (En-De)• The advantage issmaller in En-Ja; PA
failed to induce enoughshort prefix pairs thathelps SimulMTCJNLP 2023 (November 11th, 2023)

IWSLT Evaluation Campaign – SimulST track• History• 2020: text/speech-to-text, En-De• 2021-2022: text/speech-to-text, En-De/Ja/Zh• 2023: speech-to-text/speech, En-De/Ja/Zh• 2024: TBA• Regulations• Use publicly-available speech/language resources• Configure SimulST systems to satisfy given latency limits• Submit systems in forms of Docker images
CJNLP 2023 (November 11th, 2023)

Automatic Evaluaiton in IWSLT 2023

Speech-to-text

QualityLatency

BLEU

Average Lagging (AL) and variants (LAAL, DAL)
Average Propotion (AP)Average Token Delay (ATD)

ASR-BLEU
Speech-to-speech
BLASER

Start/End Offset
Average Token Delay (ATD)*Please find more details in the overview paper (archived in ACL Anthology)
CJNLP 2023 (November 11th, 2023)

Average Lagging (AL)• Average delay from the ideal policy through the diagonalline in a read-write chartg. 3. More
More

Target→ Target→

Source→Source→Source→(6)
Source→|}(6)Target→ Target→|x|)ex- is exison,step on,
heremainain7 118 129 13
10 11 12 13
1 2 3 41 52 63 74 85 96 107 8 9 101 2 3 41 52 63 74 85 96 10
generated
ated*Figures are from Ma+ (2019, ACL)CJNLP 2023 (November 11th, 2023)

Unintuitive Latency Measurement by AL• Latency can be negative for outputs with long chunks
Target (write) →Target (write) →Source (read) →Source (read) →CJNLP 2023 (November 11th, 2023)

Average Token Delay (ATD) [Kano+ 2023 Interspeech]• Inspired by Ear-Voice Span (EVS) in interpretation studies
Time →Target (write) →s1

Source (read) →

s2

s3

s4

s5s6

s7

Chunk1 from s1-2

t1

t2

t3

t4

t5

Chunk2 from s3s8s9

t6

Chunk3 from s4

t7

Chunk4 from s5

t8

Chunk5 from s6-9

CJNLP 2023 (November 11th, 2023)

t9

Average Token Delay (ATD) [cont’d]• Measure the diff. btw. the end of input chunk and thecorresponding output tokens
s1 s2 s3 s4 s5 s6 s7 s8 s9• Delays are always positive• The largest difference from AL• AL ignores the output duration• ATD takes it into account;• A long outputs can cause further
delay in later outputs• Differences are measured by # tokens (text)
or actual time (speech)

s1

s2

s1

s2

s1

s2

s1

s2

s1

s2

CJNLP 2023 (November 11th, 2023)

t1
t2
t3
t4
t5
s3

t6
s4

t7
s5

t8
s6

s7

s8

s9t9

Results in Computation-aware Latency• In real situations, SimulST systems pose some delays due totheir computations
5

6

Average Token Delay (sec.)

Average Lagging (sec.)

4.5
4
3.5
3HW-TSC

NAIST

2.5
21.5CUNI-KIT

10.5
014.5

HW-TSC

5432

NAIST

1CUNI-KIT

01515.5

1616.51714.5BLEU
CJNLP 2023 (November 11th, 2023)

1515.5BLEU

1616.517

NAIST-SIC (Simultaneous Interpretation Corpus)• A collection of Simultaneous Interpretation• https://dsc-nlp.naist.jp/data/NAIST-SIC/• You can (easily) find by searching “NAIST-SIC”• A part of this corpus (NAIST-SIC 2021) was used for IWSLT
Simutaneous Translation shared task
• It was also presented at IWSLT 2021
• Doi et al., Large-Scale English-Japanese Simultaneous Interpretation Corpus:
Construction and Analyses with Sentence-Aligned Data, Proc. IWSLT 2021.

• An additional release (NAIST-SIC 2022) includes automatic
source-target sentence alignmentCJNLP 2023 (November 11th, 2023)

Robustness of discrete prompts
for pre-trained models
Y. Ishibashi+, “Evaluating the Robustness of Discrete Prompts,” EACL 2023
CJNLP 2023 (November 11th, 2023)

Prompt-based Problem Solving• E.g., Natural Language Inference (NLI)[Hypothesis] [Prompt] | <MASK>, [Premise]
Manually-written
prompts andfine-tuning (MP)AutoPrompt (AP)(found through gradientguided search)

Discrete Prompts
CJNLP 2023 (November 11th, 2023)

Pre-trained
Masked LM

Robustness of Discrete Prompts• In this work, we evaluated the following on the NLI task:• Robustness against perturbations on discrete prompts• Token reordering (shuffling)• Token deletion• Robustness against different datasets• Out-of-domain data
• Perturbed dataCJNLP 2023 (November 11th, 2023)

Experimental Results (1/2)- Token reordering
CommitmentBank

MultiNLI3%↓10%↓9%↓APMPAPMPAccuracy

21%↓- Tokendeletion
MPAP123456 789
Position of the deleted prompt token

10Larger drop in AutoPrompt;“AP relies on token (word)
order more than MP”
Even a single word deletion
may hurt manually-written
prompts seriously

CJNLP 2023 (November 11th, 2023)

Experimental Results (2/2)
Accuracy- Out-of-domain data (cross-dataset evaluation)
CB→MNLI
47%↓
54%↓

AP

MP

MNLI→CB12%↓
33%↓

AP

Larger drop in MP;“MP does not generalize well”

MPAccuracy- Evaluation data perturbation (rewrite hypotheses and labels)
22%↓

APCB

56%↓

3%↓

MP

AP

MNLI

14%↓

MP

Larger drop in MP;“MP may overfit with specificdata distribution”

CJNLP 2023 (November 11th, 2023)

Adaptive and efficient speechsegmentation for speechtranslation
R. Fukuda+, “Speech Segmentation Optimization using Segmented Bilingual Speech
Corpus for End-to-end Speech Translation,” Interspeech 2022+ Recent progress (not published yet)CJNLP 2023 (November 11th, 2023)

Background: Speech Segmentation
Segmentation is a fundamental process required for Speech Translation (ST).
• Splitting continuous speech into translation units (segments).
• It’s difficult because explicit boundaries such as punctuation marks are not available.
• It’s important because it greatly affects translation results.
Proper segmentation…

…
I love dogs

but I’m allergic to dogs

ST
好きなのは⽝

ST…

…
I lovedogs

but I’m allergic to dogs
STST

でも ⽝アレルギーなんだ

Propertranslation

Improper segmentation

segment

好きだ

CJNLP 2023 (November 11th, 2023)

⽝だけど、⽝アレルギーなんだ

Impropertranslation

Previous Approaches to Speech Segmentation• Pause-based• Model-based• Voice Activity Detection (VAD)• Identify segment boundaries
using a classifier model

silence31.31 s• Length-based

34.56 sinput speech frames

Segmentation
Frame Classifier
20s

20s

. . 111100000000000000000001. .

• Pause-Length Hybrid
CJNLP 2023 (November 11th, 2023)

output binary labels
for each frameProblem• Inefficiency due to passive
segmentations
Previous work (SHAS)

200
180
160

# samples• Probablistic divide-andconquer [Tsaimas+ 2022 Interspeech]
Classifier model

140
120
100
8060

Transformer Encoder

wav2vec 2.040
2001234567891011duration (seconds)

12131415 >15

Corpus segments

400

𝑝𝑟𝑜𝑏𝑠
= [𝑝! , 𝑝" , … , 𝑝# ]350

# samples

300
250

Split atargmin 𝑝𝑟𝑜𝑏𝑠

200
150
100
50
0345

678

9

10

11duration (seconds)

1213

14

15 >15CJNLP 2023 (November 11th, 2023)

Terminate whensegment falls below acertain length

…

2…

1

Hierarchicalsegmentation

Proposed Method• Fine-tuning wav2vec 2.0
Transformer layers• Smoothed by moving average

Threshold for𝑖-th segment

Threshold for𝑖 + 1-th segment

0𝑠!

𝑠! + 𝑚𝑖𝑛 + 𝑙𝑒𝑟𝑝"!#𝑠! + 𝑚𝑖𝑛𝑒!

𝑠!&'Time (s)𝑠! + 𝑚𝑎𝑥)

Feed Forward
LayerNorm

LayerNorm

Attention

Attention

LayerNorm

LayerNorm

Feature Extractor

𝑠! + 𝑚𝑎𝑥 − 𝑙𝑒𝑟𝑝"$%CJNLP 2023 (November 11th, 2023)

Feed Forwardx

Probability

SMA ofprobabilityx(1Transformer Encoder

Adapter• Thresholding segmentation
probability

wav2vec 2.0

Experimental Results• Higher BLEU on MuST-C ende
MuST-C segmentation
SHAS [Tsiamas+ 2022]
Proposed• More efficient than SHAS

26.99
25.67
26.30

350

120

300

100250# samples

140

80
60

200
150
100

20500001234567891011121314151617181920>204001234567891011121314151617181920>20# samples• Derives shorter segments

duration (seconds)

duration (seconds)

CJNLP 2023 (November 11th, 2023)

Summary• NAIST’s recent activity on simultaneous speech translation• Prototype SimulST system• Prefix Alignment• Average Token Delay• MQM-based human evaluation• Recent results in IWSLT evaluation campaign• Simultaneous Interpretation Corpus: NAIST-SIC
CJNLP 2023 (November 11th, 2023)