Speech Segmentation Optimization
using Segmented Bilingual Speech Corpus
for End-to-End Speech Translation

Ryo Fukuda, Katsuhito Sudoh, Satoshi Nakamura
Nara Institute of Science and Technology, Japan

Segmentation for Speech Translation
Speech segmentation is essential for automatic speech translation (ST)• splits continuous speech into short segments before translation…

Retten Sie die Schuhe.

…

Vielen Dank.

segmentationtranslation• existing ST systems cannot directly translate long continuous speech• explicit segment boundaries are unavailable in speech
Introduction

MethodExperiments

Conclusions

1

Related work: Pause-based segmentation
Voice Activity Detection (VAD) - traditional approach [Sohn+1999][Bangalore+2012]• splits speech based on detected silencessilence• large gap with the manual segmentation(quote from [Papi+2021])

Introduction

MethodExperiments

Conclusions

2

Related work: Pause-based segmentation
Voice Activity Detection (VAD) - traditional approach [Sohn+1999][Bangalore+2012]• pauses do not always match sentence boundaries→ Over-/Under-segmentation problem
n Long silences in a (oracle) segment → Over-segmentation(“i think my personal storyis explained simply on slides”)

n Short silences between (oracle) segments → Under-segmentation(“no he is in trouble what is the problem ”)

Introduction

MethodExperiments

manual segmentation
VAD segmentation

Conclusions

3

Related work: Length-based segmentation• Fixed-length segmentation [Sinclair+2014]• simple but works better than pause-based segmentation [Gaido+2021]• does not take acoustic and linguistic clues into account
…
20s

20s

• Hybrid of pause- and length-based segmentation [Potapczyk+2020][Gaido+2021][Inaguma+2021]• heuristic concatenation of VAD segments up to a fixed length to address
the over-segmentation problem
length threshold[min_len, max _len]• still splits audio at inappropriate boundaries…
VAD
Introduction

Method4s+5sExperiments

Conclusions

4

Related work: Re-segmentation of transcripts• Re-segmenting ASR results• punctuation restoration [Lu+2010][Rangarajan Sridhar+2013][Niehues+2015]• language model [Stolcke and Shriberg, 1996][Wang+2016]• corpus-based segmentation model [Wan+2021][Wang+2019][Iranzo-Sánchez+2020]…

hi there the weather…
segmentationASRtoday was warm

re-segmentation
Retten Sie die Schuhe.
Vielen Dank.

Hi there.

MTThe weather today was warm.

• difficult to use in end-to-end ST, and cannot recover ASR errors due to improper
segmentation
Introduction

MethodExperiments

Conclusions

5

Proposed Method6Corpus-based segmentation
Speech segmentation as a frame-level sequence labeling task
• use a bilingual speech corpus as training data for speech segmentation• bilingual speech corpus includes speech segments aligned to sentence-like unit
Bilingual speech corpus
27.46 s30.07 s 31.31 s

34.56 ssource speech

Introduction

Save the shoes.

Thank you.

Retten Sie die Schuhe.

Vielen Dank.

Methodsource text
target text

Experiments

Conclusions

7

Corpus-based segmentation
Speech segmentation as a frame-level sequence labeling task
• use a bilingual speech corpus as training data for speech segmentation• bilingual speech corpus includes speech segments aligned to sentence-like unit
Bilingual speech corpus
27.46 s30.07 s 31.31 s

34.56 sinput speech frames

segmentation
model. . 111000000000000000000001111111100000000000000000001. .

Introduction

MethodExperiments

output binary labels
for each frame

Conclusions

8

Speech segmentation model• Model: 2D convolution ＋ Transformer Encoderℒ%&'[ 𝑥! 𝑥" 𝑥# 𝑥$][𝑥!! 𝑥!" 𝑥!# 𝑥!$]Linear + Softmax
Transformer encoder
2D convolution

Introduction

MethodExperiments

Conclusions

9Speech segmentation model• Model: 2D convolution ＋ Transformer Encoder• Data: two consecutive segments are concatenatedand assigned a sequence of label 𝑥 ∈ {0,1}ℒ%&'[ 𝑥! 𝑥" 𝑥# 𝑥$][𝑥!! 𝑥!" 𝑥!# 𝑥!$]Linear + Softmax
Transformer encoder
2D convolution

Introduction

MethodExperimentsConclusions

10

Speech segmentation model• Model: 2D convolution ＋ Transformer Encoder• Data: two consecutive segments are concatenatedand assigned a sequence of label 𝑥 ∈ {0,1}• Training objective: cross-entropy ℒ!"# (𝑥,* 𝑥) withrescaling weight 𝑤!
*ℒ%&' (𝑥,! 𝑥) = − + ,𝑤% log()!

[𝑥!! 𝑥!" 𝑥!# 𝑥!$]Linear + Softmax
Transformer encoder

exp(𝑥!(,!)
𝑥
exp(𝑥!(,, + 𝑥!(,!) (,!

+ (1 − 𝑤% ) log

Introductionℒ%&'[ 𝑥! 𝑥" 𝑥# 𝑥$]

2D convolution

exp(𝑥!(,,)
𝑥5exp(𝑥!(,, + 𝑥!(,!) (,,MethodExperiments

Conclusions

11

Inference Process
1. speech is segmented at a fixed-length 𝑇 and input into the segmentation model
2. fixed-length segments are re-segmented according to the labels predicted by
the segmentation modelPrediction• segmentation model selects label 𝑙! ∈ {0,1}with the highest probability at each time 𝑛:

1 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 0 0 0 0 0 0 0 0 1 1

re-segmentation
𝑥! 𝑥" 𝑥# 𝑥$ 𝑥% 𝑥& 𝑥' 𝑥( 𝑥) 𝑥!*𝑥!!𝑥!"𝑥!#𝑥!$𝑥!%𝑥!&𝑥!'𝑥!(𝑥!)𝑥"*𝑥"!𝑥""𝑥"#𝑥"$𝑥"%𝑥"&segmentation model

segmentation model…

𝑙! ≔ argmax 𝑥1! (𝑥1! ∈ 𝑅 " )Hybrid method• combine the model predictions with the VAD
results vad! ∈ {0,1}:𝑙( ≔ 9fixed-length 𝑇 (t = 1)

Introduction

fixed-length 𝑇 (t = 2)

Methodargmax 𝑥!( ∧ vad( (𝑠𝑒𝑔𝑚 𝑙𝑒𝑛 < 𝑚𝑎𝑥𝑙𝑒𝑛)
argmax 𝑥!( ∨ vad( (𝑠𝑒𝑔𝑚 𝑙𝑒𝑛 ≥ 𝑚𝑎𝑥𝑙𝑒𝑛)

Experiments

Conclusions

12

Experiments

13

Experimental Settings
Data: Multilingual Speech Translation Corpus (MuST-C) [Gangi+2019]• English-German: 230k segments from MuST-C v1• English-Japanese: 330k segments from MuST-C v2

Segmentation methods• Baseline: WebRTC VAD (VAD), pre-defined fixed length (Fixed-length)• Proposal: Segmentation model (Our model), hybrid method (VAD hybrid)ST systems• Cascade ST: cascade of ASR and MT models• End-to-end ST: an ST model that directly translates English speech

Evaluation• WER and BLEU for hypotheses re-segmented by the edit distance-based algorithm[Matusov+2005]

Introduction

MethodExperiments

Conclusions

14

Overall Results
MuST-C v1 English-GermanOracleBest VAD
Best Fixed-length
Our model+ VAD hybrid

Cascade ST End-to-end ST
WER BLEU
BLEU
12.60 23.59
22.50
30.59 17.02
16.40
20.60 19.29
17.96
20.99 20.18
19.10
19.06 20.99
19.87

MuST-C v2 English-JapaneseOracle
Best VAD
Best Fixed-length
Our model+ VAD hybrid

Cascade ST End-to-end ST
WER BLEU
BLEU
9.30
12.50
10.60
25.81
9.26
8.14
18.89
9.64
8.52
16.21
9.71
8.77
13.67 10.60
9.24• Our model outperformed VAD and the fixed-length baselines for both cascade and endto-end STs• The hybrid method with VAD significantly improved translation performance• room for improvement remains compared to the oracle segments contained by theMuST-C corpus (Oracle)
Introduction

MethodExperiments

Conclusions

15

Case study (1)Example of ASR and MT outputs with segmentation positions (■)
Oracle (ASR)

bonobos are together with chimpanzees you aposre living closest relative ■Oracle (MT)Bonobos sind zusammen mit Schimpansen, Sie leben am nächsten Verwandten. ■Best VAD (ASR)

bonobos are ■ together with chimpanzees you aposre living closest relative that ...

Best VAD (MT)Bonobos sind es. ■ Zusammen mit Schimpansen leben Sie im Verhältnis zum ...

Our model (ASR)

bonobos are together with chimpanzees you aposre living closest relative ■Our model (MT)Bonobos sind zusammen mit Schimpansen, Sie leben am nächsten Verwandten. ■• VAD resulted in over-segmentation (“bonobos are ■ together”) and under-segmentation(“relative that ...”).
• our model split the speech at a boundary close to an oracle segment and obtained the same ASR
and MT results
Introduction

MethodExperiments

Conclusions

16Case study (2)Visualization of waveforms and segmentation positions• hybrid decoding alleviated the over-segmentation problem by requiring anagreement between our model and VAD
Oracle segments
Predicted segments
Predicted boundaries

Our modelVAD

Our model +VAD hybrid
28.36 sIntroduction

36.79 s
MethodExperiments

Conclusions

17

Conclusions
Speech segmentation method based on bilingual speech corpus• directly split speech into segments that correspond to sentence-like units

Experimental results• our method outperformed the existing methods on both cascade and end-toend STs• hybrid approach with VAD further improved the translation performance

Future work• investigation on different domains and noisy environments• integration of segmentation function into an end-to-end STIntroduction

MethodExperiments

Conclusions

18

Appendix

19

ST model Settings

20

Inference Process
1. speech is segmented at a fixed-length 𝑇 and input into the segmentation model
2. fixed-length segments are re-segmented according to the labels predicted by
the segmentation modelPrediction• segmentation model selects label 𝑙! ∈ {0,1}with the highest probability at each time 𝑛:

1 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 0 0 0 0 0 0 0 0 1 1

re-segmentation
𝑥! 𝑥" 𝑥# 𝑥$ 𝑥% 𝑥& 𝑥' 𝑥( 𝑥) 𝑥!*𝑥!!𝑥!"𝑥!#𝑥!$𝑥!%𝑥!&𝑥!'𝑥!(𝑥!)𝑥"*𝑥"!𝑥""𝑥"#𝑥"$𝑥"%𝑥"&segmentation model

segmentation model…

𝑙! ≔ argmax(𝑥! )Hybrid method• combine the model predictions with the VAD
results vad! :𝑙( ≔ #fixed-length 𝑇 (t = 1)

Introduction

fixed-length 𝑇 (t = 2)

Methodargmax 𝑥( ∧ vad( (𝑠𝑒𝑔𝑚 𝑙𝑒𝑛 < 𝑚𝑎𝑥𝑙𝑒𝑛)
argmax 𝑥( ∨ vad( (𝑠𝑒𝑔𝑚 𝑙𝑒𝑛 ≥ 𝑚𝑎𝑥𝑙𝑒𝑛)

Experiments

Conclusions

21