CROSS-LINGUAL SPEECH-BASED TOBI LABEL GENERATIONUSING BIDIRECTIONAL LSTM
1Marco Vetter

1,2
Sakriani Sakti

marco.vetter.mp8@is.naist.jp

ssakti@is.naist.jp

1 Nara Institute of Science and Technology, Japan

1,2
Satoshi Nakamuras-nakamura@is.naist.jp

2 RIKEN, Center for Advanced Intelligence Project AIP, Japan

Introduction

Cross-Lingual ToBI Break Index Labels

Experimental Results (cont.)

Motivation• Documenting under-resourced languages automatically fromspeech requires word segmentation and lexical discovery• Word segmentation is a difficult task
• Infants learn prosody before they learn words• Prosodic information has also been experimentally shown toimprove word segmentation [Ludusan, 2015]
Existing work
• Existing work on ToBI label prediction is usually monolingual[Chen, 2004] [Elvira-Garcia, 2016]• Existing approaches use additional information at test time,
not just speech [Rosenberg, 2010]
Problem• Lack of training data on under-resourced languages
This work proposes• We use bidirectional LSTM to extract prosodic informationfrom speech in a cross-lingual fashion• No labelled target language data is requiredCorpora• Corpus of spontaneous Japanese (∼38h of spontaneousJapanese speech)• Boston Radio Corpus (∼78m of English radio speech)• For this study we treat Japanese as a richly sourced language
and English as a low-resource target language
Labelling system• ToBI break labels are a standard available for various
languages to mark prosody in speech.
• For cross-lingual prosody detection we map ToBI (EN) andJ_ToBI (JP) labels to produce a common inventory

Results for multi-class labels• These experiments discriminate between three classes• (1) word boundary, (2) intermediate/accentual phrase, (3)intonational phrase

Speech + Labels Train(1)(Language S)

Model(Language S)

Test

Model(Language S)

Speech(2)(Language T)

Labels(Language T)

Speech based label generation• Our systems are entirely speech-based at test time
• No transcriptions, existing word segmentation or lexicon used
MFCC (39-dim)
Input
BiLSTM 1
BiLSTM 2OutF
0.55040.21140.39910.41770.02250.25550.47650.12290.2050

Visualized Output• Cross-lingually generated boundary labels for English speech

Network, Features and Metrics
Bidirectional LSTM using MFCC features• The network is a bidirectional LSTM consisting of two hiddenlayers with 1024 BiLSTM cells each
• Features were standard 39-dimensional MFCC with Δ and ΔΔ
F-Score and tolerance• Segmentation is evaluated using Precision, Recall and F-score• As in other boundary detection tasks we apply a tolerance
during evaluation (80ms)

Experimental ResultsConclusions

Binary and multi-class models• Binary models only discriminate "break" or "no break"• Multi-class models differentiate three label types (see "Data")

Summary• We cross-lingually predict ToBI-style prosodic boundaries• Approach requires no prior knowledge of target language• Uses no information besides speech at test time
• Models retain much of their predictive power when applied
across languages.
Future Work• Apply the extracted information to word segmentation• Evaluate additional language pairs

Results for binary labels• These experiments discriminate only "break" and "no break"
Language pairing P
0.4932JP ➞ JP0.6308EN ➞ EN
0.5914
JP ➞ ENLabels

The 44th International Conference on Acoustics, Speech, and Signal Processing
May 12-17, 2019, Brighton, UKR
0.62720.16200.37700.36140.01370.17000.65780.12990.1974ICASSP 2019

R
0.61370.78800.5216

F
0.54000.6956
0.5533refCross-lingual model application• Cross-lingual model application allows training on richly
sourced languages (1)• Models are then applied on an under-resourced target
language (2)P
0.49470.33060.47230.52380.63300.7031
0.38160.12790.2204hypMethods

Break level description
ToBI J_ToBI Mapping
Word boundary
111
Lower-level grouping2
n/a2
Intermediate/accentual phrase
322
Intonational phrase
433

Language pairing Class
12JP ➞ JP
312EN ➞ EN
312JP ➞ EN
3

• Ground truth labels are correctly identified, but placement
does not match exactly• False positives also occur with some frequency

Part of this work was supported by JSPS KAKENHI Grant Numbers JP17H06101 and JP17K00237.