Using Functional Load for Optimizing
DPGMM based Zero Resource Sub-word
Unit Discovery
Bin Wu1 , Sakriani Sakti1,2 , Jinsong Zhang3 and Satoshi Nakamura1,2{wu.bin.vq9,ssakti,s-nakamura}@is.naist.jp, jinsong.zhang@blcu.edu.cn

1. Nara Institute of Science and Technology, Japan
2. RIKEN, Center for Advanced Intelligence Project AIP, Japan
3. Beijing Language and Culture University, China
2018/12/10

1Background2018/12/10

2

Research Questiono h a y o o sil• How to find phoneme-like units from zero-resource speech?

line-girl1-ohayou1

2018/12/10

3Why important• Problem: zero-resource phoneme-like unit discovery• Why the problem important?
• State-of-art DNN needs labels (phonemes,…)• manual labelling needs money and effort• Knowledge of the labels (phonological system, …)• Zero-resource technology helps to create these labels (phonemes, …)

2018/12/10

4

Previous methods• Unsupervised sub-word unit discovery of Zerospeech• Pre-trained labels + DNN• spoken term detection + autoencoder [Badino 2014, Kamper, 2015; Pitt, 2015]• spoken term detection + ABNet [Synnaeve 2014, Thiolliere, 2015]• Unsupervised clustering• Variational autoencoders [Ondel, 2016; Ebber, 2017]• Dirichlet Process Gaussian Mixture Model (DPGMM Clustering) [lee, 2012; Chen, 2015]• DPGMM + ASR feature transformations [Heck, 2016]• DPGMM + ASR alignment [Heck, 2017]• DPGMM clustering gets top results of the Zerospeech Challenge 2015, 20172018/12/10

5

Problem

2018/12/10

6Human cognitive process of phoneme• Goal: Audio -> Phoneme-like unitso h a y o o sil• How does the human find the phonemes?
Top-down knowledge interpretation
phone sequence, words, grammar and semantics (Contextual)
Human cognitive process of speech(o h a y o osil )1 2 3 4 1 1
2018/12/10

Bottom-up acoustic-to-category process5(Acoustical)
DPGMM
7Problem1:DPGMM is toosensitive to acoustics

2018/12/10

8

Problems of DPGMM clustering• Problem1: DPGMM is too sensitive to acoustics• High frequency acoustics make lots of small DPGMM clusters• Rapid formant changes make lots of small DPGMM clusters• # of clusters > # of phonemes of usual languages

Example:f: high frequency
i: rapid format changeDPGMM Clusters
True phonemes
True words
2018/12/10

DPGMM clustering results on timit training corpus

10Problem2: DPGMM is weakin contextual modelling

2018/12/10

10

Contextual modelling• Context is important

School/sk1u:l/Kite/k2ait/
2018/12/10

K1 and K2 is acoustically different
However,
K1 is always following s
K2 is always following some word boundary

K1 and K2 are in completely different context
They belong to same phoneme.

11Problems of DPGMM clustering• Problem2: DPGMM is weak in contextual modelling

Example:• pack: /æ 1/ after pand: /æ 2/ before word boundary• acoustically different andbut complementary distribution• /æ 1/ and /æ 2/ belong to samephoneme /æ /• Acoustically different sub-word units are always treated as different labels by DPGMM.
• Although they are in completely different context and belongs to same phoneme

DPGMM Clusters
True phonemes
True words
2018/12/10

DPGMM clustering results on timit training corpus

12

Contextual modelling• Context is important
Assume B and 13 are two different phonemes,
But they are acoustically similar,
Sometimes B is between A and C
Sometimes 13 is between 12 and 14

We can distinguish B and 13 by the specificcontext A, C and 12, 14

2018/12/10

13Problems of DPGMM clustering

Example:• Shed: /ʃ/ and fields: /s/• /ʃ/ and /s/ acoustically similar• Only /s/ will following /d/fields can’t be ended as /d/ + /ʃ/• Problem3: DPGMM is weak in contextual modelling• Context can help distinguish acoustically similar phonemes

DPGMM Clusters
True phonemes
True words
2018/12/10

DPGMM clustering results on timit training corpus

14Problems of DPGMM• Human use context to distinguish phonemes• Acoustic different units with completely different context tends to bethe same phoneme• Context also helps distinguishing acoustic similar phonemes• Problems of DPGMM• weak in context modeling (top-down)• sensitive to acoustics (bottom-up)

2018/12/10

15

Proposal

2018/12/10

16

Proposal• But How to deal with the contextual effects?
• Statement:• If two units can be easily distinguished by the context.
• It means the contrast of two units are not important in communication• (a.k.a Functional Load (FL) is small)• Equivalently, the contrast conveys little information in communication• Extremely,
if two units are inCompletely different context,It means
FL = 0;It means conveying no info.

2018/12/1017

Computation of functional load
• The measurement of functional load of the contrasts• Information loss ignoring the contrast (Hockett, 1955)• functional load of a contrast of a label pair x and yFL( x, y ) H ( L)  H ( Lxy )H ( L)• eg. In English, K1 and K2 are in completely different context• Mathematically, 𝐹𝐿 𝑘1, 𝑘2 = 0

2018/12/10

School/sk1u:l/Kite/k2ait/
18

System configuration• Proposal: greedy mergers based on least functional load criteria• Iteratively merge the DPGMM label pairs with lowest functional load and enhance
our features by ASR

2018/12/10

19

Experiment & Result

2018/12/10

20

Experiment and result• Xitsonga corpus• an excerpt the NCHLT corpus of South African read speech (length: 2 h 29 min)• with the official segmentation of Interspeech Zero Resource Speech Challenge 20152018/12/1021Conclusion• DPGMM is weak in context modeling and sensitive to acoustics• We enhance the contextual modeling of DPGMM labels by minimum
functional criteria• Result shows we can get posterigram of much lower dimension withsimilar ABX error

2018/12/10

22

Thank you for listening

2018/12/10

23