Incorporating Discriminative DPGMM
Posteriorgrams for Low-resource ASR
Bin Wu1 , Sakriani Sakti1,2 , and Satoshi Nakamura1,2{wu.bin.vq9,ssakti,s-nakamura}@is.naist.jp

1. Nara Institute of Science and Technology, Japan
2. RIKEN, Center for Advanced Intelligence Project AIP, Japan
Copyright © 2021 NAIST Japan, All rights reserved

1Incorporating Discriminative DPGMM
Posteriorgrams for Low-resource ASR
Bin Wu, Sakriani Sakti , and Satoshi Nakamura

Speech features

Phonemic discriminability comparison• First step of ASR system: extracting proper
speech feature• Feature discriminability are compared based on
ABX error rate in Zerospeech challenges• Typical speech features: MFCC and PLP• Effective in discriminating phonemes by extracting
smooth formant envelopes• Speech feature for ASR should have sufficientability to discriminate phonemes (words),
robust to genders, emotions, and noises.

• Acoustic features• MFCC and PLP• Neural network features• autoencoder, ABnet and VQVAE• Parametric clustering features• GMM and k-means• Nonparametric clustering features• DPGMM (Dirichlet Process Gaussain Mixture Model)• DPGMM gets lowest ABX error as strongest phonemic
discriminability in Zerospeech 2015, 2017 and 2019
Copyright © 2021 NAIST Japan, All rights reservedIncorporating Discriminative DPGMM
Posteriorgrams for Low-resource ASR
Bin Wu, Sakriani Sakti , and Satoshi Nakamura

Proposal
Concatenate DPGMM Posteriorgrams withMFCC features to increase phonemic
discriminability to improve ASR

Copyright © 2021 NAIST Japan, All rights reserved

MFCC+DGPMM vs. MFCC on ASR• MFCC+DPGMM better than MFCC on ASR• More absolute improvement on smaller corpus

Incorporating Discriminative DPGMM
Posteriorgrams for Low-resource ASR
Bin Wu, Sakriani Sakti , and Satoshi Nakamura

MFCC+DGPMM vs. tandem on ASR• Compare discriminability of proposal (MFCC+DPGMM)
and tandem (MFCC+BNF99)• Tandem bottleneck features from Kaldi default script• Proposed feature better than tandem on smaller corpus• Tandem system needs the knowledge of phonemes andbeginning and end time information of phonemes, which
is not needed by our proposal

Copyright © 2021 NAIST Japan, All rights reservedConclusion• DPGMM features strong at discriminating phonemes• Propose DPGMM + MFCC to improve the ASR• The concatenated features work well on LVCSR,
especially with fewer resources

Speech features• First step of ASR system: extracting proper speech feature• Typical speech features: MFCC and PLP• Discriminating phonemes by extracting smooth formant envelopes
Frequency

Formants

Copyright © 2021 NAIST Japan, All rights reserved

5Measure the phoneme discriminability• Speech feature for ASR should have sufficient ability to discriminate
phonemes (words) and be robust to genders, emotions, and noises.
• Measure phoneme discriminability of a feature by ABX error rate
• Each utterance to segments of features• Make an ABX error: d(seg. in same phn category) > d(seg. in different phn categories)• ABX error rate: ratio of samples making ABX errors

𝐴𝐵𝑋𝐸𝑟𝑟 𝐼 , 𝑢=#𝑡𝑟𝑖𝑝𝑙𝑒𝑡𝑠(𝑑 𝐴, 𝐵 > 𝑑 𝐴, 𝑋 )#𝑡𝑟𝑖𝑝𝑙𝑒𝑡𝑠

Copyright © 2021 NAIST Japan, All rights reserved

Any A, B and X triplet (three segments as an sample)
A, B from same category 𝐼A, X from different categories 𝑢6

Feature discriminability comparison• Compare Feature discriminability based on ABX error rate in Zerospeechchallenges• Acoustic features• MFCC and PLP• Neural network features• autoencoder, ABnet and VQVAE• Parametric clustering features• GMM and k-means• Nonparametric clustering features• DPGMM (Dirichlet Process Gaussain Mixture Model)• DPGMM gets lowest ABX error rate in Zerospeech 2015, 2017 and 2019
Copyright © 2021 NAIST Japan, All rights reserved7Why DPGMM strong at discriminatingphonemes• DPGMM dynamically changes number of clusters to fit data with max
probability with segmental patterns clearly discriminatedCopyright © 2021 NAIST Japan, All rights reserved

8Why DPGMM strong at discriminatingphonemes• DPGMM dynamically changes number of clusters to fit data with max
probability with segmental patterns clearly discriminatedCopyright © 2021 NAIST Japan, All rights reserved

9Why DPGMM strong at discriminatingphonemes• DPGMM dynamically changes number of clusters to fit data with max
probability with segmental patterns clearly discriminatedCopyright © 2021 NAIST Japan, All rights reserved

10Why DPGMM strong at discriminatingphonemes• DPGMM dynamically changes number of clusters to fit data with max
probability with segmental patterns clearly discriminatedCopyright © 2021 NAIST Japan, All rights reserved

11Why DPGMM strong at discriminatingphonemes• DPGMM dynamically changes number of clusters to fit data with max
probability with segmental patterns clearly discriminatedCopyright © 2021 NAIST Japan, All rights reserved

12Proposal

Copyright © 2021 NAIST Japan, All rights reserved

13Proposal• Proposal: Combine DPGMM features and acoustic features toimprove the ASR system
Utterance “hi”

Typical speech featureWaveform

Speech feature (MFCC)

1Feature
Extraction

Proposed speech featureConcatenated feature of MFCC
and DPGMM Posteriorgram

25DPGMM
Clustering
Copyright © 2021 NAIST Japan, All rights reserved

4

DPGMM posteriorgram(DPGMM posterior vectors)
1 1 0.9 0.2 00 0 0.1 0.8 1FeatureCombination

6
1 1 0.9 0.2 00 0 0.1 0.8 1

14ASR System
Decoder
Attention

Encoder

15

Corpora• TIMIT: English corpus of reading speech• WSJ: Spontaneous speech for large vocabulary continuous speech recognition(LVCSR)• Training datasets• “train si84” (about 15 hours)• “train si284”(about 80 hours)• development dataset of “dev93”• evaluation dataset of “eval92”
Copyright © 2021 NAIST Japan, All rights reserved

16ASR needs concatenated feature
instead of DPGMM posteriorgram aloneAnalysis on TIMIT test set
Phonemes clearly classified by DPGMM
posteriorgrams, although not MFCC

Copyright © 2021 NAIST Japan, All rights reserved

Framewise contextual modeling createsfragmented DPGMM posteriorgrams

17Performance on WSJ
Proposed feature (DPGMM+MFCC) converged
faster and retained the improvement

It improves more obviously on the systemtrained on the small dataset (train_si84)

Copyright © 2021 NAIST Japan, All rights reserved18Performance on WSJ(Espnet, 2017)(Speech Chain, 2020)
Absolute improvement (%)
1.75

Absolute improvement (%)
0.9
19Compare with tandem features• Compare our proposed with Kaldi default tandem features• On a relatively small dataset TIMIT (~3h training), our method had better
performance on TIMIT corpus• On a bigger dataset of WSJ SI-284 (~80h training), tandem system got slightly
better performance• Tandem bottleneck features need the knowledge of phonemes and forcealignment information

Copyright © 2021 NAIST Japan, All rights reserved

20Conclusion• DPGMM features strong at discriminating phonemes• DPGMM + MFCC to improve the ASR• The concatenated feature works well on LVCSR, especially with fewer
resources

Copyright © 2021 NAIST Japan, All rights reserved

21