MFCC-DPGMM Features for Enhancing Low-Resource ASR∗
◎ Bin Wu1 , Sakriani Sakti1,2 , Satoshi Nakamura1,2(1 NAIST, 2 RIKEN AIP)

1Introduction
For years, Zerospeech challenge 2015, 2017, andInputspeech2019 have constantly conﬁrmed the power of Dirich-Feature MFCC
extraction

Our proposed
ASR

let Process Gaussian Mixture Model (DPGMM)

DPGMM
DPGMM posteriorgram
clustering

ASR

features to discriminate phonemes across diﬀerent
speakers, diﬀerent languages under such harsh conditions as interviews with randomly interrupted dis-Fig. 1Proposed features for ASR: concatenation ofMFCC features and their DPGMM posteriorgrams.

ﬂuent dialogues, and read speech in wild or noisy
recording environments.

DPGMM(α, NIW(µ0 , λ, Σ0 , ν)).

DPGMM clustering can discriminate phonemes

Given the model deﬁnition and data {xi }ni=1 , wewell because it dynamically changes the number of

can infer from the Gibbs sampling to get posterior

Gaussians until each one ﬁts one segmental patternp(zi |xi ) and the cluster k ∗ of any data point xi byof the whole speech corpus with the highest proba-k ∗ = argmaxk p(zi = k|xi ).

bility such that the linguistic units of diﬀerent segmental patterns are clearly discriminated.

2.2

Concatenating

DPGMM

Posterior-grams with MFCC Features

However, to the best of our knowledge, DPGMM
features have not been applied to large vocabulary

Compared with a traditional ASR system which

continuous speech recognition (LVCSR) before. In-

directly extracts such acoustic features as MFCC forrecognition tasks, our proposal applies the DPGMM

spired by DPGMM s relatively strong discriminability, we applied it to an LVCSR system by concatenating acoustic features with DPGMM posterior-clustering algorithm on the acoustic features, getsgrams such that the concatenated features combine

catenates the DPGMM posteriorgrams with thethe power of both to enhance the ASR system.

MFCC features as enhanced features for the ASR

the unsupervised DPGMM posteriorgrams and con-system (Fig. 1).

2Method

2.1

DPGMM Clustering

We can treat DPGMM as an inﬁnite GMM with∑∞density function p(xi ) =k=1 πk p(xi |µk , Σk ) (al∑∞ternatively, p(xi ) = k=1 p(Zi = k)p(xi |Zi = k)).

The DPGMM posteriorgrams are of relatively
high dimension, the probabilities are usually concentrated on one or two dimensions for each frame,
and most of the other dimensions are zeros. MFCC
is full of acoustic details in all the dimensions, butThis generative model samples mixture weights

the DPGMM posteriorgram is discriminative with{πk }∞k=1 from the stick-breaking process (with con-few dimensions; they complement each other in fea-centration parameter α) and the means and vari-ture combinations. We will show that concatenating

ances {µk , Σk }∞k=1 from the normal-inverse-Wishart

an MFCC feature and its DPGMM posteriorgram(NIW) distribution (with a belief of mean µ0 , thebelief of variance Σ0 , the belief-strength of mean λ,and the belief-strength of variance ν). The genera-

improves the ASR performance.

3Experiment Settings

tive model also samples Gaussian cluster indicator

We trained an attentional encoder-decoder ASR

hidden variable Zi by mixture weights and each datasystem. We set batch size to 32 and used the Adam

point Xi by the Gaussian cluster indicated by Zi .

optimizer with an initial learning rate of 0.001. We

The joint distribution of model can be described asevaluated our ASR system with a beam search where
the beam size was 10.

∗低リソース ASR の性能を向上するための MFCC-DPGMM 特徴量、
◎ビンウー 1 、 サクリア二サクチイ 1,2 、中村 哲 1,2（1 NAIST, 2 RIKEN AIP)Character accuracy of development set

Table 10.9

We compared the attentional encoder-decoder ASR systems with or without feature exten-0.8

sion of the DPGMM posteriorgrams, along with two0.7baselines [3, 2], by the character error rates (CERs)

0.6

WSJ train_si284 MFCC+DPGMM
WSJ train_si284 MFCC
WSJ train_si84 MFCC+DPGMM
WSJ train_si84 MFCC

0.5051015Epoch202530

on the WSJ speech corpus. No systems used pronunciation dictionaries or language models in thedecoding process. We divided the WSJ corpus into
the following datasets based on the Kaldi recipe:training datasets of “train si84” (about 15 hours) orFig. 2 Comparison between ASR systems with“train si284” (about 80 hours); an identical develop-acoustic MFCC features (solid lines) and concate-ment dataset of “dev93” and an identical evaluationnated MFCC features and their DPGMM posterior-dataset of “eval92” for all systems.

grams (dashed lines) by character accuracy on thedevelopment set “dev93” of WSJ corpus.

ASR on WSJ train si84 (15 hrs)

CER%

Att Enc-Dec (Baseline ASR1) [2]
Att Enc-Dec (Baseline ASR2) [3]

17.01
17.35

model, whose training process used the same pa-

Att Enc-Dec (Ours MFCC)

16.61

rameter setting as previous works [1]. We set theAtt Enc-Dec (Ours MFCC+DPGMM)

14.86

concentration parameter to 1 and the mean and vari-

ASR on WSJ train si284 (80 hrs)

CER%

ance of the prior to the global mean and the global

Att Enc-Dec (Baseline ASR1) [2]

8.17

variance of the MFCC features with belief-strengths

Att Enc-Dec (Baseline ASR2) [3]7.12

1 and D + 2, where D is the number of dimensions

Att Enc-Dec (Ours MFCC)

6.57

of the MFCC features. We obtained cluster labels

Att Enc-Dec (Ours MFCC+DPGMM)

5.67

We used python to implement the DPGMMafter 1500 sampling iterations.

4show that the concatenated feature works well on

ResultLVCSR, especially with fewer resources.

We veriﬁed the eﬀectiveness and stability ofour proposed method with the spontaneous speech
recognition task on the WSJ corpus. Table 1 showsthat on both tasks with identical ASR system set-6

Acknowledgements
Part of this work was supported by JSPS KAK-ENHI Grant Number JP17H06101.

tings, we observed a more constant decrease of
CER with the feature with extension (MFCC +DPGMM) than in the original feature (MFCC).
We analyzed the performance of the ASR systems
during the entire training process.

Fig. 2 showsthat the ASR systems with feature extension by
the DPGMM posteriorgram converged faster and re-References[1] M. Heck, S. Sakti, and S. Nakamura. Unsupervised linear discriminant analysis for supporting DPGMM clustering in the zero resource scenario.

Procedia Computer Science, 81:73–79,2016.

tained improvement compared to that without thefeature extension on the character accuracy of the[2] S. Kim, T. Hori, and S. Watanabe.

Jointdevelopment set (“dev93”). Our proposed featureimproves more obviously on the system trained onCTC-attention based end-to-end speech recog-

the small dataset (“train si84”) than on the large

International Conference on Acoustics, Speechdataset (“train si284”).

and Signal Processing (ICASSP), pp. 4835–4839.

nition using multi-task learning. In 2017 IEEEIEEE, 2017.

5Conclusion
Since DPGMM features are strong at discriminat-ing phonemes, we propose to concatenate MFCC
and DPGMM to improve the ASR system. Results[3] A. Tjandra, S. Sakti, and S. Nakamura. Machine speech chain. IEEE/ACM Transactions on
Audio, Speech, and Language Processing (IEEETASLP), 28:976–989, 2020.