INTERSPEECH 2016
September 8–12, 2016, San Francisco, USA

Supervised Learning of Acoustic Models in a Zero Resource Setting
to Improve DPGMM Clustering
Michael Heck, Sakriani Sakti, Satoshi Nakamura
Augmented Human Communication Laboratory,
Graduate School of Information Science,
Nara Institute of Science and Technology,
Nara, Japan{michael-h,ssakti,s-nakamura}@is.naist.jp

Abstract

been successfully applied to speech processing tasks such as unsupervised lexical clustering [8]. Chen et al. [9] cluster standard
MFCC speech features by inferring a DPGMM and demonstrate
its suitability for automatic detection of sound classes in untranscribed data. Their work is the best-performing contribution to
the zero resource speech challenge 2015 [1].
Speech processing systems typically utilize feature transformations to increase sound class discriminability. Linear discriminant analysis (LDA) [10] is a standard technique to minimize intra-class discriminability, to maximize inter-class discriminability and to extract relevant informations from highdimensional features spanning larger contexts. Maximum likelihood linear transforms (MLLT) [11, 12] and feature-space
maximum likelihood linear regression (fMLLR) [13, 14] arecommonly used to de-correlate feature components and forspeaker adaptation. Naturally, class discriminating propertiesare critical for clustering, and adaptive feature transformations
can help dealing with speaker variability. However, methods
such as LDA need class labels for estimating the feature transformations, making it difficult to use them in a zero resource
setting where the classes and even their amount are unknown.
In previous work [15] we demonstrated that it is possible
to learn LDA transformations on automatically generated labels, and that these transformations can be used to produce feature vectors that considerably improve clustering performance.
There has been work that utilize k-means clustering to automatically obtain pseudo labels for LDA estimation [16, 17]. But
unlike in these studies we were able to overcome the limitationof having to predefine the size of prospective label sets by utilizing the non-parametric DPGMM sampler for clustering.
In this work we improve the DPGMM clustering by utilizing multiple feature transformations that can be combined withthe previously exploited LDA transformations. Labels that wereautomatically generated in a first pass of DPGMM clustering
serve as basis for learning an acoustic model using LDA, MLLT
and fMLLR in an entirely unsupervised fashion. We demonstrate that each feature transformation helps improve cluster
quality and that the conjunction of transformations leads to thebest results. We also demonstrate that combining multiple clustering runs can greatly boost sound class discriminability.

In this work we utilize a supervised acoustic model training
pipeline without supervision to improve Dirichlet process Gaussian mixture model (DPGMM) based feature vector clustering.
We exploit methods common in supervised acoustic modelingto unsupervisedly learn feature transformations for application
to the input data prior to clustering. The idea is to automaticallyfind mappings of feature vectors into sub-spaces that are morerobust to channel, context and speaker variability. The need oflabels for these techniques makes it difficult to use them in azero resource setting. To overcome this issue we utilize a first
iteration of DPGMM clustering to generate frame based classlabels for the target data. The labels serve as basis for learning an acoustic model in the form of hidden Markov models(HMMs) using linear discriminant analysis (LDA), maximum
likelihood linear transform (MLLT) and speaker adaptive training (SAT). We show that the learned transformations lead tofeatures that consistently outperform untransformed features on
the ABX sound class discriminability task. We also demonstrate that the combination of multiple clustering runs is a suitable method to further enhance sound class discriminability.
Index Terms: acoustic unit discovery, Bayesian nonparametrics, Dirichlet process, feature transformation, Gibbs sampling,
unsupervised linear discriminant analysis, zero resource

1. Introduction
In a zero resource scenario, large amounts of labeled training
data, parallel data, and knowledge about the target language areunavailable for developing speech processing systems with supervised techniques. Albeit significant advances in developing
methods for unsupervised learning, current speech processing
technology is not yet capable to imitate the natural capacities
of infants to robustly learn acoustic and language models in an
unsupervised way. Specialized evaluations such as the zero resource speech challenge [1] address this demanding task.
Confronted with an unknown language, phonologists usually attempt to define a set of acoustic units to fully cover theunderlying sound repertoire. Machine learning approaches tothis task are pattern matching [2, 3] on raw audio data and unsupervised sound unit detection [4]. These techniques have beensuccessfully applied to solve tasks such as spoken term detection [5], topic segmentation [6] or document classification [7].
Model complexity usually is not known a priori when dealing with new data sets. Bayesian models such as the Dirichlet
process Gaussian mixture model (DPGMM) can automaticallyadjust the model complexity given some data and have already

Copyright © 2016 ISCA

2. Dirichlet process Gaussian mixture
model
DPGMMs (also known as infinite GMMs) extend finite mixture
models by the aspect of automatic model selection: The model1310

http://dx.doi.org/10.21437/Interspeech.2016-988

DPGMM
clusteringfinds its complexity automatically given the data. Inference
is typically sample based using a Markov chain Monte Carlo(MCMC) scheme such as Gibbs sampling. The sampler usedhere combines a restricted Gibbs sampler with a split/merge
sampler. For more in-depth informations, please refer to [9, 18].

ABX
scoringLabels2.1. Generative process

Supervised acoustic model training

Let X = {x1 , · · · , xn } be a set of observations. The generative
process of X given a DPGMM is as follows:• Mixing weights π = {π1 , · · · , πk } are generated according to a stick-breaking process• GMM parameters θ = {θ1 , · · · , θk } are generated according to a prior distribution NIW(mk , Sk , κk , νk )• A label zi is assigned to every xi , according to π
• xi is generated according to the zi -th GMM component
θk = {µk , Σk } are Gaussian parameters, and the parameter setof the prior Normal-inverse-Wishart (NIW) distribution consistsof a prior m0 for µk , a prior S0 for Σk , the belief-strength κ0
in m0 and the belief-strength ν0 in S0 .

Monophones

Triphones

Triphones
LDA+MLLT

Triphones
SAT

LDA

MLLT

fMLLR

DPGMM
clustering

DPGMM
clustering

DPGMM
clustering

ABX
scoring

ABXscoring

ABXscoring

Figure 1: Scheme of the iterative sampling process.

2.2. Inference
The parallelizable sampler alternates between a non-ergodic restricted Gibbs sampler and a split/merge sampler to form an
ergodic MCMC sampler.
Restricted Gibbs sampling allows labels zi to be sampledfrom a finite set Z. By definition of the DPGMM, the distribution of the mixture weights follows a Dirichlet distribution.
Split/merge sampling performs on the existing components. To provide good split candidates, each component isaugmented with two sub-clusters ckl and ckr , and each observation of a component is augmented with a sub-cluster label
zisub ∈ l, r. Split moves are proposed in a Metropolis-Hastings
fashion. Merge steps are proposed randomly.

3.1. Two-stage clustering
An initial DPGMM clustering on standard feature vectors with
derivatives (x00i ) provides generic class labels and the hypothesized class membership of every speech frame. Each class is
simply named with the numeric ID of the Gaussian that most
likely produced the respective feature vector.
The frame-wise labels serve as basis for the subsequentmodel training. For each utterance, we collapse the labels to approximate a more natural textual reference type by compressing
all subsequent tokens of the same type to a single token, imitating a phone based transcription of the audio recordings.
Figure 1 is a graphical overview of the iterative acoustic
model training and clustering pipeline. Once we extract newfeature vectors yi using one or more of the transformations inconjunction, we perform another run of frame-based DPGMM
clustering. Stages at which we extract the posteriorgrams forevaluation are named ABX scoring. Each stage produces a moreadvanced set of features for clustering, ranging from untransformed features to LDA+MLLR+fMLLR transformed features.

2.3. Posteriorgram generation
The posterior probability of cluster ck , given observation xi is
πk N (x|θk )p(ck |xi ) = P(1)
N (x|θj )and pi = (p(c1 |xi ), · · · , p(cK |xi )) is the posteriorgram for xi .

3. Unsupervised speech featuretransformation

3.2. Supervised acoustic model training
We initialize the acoustic model by context-independent monophone training. Then we subsequently train context dependent
triphones on untransformed standard features, followed by LDA
and MLLT estimation and model training on transformed features. Finally, we train a SAT model with fMLLR.
We use a 3-state HMM topology with a skip from the firststate to the next HMM to allow a more dynamic alignment. Dueto the nature of the automatic labels, some utterances might be
represented with a relatively high number of tokens. The skip
state guarantees that an alignment is always found.

We showed in [15] that the quality of DPGMM based speechfeature vector clustering can be improved by using LDA transformed features as input, where the LDA transformations were
estimated in an unsupervised fashion. To further improve theclustering quality we propose an extension to this work by utilizing transformations that can be used in conjunction to benefit
from additive effects. The transformations help to project feature vectors into a more suitable sub-space for sound class discrimination by feature de-correlation and speaker adaptation.
The need of labels and models makes it difficult to use thesemethods in a zero resource setting out of the box. Not only are
there no labels for the target data available, but class identities
and even the amount of classes are also unknown. Confronted
with an unknown language there is often no easy way to bootstrap acoustic models. In order to overcome these issues, weuse a two-staged clustering framework that automatically finds
frame-based class labels in a first clustering of the target data.

3.2.1. Dimension-reducing LDA
LDA is a simple linear transformation that we use to minimize
intra-class discriminability and maximize inter-class discriminability of the speech features. LDA also enables us to do
dimensional reduction of high-dimensional stacked feature vectors (x̂i ) that span a larger context c by omitting lower-ranked

1311

5. Experiments

coefficients. Estimation of the transformation requires the feature vectors and respective class labels. In our pipeline we learn
LDA transformations using the acoustic states as classes.

5.1. Data
The database for all our experiments is the official data set of theInterspeech zero resource speech challenge [1], which contains
two separate data sets of pure speech for American English (4h59min) and Xitsonga (2h 29min), a southern African Bantu language. The segments contain non-overlapping speech of exactly
one speaker and noise or pauses. The English data is extracted
from the Buckeye corpus and consists of conversational speech.
The Xitsonga data is an excerpt of the NCHLT corpus and iscomprised of read speech.

3.2.2. De-correlating MLLT
We attempt to apply MLLT to feature vectors so that correlations between the feature vector components are captured.
MLLT is computed for distributions of speech observations in
the HMMs of speech recognizers. The state-dependent transformations are estimated so that the likelihood of the adaptation
data is maximized. We learn the transformations given the initialized HMMs of our unsupervisedly trained acoustic model.

5.2. Evaluation

3.2.3. Speaker-adapting fMLLR

The evaluation metric we use to measure the cluster quality is based on the minimal pair ABX phone discriminability
task [19], which is related to the ABX task used in psychophysics [20]. Each cluster is considered a phone in the context
of the evaluation. We score GMM posteriorgrams that are computed for each speech frame after clustering, as described in
Section 2.3. Let A and B be stimuli belonging to sound categories a and b. The ABX phone discrimination accuracy is
XX X
1c(a, b) =|a| · |b| · (|a| − 1) A∈afMLLR is an algorithm for speaker adaptive training (SAT). The
idea of SAT is to capture inter-speaker variability in speaker dependent transforms and to generate speaker independent state
distributions instead. The transformations are estimated basedon alignments with speaker-independent features so that thelikelihoods are maximized. We apply fMLLR in this zero resource setting because we expect the transformations to help
eliminate variance caused by multiple speakers.

4. Clustering combinationB∈b X∈a\{A}1(δd(A,X)<d(B,X) + δd(A,X)=d(B,X) )(2)
2
where d(a, b) is the dynamic time warping (DTW) divergenceand δ is an indicator function. As in Schatz et al. [19], we use
the Kullback-Leibler divergence to compute the DTW divergences. Our scores are the error rates within and across speakers. The rates are averaged over all contexts for a given pair ofcentral phonemes and then over all pairs of central phonemes.

For further improvement of the DPGMM clustering quality wedeveloped a method to combine the results of n clustering runs,
which are sets of posteriorgrams. For each frame, we add together the n individual posteriorgrams and normalize the newvectors so that they form proper posteriorgrams again.
Because the amount of found classes differs for each clustering run, a mapping between any two sets of posteriorgrams isneeded. Given n sets of posteriorgrams, we randomly pick oneof these sets as target set, and consider all other sets as source
sets. We use the numeric frame-wise class labels as transcriptions for our data. For each source/target pair we first align thetranscriptions and count the co-occurrences of classes. Then wekeep the single most probable “translation” for each class andmap the posteriorgrams into the target space as shown by theexample in Figure 2.

5.3. Setup
We utilize the Kaldi speech recognition toolkit [21] to train the
acoustic model used in our framework by following a standard
scheme for speaker adaptive training.
We use the same parameters than Chen et al. [9] to ensurecomparability. DPGMM sampling is done for 1500 iterations,and the priors are set so that m0 is the global mean, S0 is theglobal covariance, κ0 = 1, and α = 1. The value of ν0 slightly
varies and is set to the toolkit’s default of ν0 = D + 3, where
D is the dimension of the input feature vectors. All feature vector types are extracted for a frame length of 25msec and frame
shift of 10msec. Mean variance normalization (MVN) and vocal tract length normalization (VTLN) is applied.

Algorithm 1 Combination of dynamically sized posteriorgrams
Require: Set P = {P1 , · · · , Pn } of sets of posteriorgrams
Ensure: Combined posteriorgrams P̂
1: ptgt ← random set from P
2: ltgt ← generate labels from posteriorgrams ptgt
3: P̂ ← ptgt
4: for all psrc ∈ P \ ptgt do
5:lsrc ← generate labels from posteriorgrams psrc
6:
count symbol pair occurrences in align(lsrc , ltgt )7:m ← 1-best mapping for all symbols in unique(lsrc )8:P̂ ← P̂ + map(psrc , ptgt , m)
9: end for
10: P̂ ← normalize P̂

5.4. Baseline
For our baseline we extract 39 dimensional MFCC+∆+∆∆ as
input to the DPGMM sampler. We also compare to the results ofChen et al. [9] as reference due to the identical clustering setup.
The details are listed in Table 1. Despite using the same sampling setup and input feature types, there is a mismatch between
the results of Chen et al. [9] and our baselines. We believe thismismatch is caused by the fact that Chen et al. uses a custom
voice activity detection for segmenting the full 10 hours of English data and does not mention any segmentation attempts forthe 5 hours of Xitsonga data, where we use the officially provided segmentation that limits both data sets to about half theamount. Due to the differences we start with a higher error rate
on English, but a lower error rate on Xitsonga.

Posteriorgram from source set: (0.00, 0.01, 0.15, 0.70, 0.09, 0.00, 0.04)
Class labels from source set:
0,
1,
2,
3,
4,
5,
6
1-best map:
2,
0,
1,
1,
3,
5,
4
Class labels mapped to target set:
0,
1,
2,
3,
4,
5
Posteriorgram in target space: (0.01, 0.85, 0.00, 0.09, 0.04, 0.00)

Figure 2: Example of a posteriorgram mapping.

1312

Table 1: The optimal results for each input feature type.

10.8 English (error within speakers)
10.6
10.4
10.2
10.0
9.8 (a) (b) (c) (d) (e)Combination scheme

Error rate

15.8 English (error across speakers)
15.6
15.4
15.2
15.0
14.8 (a) (b) (c) (d) (e)Combination scheme

12.6 Xitsonga (error across speakers)
12.4
12.2
12.0
11.8
11.6 (a) (b) (c) (d) (e)Combination scheme

Error rate

Error rate

Xitsonga
within across
9.6
17.2
8.9
14.2
8.5
13.9
8.3
12.8
8.4
12.9
8.4
12.2
8.1
11.7

Error rate

Features
MFCC+∆+∆∆ ([9])
MFCC+∆+∆∆ ([15])PLP+∆+∆∆
PLP+LDA
PLP+MLLT
PLP+MLLT+fMLLR
Best combination

Englishwithin across
10.8
16.3
12.2
19.5
11.8
19.6
10.5
16.1
10.5
16.2
10.6
15.7
10.0
14.9

8.8 Xitsonga (error within speakers)
8.6
8.4
8.2
8.0
7.8 (a) (b) (c) (d) (e)Combination scheme

5.5. Untransformed features
Figure 3: Error rates after combination of clustering results.
The dotted line is the best performance before combination.

In our previous work [15] we found that PLP feature vectors are
consistently leading to a higher clustering quality than MFCC
feature vectors. We therefore conducted all clustering experiments based on this feature type.

ficient scheme, boosting the discrimination quality across languages considerably. Combination across LDA output dimensions d ∈ {16, 20, 23, 26} also led to a better performanceacross the board, but to a lesser extent. These results let us
assume that higher dimensional coefficients still carry complimentary information that can help in a combination scheme, albeit performing worse for clustering in isolation [15].
Clustering one feature set five times with subsequent combination of the results had a slightly positive effect on English,
but was not helpful on Xitsonga. We assume the DPGMM cluster generally leads to consistent output and multiple parallel iterations are therefore not particularly useful.

5.6. Dimension-reducing LDA
The LDA transformation takes stacked standard feature vectors
without their derivatives as input. Following our findings in [15]we fix the stacking context parameter set to c = 4, and the output dimensionality to d = 20. With the application of LDA wewere able to produce feature vectors that considerably helped
the DPGMM clustering process to find better clusters. The
error rates for both languages dropped consistently, and especially across speakers a clear performance boost is observable.
LDA features outperform our own baseline and also undercut
the numbers of Chen et al. [9], thus compensating for the deficit
in the baseline numbers that we had to begin with.

6. Conclusion
We successfully utilized a supervised acoustic model training
pipeline without supervision to improve DPGMM based feature vector clustering. Feature transformations estimated during model training can be used to map speech feature vectors
into subspaces that are more suitable for clustering. Gaussian posteriorgrams extracted from a DPGMM that was sampled on transformed vectors carry better sound class discriminating characteristics than the ones sampled on untransformed
standard features. We showed that LDA greatly benefits soundclass discriminability within and across speakers. Consecutive
fMLLR transformation noticeably decreases the discrimination
error across speakers, proving the importance of speaker adaptation in solving the clustering task.
Combining multiple runs of LDA+MLLT+fMLLRtransformed feature vector clustering that use varying input
dimensionalities yielded the the best results. We achieved error
rates of 10% within and 14.9% across speakers for English,
and 8.1% within and 11.7% across speakers for Xitsonga,
respectively. Given these results, our proposed two-stage
clustering framework clearly outperforms our own baseline, as
well as the baseline set by Chen et al. [9].
Our two-staged clustering approach is particularly suitable for low-resource languages and the zero-resource scenario.
Moreover, this framework might as well be of help for moregeneral purposes beyond low-resource languages. In future
work we will explore the applicability of our model training
and clustering pipeline to solving other tasks beyond sound unit
detection.

5.7. De-correlating MLLT
Applying MLLT to the LDA transformed features did not leadto a better clustering quality. This lets us assume that the decorrelating effects might not be able to aid this particular task,
albeit being useful during a decoding task, as experience shows.
5.8. Speaker-adapting fMLLR
The transformations learned with fMLLR during the speaker
adaptive acoustic model training prove to be very useful forboosting the discrimination capabilities across speakers. A relative improvement of 3% for English and almost 6% for Tsonga
proves that fMLLR based speaker adaptive transformations canconsiderably improve clustering quality in the face of speakervariations and greatly benefit the clustering task.
5.9. Posteriorgram combination
We tested combining several clustering results across (a) transformations, (b) features, (c) LDA input and (d) output dimensionalities, and (e) given multiple identical clustering runs. The
results are plotted in Figure 3.
For the combination across transformations, we ran separate clusterings for each type of transformed features. However,this seems not particularly helpful. Discrimination errors forEnglish slightly drop, but increase for Xitsonga. As fMLLR isapplied to LDA+MLLT features, all useful information seems
already encapsulated in the fMLLR-transformed features. Similarly, combining separate MFCC and PLP feature clusteringresults led to lower errors on English, but not on Xitsonga.
Combining across LDA input dimensions, where the context parameter c ranges from 1 to 8, was by far the most ef-

7. Acknowledgements
Part of this research was supported by JSPS KAKENHI Grant
Number 24240032 and 26870371.

1313

8. References[19] T. Schatz, V. Peddinti, F. Bach, A. Jansen, H. Hermansky, and
E. Dupoux, “Evaluating speech features with the minimal-pair
ABX task: Analysis of the classical MFC/PLP pipeline,” in Proceedings of Interspeech, 2013.

[1] M. Versteegh, R. Thiolliere, T. Schatz, X. N. Cao, X. Anguera,
A. Jansen, and E. Dupoux, “The zero resource speech challenge
2015,” in Proceedings of Interspeech, 2015.

[20] N. A. Macmillan and C. D. Creelman, Detection theory: A user’s
guide. Psychology press, 2004, ch. 9.

[2] A. Park and J. Glass, “Towards unsupervised pattern discovery
in speech,” in Automatic Speech Recognition and Understanding,2005 IEEE Workshop on. IEEE, 2005, pp. 53–58.

[21] D. Povey, A. Ghoshal, G. Boulianne, N. Goel, M. Hannemann,
Y. Qian, P. Schwarz, and G. Stemmer, “The Kaldi speech recognition toolkit,” in Proceedings of IEEE, 2011.

[3] ——, “Unsupervised pattern discovery in speech,” Audio, Speech,
and Language Processing, IEEE Transactions on, vol. 16, no. 1,
pp. 186–197, 2008.
[4] B. Varadarajan, S. Khudanpur, and E. Dupoux, “Unsupervised
learning of acoustic sub-word units,” in Proceedings of the 46th
Annual Meeting of the Association for Computational Linguistics
on Human Language Technologies: Short Papers. Association
for Computational Linguistics, 2008, pp. 165–168.
[5] Y. Zhang and J. Glass, “Unsupervised spoken keyword spotting
via segmental DTW on Gaussian posteriorgrams,” in Automatic
Speech Recognition & Understanding, 2009. ASRU 2009. IEEE
Workshop on. IEEE, 2009, pp. 398–403.
[6] I. Malioutov, A. Park, R. Barzilay, and J. Glass, “Making senseof sound: Unsupervised topic segmentation over acoustic input,”in Association for Computational Linguistics Annual Meeting,vol. 45, no. 1. Citeseer, 2007, p. 504.
[7] M. Dredze, A. Jansen, G. Coppersmith, and K. Church, “NLP
on spoken documents without ASR,” in Proceedings of the 2010
Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2010, pp. 460–
470.
[8] H. Kamper, A. Jansen, S. King, and S. Goldwater, “Unsupervised
lexical clustering of speech segments using fixed-dimensional
acoustic embeddings,” in Spoken Language Technology Workshop(SLT), 2014 IEEE. IEEE, 2014, pp. 100–105.
[9] H. Chen, C.-C. Leung, L. Xie, B. Ma, and H. Li, “Parallel inference of dirichlet process gaussian mixture models for unsupervised acoustic modeling: A feasibility study,” in Proceedings ofInterspeech, 2015.
[10] R. A. Fisher, “The use of multiple measurements in taxonomic
problems,” Annals of eugenics, vol. 7, no. 2, pp. 179–188, 1936.
[11] R. A. Gopinath, “Maximum likelihood modeling with gaussian
distributions for classification,” in Acoustics, Speech and Signal
Processing, 1998. Proceedings of the 1998 IEEE International
Conference on, vol. 2. IEEE, 1998, pp. 661–664.
[12] M. J. Gales, “Semi-tied covariance matrices for hidden markov
models,” Speech and Audio Processing, IEEE Transactions on,
vol. 7, no. 3, pp. 272–281, 1999.
[13] T. Anastasakos, J. McDonough, R. Schwartz, and J. Makhoul,“A compact model for speaker-adaptive training,” in Spoken Language, 1996. ICSLP 96. Proceedings., Fourth International Conference on, vol. 2. IEEE, 1996, pp. 1137–1140.
[14] M. J. Gales, “Maximum likelihood linear transformations for
hmm-based speech recognition,” Computer speech & language,
vol. 12, no. 2, pp. 75–98, 1998.
[15] M. Heck, S. Sakti, and S. Nakamura, “Unsupervised linear discriminant analysis for supporting DPGMM clustering in the zero
resource scenario,” in Proceedings of SLTU, 2016.
[16] C. Ding and T. Li, “Adaptive dimension reduction using discriminant analysis and k-means clustering,” in Proceedings of the 24thinternational conference on Machine learning. ACM, 2007, pp.
521–528.
[17] J. Tang, X. Hu, H. Gao, and H. Liu, “Discriminant analysis for
unsupervised feature selection.” in SDM. SIAM, 2014, pp. 938–946.
[18] J. Chang and J. W. Fisher III, “Parallel sampling of dp mixture
models using sub-cluster splits,” in Advances in Neural Information Processing Systems, 2013, pp. 620–628.

1314