Available online at www.sciencedirect.com

ScienceDirect
Procedia Computer Science 81 (2016) 73 – 79

5th Workshop on Spoken Language Technology for Under-resourced Languages, SLTU 2016,9-12 May 2016, Yogyakarta, Indonesia

Unsupervised Linear Discriminant Analysis for Supporting
DPGMM Clustering in the Zero Resource Scenario
Michael Heck∗, Sakriani Sakti, Satoshi Nakamura
Augmented Human Communication Laboratory, Graduate School of Information Science, Nara Institute of Science and Technology,
8916-5 Takayama-cho, Ikoma, Nara 630-0192, Japan

Abstract
In this work we make use of unsupervised linear discriminant analysis (LDA) to support acoustic unit discovery in a zero resource
scenario. The idea is to automatically ﬁnd a mapping of feature vectors into a subspace that is more suitable for Dirichlet process
Gaussian mixture model (DPGMM) based clustering, without the need of supervision. Supervised acoustic modeling typicallymakes use of feature transformations such as LDA to minimize intra-class discriminability, to maximize inter-class discriminability
and to extract relevant informations from high-dimensional features spanning larger contexts. The need of class labels makes it
diﬃcult to use this technique in a zero resource setting where the classes and even their amount are unknown. To overcome thisissue we use a ﬁrst iteration of DPGMM clustering on standard features to generate labels for the data, that serve as basis forlearning a proper transformation. A second clustering operates on the transformed features. The application of unsupervised LDA
demonstrably leads to better clustering results given the unsupervised data. We show that the improved input features consistently
outperform our baseline input features.
c 2016
2016The
TheAuthors.
Authors.
Publishedby Elsevier©
Publishedby Elsevier
B.V.B.V.
This is an open access article under the CC BY-NC-ND license(http://creativecommons.org/licenses/by-nc-nd/4.0/).
Peer-review under responsibility of the Organizing Committee of SLTU 2016.
Peer-review under responsibility of the Organizing Committee of SLTU 2016
Keywords: acoustic unit discovery; Bayesian nonparametrics; Dirichlet process; feature transformation; Gibbs sampling; unsupervised linear
discriminant analysis; zero resource

1. Introduction
In a zero resource scenario, large amounts of labeled training data, parallel data, and knowledge about the target
language are unavailable for developing speech processing systems with supervised techniques. Where infants arecapable of robustly modeling acoustic and language models in an unsupervised way, current speech technology is not
yet capable to imitate these capacities.
Confronted with an unknown language, human experts usually attempt to deﬁne a set of acoustic units that fully
covers the underlying sound repertoire. Core techniques of machine learning approaches to this task are pattern∗ Corresponding author. Tel.: +81-743-72-5265 ; fax: +81-743-72-5269.

E-mail address: michael-h@is.naist.jp

1877-0509 © 2016 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license(http://creativecommons.org/licenses/by-nc-nd/4.0/).
Peer-review under responsibility of the Organizing Committee of SLTU 2016
doi:10.1016/j.procs.2016.04.032

74Michael Heck et al. / Procedia Computer Science 81 (2016) 73 – 79

matching 1,2 on raw audio data and unsupervised sound unit detection 3 and have already been successfully applied tosolve tasks such as spoken term detection 4 , topic segmentation 5 or document classiﬁcation 6 .
In non-clinical situations where development data is usually unavailable, model complexity is not known a priori.
Bayesian models such as the Dirichlet process Gaussian mixture model (DPGMM) automatically adjust the model
complexity given the data. DPGMMs have been successfully applied to speech processing tasks such as unsupervised
lexical clustering 7 . Previous work 8 clustered standard MFCC speech features by inferring a DPGMM. Each Gaussian
was interpreted as modeling a speciﬁc sound class. The posteriorgrams were evaluated to show that DPGMM is asuitable technique to automatically detect sound classes in untranscribed data.
It is straightforward to assume that more advanced feature representations may lead to a better classiﬁer performance. For instance, context information is an important factor to correctly classify speech features in common speechprocessing systems. Chen et al. 8 use MFCC features with ﬁrst and second derivatives for clustering. The derivatives
help cover a small context but triple the dimensionality. Feature stacking can cover a much larger context, but at
signiﬁcantly higher expenses in terms of dimensionality. A feasible processing of high-dimensional feature vectors
makes dimension reducing feature transformations mandatory.
Traditional supervised acoustic modeling typically makes use of feature transformations such as linear discriminant analysis (LDA) 9 to minimize intra-class discriminability, to maximize inter-class discriminability and to extract
relevant informations from high-dimensional features spanning larger contexts. Class discriminating properties offeature vectors are critical for clustering. However, LDA needs class labels to estimate the feature transformations,making it diﬃcult to use in a zero resource setting where the classes and even their amount are unknown.
In this work we attempt to improve the DPGMM clustering by introducing unsupervised LDA to the sampling
pipeline. There has been work that utilize k-means clustering to automatically obtain pseudo labels for LDA estimation 10,11 . We similarly attempt to automatically produce class labels, but we want to overcome the limitation of having
to predeﬁne the size of the label set. For that, we use a non-parametric DPGMM sampler to generate labels for our untranscribed data. Our contribution is an easy to understand two-staged clustering framework that automatically ﬁnds
a dynamically sized set of framewise class labels for unsupervised LDA transformation to project high-dimensionallarge context covering feature vectors into a more suitable subspace for DPGMM clustering.
2. Dirichlet Process Gaussian Mixture Model
DPGMMs (also known as inﬁnite GMMs) extend ﬁnite mixture models by the aspect of automatic model selection:
The model ﬁnds it’s complexity automatically given the training data. Inference is typically sample based usinga Markov chain Monte Carlo (MCMC) scheme such as Gibbs sampling. The sampler used here is combining a
restricted Gibbs sampler with a split/merge sampler. For more in-depth informations, please refer to 12,8 .
2.1. Generative process
Let X = x1 , · · · , xn be a set of observations. The generative process of X given a DPGMM is as follows:• Mixing weights π = {π1 , · · · , πk } are generated according to a stick-breaking process• GMM parameters θ = {θ1 , · · · , θk } are generated according to a prior distribution NIW(mk , S k , κk , νk )• A label zi is assigned to every xi , according to π
• A data point xi is generated according to the zi -th Gaussian component
θk = {μk , Σk } are Gaussian parameters, and the parameter set of the prior Normal-inverse-Wishart (NIW) distribution consists of a prior m0 for μk , a prior S 0 for Σk , the belief-strength κ0 in m0 and the belief-strength ν0 in S 0 .
2.2. Inference
The parallelizable sampler used here alternates between a non-ergodic restricted Gibbs sampler and a split/merge
sampler to form an ergodic MCMC sampler.

75

Michael Heck et al. / Procedia Computer Science 81 (2016) 73 – 79
stacked feature vectors

class labels
DPGMM
clustering

LDA-transformed feature vectors
LDA
transformation

posteriorgrams
ABX
scoring

DPGMM
clustering

Fig. 1. Scheme of the sampling process. The dimensionality of the stacked feature vectors x̂i is 13(2c + 1), where c is the stacking context. The
dimensionality of the LDA-transformed feature vectors yi is d  13(2c + 1).

Restricted Gibbs sampling allows labels zi to be sampled from a ﬁnite set Z. By deﬁnition of the DPGMM, thedistribution of the mixture weights follows a Dirichlet distribution.
Split/merge sampling performs operations on the existing components. To provide good split candidates, eachcomponent is augmented with two sub-clusters with mixing weights πk,l , πk,r and parameter sets θk,l , θk,r , and eachobservation of a component is augmented with a sub-cluster label z subi ∈ l, r.
The Split/merge sampler proposes split and merge moves in a Metropolis-Hastings fashion. A Hastings ratio H iscomputed according to the momentary assignment of observations of a component to it’s sub-clusters, and a move is
accepted with a probability min(1, H). For the merge step, merges of randomly picked components are proposed.
2.3. Posteriorgram generation
The posterior probability of cluster ck , given observation xi is
πk N(x|θk )p(ck |xi ) = N(x|θ j )(1)

and Pi = (p(c1 |xi ), · · · , p(cK |xi )) forms a posteriorgram for observation xi .
3. Unsupervised Linear Discriminant Analysis
To improve the quality of DPGMM based speech feature vector clustering, we propose to utilize LDA in an unsupervised fashion. The idea is to automatically ﬁnd a projection that maps high-dimensional feature vectors into a moresuitable space for DPGMM clustering. Using LDA for feature transformation is motivated by several reasons: Firstof all, LDA is a simple linear transformation and a standard technique that attempts to minimize intra-class discriminability and to maximize inter-class discriminability of speech features. As such, LDA is widely used in automatic
speech processing systems. Naturally, class discriminating properties would greatly beneﬁt clustering approaches foracoustic unit discovery. Secondly, LDA allows for dimensional reduction of high-dimensional stacked feature vectorsthat span a larger context by omitting lower-ranked coeﬃcients. Thirdly, LDA transformations are fast and easy tocompute and only require the feature vectors and respective class labels.
The need of class labels makes it diﬃcult to use LDA in a zero resource setting out of the box, since the class
identities and even the amount of classes are unknown, let alone the class memberships of our feature vectors. To
overcome this issue, we propose to use a two-staged clustering framework that automatically ﬁnds frame-based classlabels in a ﬁrst clustering run on the untranscribed data, performs LDA estimation and feature transformation and runs
a second run of clustering on the transformed vectors. Figure 1 is a graphical overview of the proposed framework.
The DPGMM is a Bayesian non-parametric model that automatically detects the optimal number of classes given
a set of data. We make use of this property and run an initial clustering on standard feature vectors to get a set of classlabels and the hypothesized class membership of every speech frame. These classes are generic and simply named

76

Michael Heck et al. / Procedia Computer Science 81 (2016) 73 – 79

with the numeric ID of the Gaussian that most likely produced the respective feature vector. Having the labels forevery frame at hand, we can simply estimate an LDA transformation.
Context is an important source of information to correctly classify speech features. Feature stacking can cover amuch larger context than appending the ﬁrst and second derivatives. Thus, we compute the LDA for stacked MFCC
feature vectors, where we use a context of c, meaning that we stack the c left and c right feature vectors on top of thecurrent vector, which is the center vector. To keep the dimensionality low for any feasible clustering, we reduce theoutput dimension d of the LDA transformation to a signiﬁcantly lower value than the input dimension.
Once we produced new feature vectors by using the LDA transformation, we perform another run of DPGMM
based clustering on these. The class discriminating properties of the LDA are assumed to support the clustering andlead to a better cluster quality. We demonstrate in Section 4 that the LDA in fact signiﬁcantly boosts the clustering
quality.
4. Experiments
4.1. Data
All our experiments were conducted on the oﬃcial data set of the Interspeech zero resource speech challenge 13 ,which contains two separate data sets of pure speech for American English (4h 59min) and Xitsonga (2h 29min), asouthern African Bantu language. Segments contain no overlapping speech, noise or pauses, and speech of exactly
one speaker. The English data is extracted from the Buckeye corpus and consists of conversational speech. The
Xitsonga data is an excerpt of the NCHLT corpus and is comprised of read speech.
4.2. Evaluation
The evaluation metric used to measure the cluster quality is based on the minimal pair ABX phone discriminability
task 14 . After clustering, GMM posteriorgrams can be computed for each speech frame, as described in Section 2.3.
Evaluation is performed on these posteriorgrams, that serve as new speech representations for each frame. Let A and
B be stimuli belonging to sound categories a and b. Then the ABX phone discrimination accuracy isc(a, b) =  11
δd(A,X)<d(B,X) + δd(A,X)=d(B,X)|a| · |b| · (|a| − 1) A∈a B∈b X∈a\{A}2(2)

where d(a, b) is the DTW divergence and δ is an indicator function. We followed Schatz et al. 14 and used the cosine
distance for standard features and KL-divergence for posteriorgrams to compute the DTW divergences. Our scores
are the error rates within and across speakers, where the rates are averaged over all found contexts for a given pair ofcentral phonemes and then over all pairs of central phonemes.
4.3. Setup
To get as close as possible to the setup of Chen et al. 8 we use the same parameters. Thus, DPGMM sampling isdone for 1500 iterations, and the priors are set so that m0 is the global mean, S 0 is the global covariance, κ0 = 1 and
α = 1. The value of ν0 slightly varies and is set to the toolkit’s default of ν0 = D + 3, where D is the dimension of the
input feature vectors. All feature vector types are extracted for a frame length of 25 milliseconds and frame shift of
10 milliseconds and make use of mean variance normalization (MVN) and vocal tract length normalization (VTLN).
4.4. Baseline
As a baseline we extract 39 dimensional MFCC+Δ+ΔΔ as input to the DPGMM sampler. We test both the ABX
discrimination error of the raw features as well as of the posteriorgrams as result of the DPGMM clustering. We also
compare to the oﬃcial results of Chen et al. 8 , the details are listed in Table 1.
Despite using the same sampling setup and input feature types, there is a mismatch between the results of Chenet al. 8 and our baselines. We believe this mismatch is caused by the fact they use a custom voice activity detection

77

Michael Heck et al. / Procedia Computer Science 81 (2016) 73 – 79

Table 1. The baselines for this work are the results of Chen et al. 8 and our own numbers produced with the oﬃcial data segmentation. DPGPG
stands for Dirichlet process Gaussian posteriorgrams.
System
MFCC+Δ+ΔΔ 8
DPGPG 8
MFCC+Δ+ΔΔ
DPGPG

EnglishwithinacrossXitsonga
withinacross

17.2
10.8
15.7
12.2

26.8
16.3
25.5
19.5

19.6
9.6
19.7
8.9

30.8
17.2
30.0
14.2

for segmenting the full 10 hours of English data and do not mention any segmentation attempts for the 5 hours of
Xitsonga data, where we use the oﬃcially provided segmentation that limits both data sets to about half the amount.
Due to the diﬀerences we start with a higher error rate on English, but a lower error rate on Xitsonga.
4.5. PCA vs. LDA
We started our experiments by doing a comparison of LDA and the closely related principal component analysis(PCA) 15,16 in order to ﬁnd out whether simpler and entirely unsupervised techniques could serve us better than the
proposed approach. PCA is a simple orthogonal linear transformation to de-correlate variables. The diﬀerence to
LDA is that the class memberships of the features subject to transformation are not taken into consideration. The data
is simply mapped to a new coordinate system so that the ﬁrst principal component has the largest variance, the second
component the second largest, and so on. Dimensional reduction is performed in a similar way to LDA. The results
are displayed in ﬁgure 2.
The transformations take stacked standard feature vectors without their derivatives as input. With the stacking
context parameter set to c = 4, and the output dimensionality set to d = 20, we found that the application of PCA led
to opposite results for the two target languages. Where the transformation helped improve the clustering process ofEnglish speech feature vectors according to both error rates, within and across speakers, using PCA did not help beat
the baseline for the Xitsonga data.
The LDA transformation on the other hand was able to produce feature vectors that signiﬁcantly helped the
DPGMM clustering process to ﬁnd better clusters. The error rates for both languages dropped consistently, andespecially across speakers a clear performance boost is observable. By using the LDA transformed features, it waspossible to outperform our own baseline, and the error rates for English also undercut the numbers of Chen et al. 8 ,thus compensating for the deﬁcit in the baseline numbers that we had to begin with.
4.6. Input features for LDA
One of the advantages of LDA is that it can be applied to a multitude of input features. To ﬁnd out whether there
is a better choice than MFCCs, we conducted experiments on perceptual linear prediction (PLP) feature vectors forcomparison. The PLP features produced evaluation errors comparable to and slightly lower than the baseline set by
MFCC: By using PLP+Δ+ΔΔ as input to the sampler we observed improvements for both languages given the error
rate within speakers. We have seen an error rate decrease across speakers for Xitsonga and a stable error rate acrossspeakers for English. When using PCA and LDA transformed PLP features, the latter proved to generate the lower
error rates in all comparisons. No matter whether MFCC or PLP features served as input, PCA was inferior to LDA,
leading us to the joint conclusion that unsupervised LDA is the better choice for mapping features into a lower spaceto perform clustering, and that PLP feature vectors seem to carry some information that beneﬁts class discriminability.
Again, the results are plotted in ﬁgure 2.
4.7. Input and output dimensions for LDA
The stacking context parameter c directly regulates the input dimensionality of the LDA transformation. The
output dimensionality of LDA can be adjusted by setting the parameter d. Since the right choice of input and output
dimensionality of the LDA transformation might be critical to the clustering performance, we conducted a grid search

78

Michael Heck et al. / Procedia Computer Science 81 (2016) 73 – 79

st acked MFCC + PCA
st acked MFCC + LDA

st acked PLP + PCA
st acked PLP + LDA

cont ext 2
cont ext 3

cont ext 4
cont ext 5

cont ext 6
cont ext 7

cont ext 8

(a)

(c)

(a)

(c)

(b)

(d)

(b)

(d)

Fig. 2. Error rates across and within speakers in dependency of feature Fig. 3. Error rates across and within speakers in dependency of stacking
type and transformation, computed for several output dimensionalities. context size and the LDA output dimensionality. (a)-(b) Error rates for(a)-(b) Error rates for English, (c)-(d) Error rates for Xitsonga.
English, (c)-(d) Error rates for Xitsongaon these two parameters. We restricted these tests to PLP features only, given our previous ﬁndings. The results ofour tests are visualized in ﬁgure 3.
It is noteworthy that the usefulness of the LDA transformation does not seem to depend on the stacking context
size. Our results demonstrate that any context larger than 2 is a suitable choice for the transformation estimation. The
output dimensionality of the transformation however does have an impact on performance. Increasing dimensionality
has an opposite eﬀect on the two target languages. Where Xitsonga beneﬁts from a higher output dimensionality, best
results for English are produced using a lower dimensionality. This eﬀect might be attributable to two circumstances:
Firstly, it is to point out that the data sets diﬀer in nature. The English data is comprised of conversational speech, thus
a transformation that maps into a signiﬁcantly lower dimensional space might produce more stable features suitable
for the DPGMM clustering. The Xitsonga data on the other hand has the characteristics of read speech, thus the higher
dimensions of the transformed features may still be able to carry useful information for sound class discrimination.
Secondly, having a look at the manually crafted phoneme sets of the original corpora (Buckeye for English, NCHLT
for Xitsonga), one ﬁnds that the optimal output dimensionalities for the respective data sets roughly correlate withthe set sizes. Where the English data is originally labeled with a 39-elemental phoneme set, and the optimal output
dimension in our experiments is located around 20, the Xitsonga data is labeled with 53 distinct tokens, and the
optimal output dimension seems to lie around 26. The results further allow to assume that class discrimination acrossspeakers beneﬁts from a slightly higher dimensionality than the within speaker class discrimination. We believe thisis because more dimensions might be necessary to discriminate classes including speaker variations.
For in-the-ﬁeld experiments we would naturally either have to apply parameters tuned on an out-of-language development set or parameters that proved to work well for solving similar tasks. The results displayed in ﬁgure 3 showthat it is possible to ﬁnd good parameters for a known language that work considerably well for a new language: By
using the parameters we tuned on English, we achieve a sound discrimination quality that is only slightly lower thanthe one that could be achieved with an optimal parameter set. Table 2 lists the best results of our tests.
5. Conclusion
We were able to demonstrate that dimension reducing unsupervisedly estimated LDA is an eﬃcient way to map
speech feature vectors into a subspace that is more suitable for DPGMM based clustering. The Gaussian posteriorgrams that can be extracted from a DPGMM sampled on transformed vectors carry better sound class discriminating
characteristics than the ones sampled on untransformed standard features. We showed that unsupervised LDA esti-

79

Michael Heck et al. / Procedia Computer Science 81 (2016) 73 – 79

Table 2. The optimal results for each input feature type.
Features

EnglishwithinacrossXitsonga
withinacross

MFCC
MFCC+PCA
MFCC+LDAPLP
PLP+PCAPLP+LDA

12.2
11.7
11.1
11.8
11.7
10.6

19.5
19.2
16.6
19.6
18.4
16.0

8.9
9.8
8.4
8.5
8.7
8.0

14.2
16.4
12.9
13.9
14.6
12.6

mation based on automatically generated labels works reliably across languages and clustering performance is fairly
independent of the stacking context prior to LDA computation. We further showed that the output dimensionality ofthe transformation does inﬂuence clustering quality, and even with a value that has been tuned on an out-of-language
test data set, a near optimal clustering quality can be achieved.
The results demonstrate that unsupervised LDA transformation supported DPGMM clustering is particularly suitable for low-resource languages and the zero-resource scenario. It is easy to conclude that this approach might beof help for more general purposes beyond low-resource languages. In future work, we will continue our research on
unsupervised feature transformations by exploring the usefulness of the proposed approach for tackling other tasks
such as unsupervised acoustic model training.
Acknowledgements
Part of this research was supported by JSPS KAKENHI Grant Number 24240032 and 26870371.
References
1. Park, A., Glass, J.. Towards unsupervised pattern discovery in speech. In: Workshop on Automatic Speech Recognition and Understanding.
IEEE; 2005, p. 53–58.
2. Park, A., Glass, J.. Unsupervised pattern discovery in speech. IEEE Transactions on Audio, Speech, and Language Processing 2008;
16(1):186–197.
3. Varadarajan, B., Khudanpur, S., Dupoux, E.. Unsupervised learning of acoustic sub-word units. In: Proceedings of the 46th Annual
Meeting of the Association for Computational Linguistics on Human Language Technologies: Short Papers. Association for Computational
Linguistics; 2008, p. 165–168.
4. Zhang, Y., Glass, J.. Unsupervised spoken keyword spotting via segmental DTW on Gaussian posteriorgrams. In: Workshop on Automatic
Speech Recognition & Understanding (ASRU). IEEE; 2009, p. 398–403.
5. Malioutov, I., Park, A., Barzilay, R., Glass, J.. Making sense of sound: Unsupervised topic segmentation over acoustic input. In:
Association for Computational Linguistics Annual Meeting; vol. 45. Citeseer; 2007, p. 504.
6. Dredze, M., Jansen, A., Coppersmith, G., Church, K.. NLP on spoken documents without ASR. In: Proceedings of the 2010 Conferenceon Empirical Methods in Natural Language Processing. Association for Computational Linguistics; 2010, p. 460–470.
7. Kamper, H., Jansen, A., King, S., Goldwater, S.. Unsupervised lexical clustering of speech segments using ﬁxed-dimensional acoustic
embeddings. In: Spoken Language Technology Workshop (SLT). IEEE; 2014, p. 100–105.
8. Chen, H., Leung, C.C., Xie, L., Ma, B., Li, H.. Parallel inference of dirichlet process gaussian mixture models for unsupervised acoustic
modeling: A feasibility study. In: Proceedings of Interspeech. 2015.
9. Fisher, R.A.. The use of multiple measurements in taxonomic problems. Annals of eugenics 1936;7(2):179–188.
10. Ding, C., Li, T.. Adaptive dimension reduction using discriminant analysis and k-means clustering. In: Proceedings of the 24th international
conference on Machine learning. ACM; 2007, p. 521–528.
11. Tang, J., Hu, X., Gao, H., Liu, H.. Discriminant analysis for unsupervised feature selection. In: SDM. SIAM; 2014, p. 938–946.
12. Chang, J., Fisher III, J.W.. Parallel sampling of dp mixture models using sub-cluster splits. In: Advances in Neural Information Processing
Systems. 2013, p. 620–628.
13. Versteegh, M., Thiolliere, R., Schatz, T., Cao, X.N., Anguera, X., Jansen, A., et al. The zero resource speech challenge 2015. In:
Proceedings of Interspeech. 2015.
14. Schatz, T., Peddinti, V., Bach, F., Jansen, A., Hermansky, H., Dupoux, E.. Evaluating speech features with the minimal-pair ABX task:
Analysis of the classical MFC/PLP pipeline. In: Proceedings of Interspeech. 2013.
15. Pearson, K.. On lines and planes of closest ﬁt to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine
and Journal of Science 1901;2(11):559–572.
16. Hotelling, H.. Analysis of a complex of statistical variables into principal components. Journal of educational psychology 1933;24(6):417.