Bayesian Models for Unit Discovery on a Very Low Resource Language

Abstract: Developing speech technologies for low-resource languages has become a very active research field over the last decade. Among others, Bayesian models have shown some promising results on artificial examples but still lack of in situ experiments. Our work applies state-of-the-art Bayesian models to unsupervised Acoustic Unit Discovery (AUD) in a real low-resource language scenario. We also show that Bayesian models can naturally integrate information from other resourceful languages by means of informative prior leading to more consistent discovered units. Finally, discovered acoustic units are used, either as the 1-best sequence or as a lattice, to perform word segmentation. Word segmentation results show that this Bayesian approach clearly outperforms a Segmental-DTW baseline on the same corpus.

Out of nearly 7000 languages spoken worldwide, current speech (ASR, TTS, voice search, etc.) technologies barely address 200 of them. Broadening ASR technologies to ideally all possible languages is a challenge with very high stakes in many areas and is at the heart of several fundamental research problems ranging from psycholinguistic (how humans learn to recognize speech) to pure machine learning (how to extract knowledge from unlabeled data). The present work focuses on the narrow but important problem of unsupervised Acoustic Unit Discovery (AUD). It takes place as the continuation of an ongoing effort to develop a Bayesian model suitable for this task, which stems from the seminal work of BIBREF0 later refined and made scalable in BIBREF1 . This model, while rather crude, has shown that it can provide a clustering accurate enough to be used in topic identification of spoken document in unknown languages BIBREF2 . It was also shown that this model can be further improved by incorporating a Bayesian "phonotactic" language model learned jointly with the acoustic units BIBREF3 . Finally, following the work in BIBREF4 it has been combined successfully with variational auto-encoders leading to a model combining the potential of both deep neural networks and Bayesian models BIBREF5 . The contribution of this work is threefold:
The AUD model described in BIBREF0 , BIBREF1 is a non-parametric Bayesian Hidden Markov Model (HMM). This model is topologically equivalent to a phone-loop model with two major differences:
In this work, we have used two variants of this original model. The first one (called HMM model in the remainder of this paper), following the analysis led in BIBREF8 , approximates the Dirichlet Process prior by a mere symmetric Dirichlet prior. This approximation, while retaining the sparsity constraint, avoids the complication of dealing with the variational treatment of the stick breaking process frequent in Bayesian non-parametric models. The second variant, which we shall denote Structured Variational AutoEncoder (SVAE) AUD, is based upon the work of BIBREF4 and embeds the HMM model into the Variational AutoEncoder framework BIBREF9 . A very similar version of the SVAE for AUD was developed independently and presented in BIBREF5 . The main noteworthy difference between BIBREF5 and our model is that we consider a fully Bayesian version of the HMM embedded in the VAE; and the posterior distribution and the VAE parameters are trained jointly using the Stochastic Variational Bayes BIBREF4 , BIBREF10 . For both variants, the prior over the HMM parameters were set to the conjugate of the likelihood density: Normal-Gamma prior for the mean and variance of the Gaussian components, symmetric Dirichlet prior over the HMM's state mixture's weights and symmetric Dirichlet prior over the acoustic units' weights. For the case of the uninformative prior, the prior was set to be vague prior with one pseudo-observation BIBREF11 .
Bayesian Inference differs from other machine learning techniques by introducing a distribution INLINEFORM0 over the parameters of the model. A major concern in Bayesian Inference is usually to define a prior that makes as little assumption as possible. Such a prior is usually known as uninformative prior. Having a completely uninformative prior has the practical advantage that the prior distribution will have a minimal impact on the outcome of the inference leading to a model which bases its prediction purely and solely on the data. In the present work, we aim at the opposite behavior, we wish our AUD model to learn phone-like units from the unlabeled speech data of a target language given the knowledge that was previously accumulated from another resourceful language. More formally, the original AUD model training consists in estimate the a posteriori distribution of the parameters given the unlabeled speech data of a target language INLINEFORM1 : DISPLAYFORM0 
The parameters are divided into two subgroups INLINEFORM0 where INLINEFORM1 are the global parameters of the model, and INLINEFORM2 are the latent variables which, in our case, correspond to the sequences of acoustic units. The global parameters are separated into two independent subsets : INLINEFORM3 , corresponding to the acoustic parameters ( INLINEFORM4 ) and the "phonotactic" language model parameters ( INLINEFORM5 ). Replacing INLINEFORM6 and following the conditional independence of the variable induced by the model (see BIBREF1 for details) leads to: DISPLAYFORM0 
If we further assume that we have at our disposal speech data in a different language than the target one, denoted INLINEFORM0 , along with its phonetic transcription INLINEFORM1 , it is then straightforward to show that: DISPLAYFORM0 
 which is the same as Eq. EQREF8 but for the distribution of the acoustic parameters which is based on the data of the resourceful language. In contrast of the term uninformative prior we denote INLINEFORM0 as an informative prior. As illustrated by Eq. EQREF9 , a characteristic of Bayesian inference is that it naturally leads to a sequential inference. Therefore, model training can be summarized as:
Practically, the computation of the informative prior as well as the final posterior distribution is intractable and we seek for an approximation by means of the well known Variational Bayes Inference BIBREF12 . The approximate informative prior INLINEFORM0 is estimated by optimizing the variational lower bound of the evidence of the prior data INLINEFORM1 : DISPLAYFORM0 
where INLINEFORM0 is the Kullback-Leibler divergence. Then, the posterior distribution of the parameters given the target data INLINEFORM1 can be estimated by optimizing the evidence of the target data INLINEFORM2 : DISPLAYFORM0 
Note that when the model is trained with an uninformative prior the loss function is the as in Eq. EQREF13 but with INLINEFORM0 instead of the INLINEFORM1 . For the case of the uninformative prior, the Variational Bayes Inference was initialized as described in BIBREF1 . In the informative prior case, we initialized the algorithm by setting INLINEFORM2 .
We used the Mboshi5K corpus BIBREF13 as a test set for all the experiments reported here. Mboshi (Bantu C25) is a typical Bantu language spoken in Congo-Brazzaville. It is one of the languages documented by the BULB (Breaking the Unwritten Language Barrier) project BIBREF14 . This speech dataset was collected following a real language documentation scenario, using Lig_Aikuma, a mobile app specifically dedicated to fieldwork language documentation, which works both on android powered smartphones and tablets BIBREF15 . The corpus is multilingual (5130 Mboshi speech utterances aligned to French text) and contains linguists' transcriptions in Mboshi (in the form of a non-standard graphemic form close to the language phonology). It is also enriched with automatic forced-alignment between speech and transcriptions. The dataset is made available to the research community. More details on this corpus can be found in BIBREF13 .
TIMIT is also used as an extra speech corpus to train the informative prior. We used two different set of features: the mean normalized MFCC + INLINEFORM0 + INLINEFORM1 generated by HTK and the Multilingual BottleNeck (MBN) features BIBREF16 trained on the Czech, German, Portuguese, Russian, Spanish, Turkish and Vietnamese data of the Global Phone database.
To evaluate our work we measured how the discovered units compared to the forced aligned phones in term of segmentation and information. The accuracy of the segmentation was measured in term of Precision, Recall and F-score. If a unit boundary occurs at the same time (+/- 10ms) of an actual phone boundary it is considered as a true positive, otherwise it is considered to be a false positive. If no match is found with a true phone boundary, this is considered to be a false negative. The consistency of the units was evaluated in term of normalized mutual information (NMI - see BIBREF1 , BIBREF3 , BIBREF5 for details) which measures the statistical dependency between the units and the forced aligned phones. A NMI of 0 % means that the units are completely independent of the phones whereas a NMI of 100 % indicates that the actual phones could be retrieved without error given the sequence of discovered units.
In order to provide an extrinsic metric to evaluate the quality of the acoustic units discovered by our different methods, we performed an unsupervised word segmentation task on the acoustic units sequences, and evaluated the accuracy of the discovered word boundaries. We also wanted to experiment using lattices as an input for the word segmentation task, instead of using single sequences of units, so as to better mitigate the uncertainty of the AUD task and provide a companion metric that would be more robust to noise. A model capable of performing word segmentation both on lattices and text sequences was introduced by BIBREF6 . Building on the work of BIBREF17 , BIBREF18 they combine a nested hierarchical Pitman-Yor language model with a Weighted Finite State Transducer approach. Both for lattices and acoustic units sequences, we use the implementation of the authors with a bigram language model and a unigram character model. Word discovery is evaluated using the Boundary metric from the Zero Resource Challenge 2017 BIBREF20 and BIBREF21 . This metric measures the quality of a word segmentation and the discovered boundaries with respect to a gold corpus (Precision, Recall and F-score are computed).
First, we evaluated the standard HMM model with an uninformative prior (this will be our baseline) for the two different input features: MFCC (and derivatives) and MBN. Results are shown in Table TABREF20 . Surprisingly, the MBN features perform relatively poorly compared to the standard MFCC. These results are contradictory to those reported in BIBREF3 . Two factors may explain this discrepancy: the Mboshi5k data being different from the training data of the MBN neural network, the neural network may not generalize well. Another possibility may be that the initialization scheme of the model is not suitable for this type of features. Indeed, Variational Bayesian Inference algorithm converges only to a local optimum of the objective function and is therefore dependent of the initialization. We believe the second explanation is the more likely since, as we shall see shortly, the best results in term of word segmentation and NMI are eventually obtained with the MBN features when the inference is done with the informative prior. Next, we compared the HMM and the SVAE models when trained with an uninformative prior (lines with "Inf. Prior" set to "no" in Table TABREF23 ). The SVAE significantly improves the NMI and the precision showing that it extracts more consistent units than the HMM model. However, it also degrades the segmentation in terms of recall. We further investigated this behavior by looking at the duration of the units found by both models compared to the true phones (Table TABREF22 ). We observe that the SVAE model favors longer units than the HMM model hence leading to fewer boundaries and consequently smaller recall.
We then evaluated the effect of the informative prior on the acoustic unit discovery (Table TABREF23 ). On all 4 combinations (2 features sets INLINEFORM0 2 models) we observe an improvement in terms of precision and NMI but a degradation of the recall. This result is encouraging since the informative prior was trained on English data (TIMIT) which is very different from Mboshi. Indeed, this suggests that even speech from an unrelated language can be of some help in the design of an ASR for a very low resource language. Finally, similarly to the SVAE/HMM case described above, we found that the degradation of the recall is due to longer units discovered for models with an informative prior (numbers omitted due to lack of space).
Word discovery results are given in Table TABREF21 for the Boundary metric BIBREF20 , BIBREF21 . We observe that i) the best word boundary detection (F-score) is obtained with MBN features, an informative prior and the SVAE model; this confirms the results of table TABREF23 and shows that better AUD leads to better word segmentation ii) word segmentation from AUD graph Lattices is slightly better than from flat sequences of AUD symbols (1-best); iii) our results outperform a pure speech based baseline based on segmental DTW BIBREF22 (F-score of 19.3% on the exact same corpus).
We have conducted an analysis of the state-of-the-art Bayesian approach for acoustic unit discovery on a real case of low-resource language. This analysis was focused on the quality of the discovered units compared to the gold standard phone alignments. Outcomes of the analysis are i) the combination of neural network and Bayesian model (SVAE) yields a significant improvement in the AUD in term of consistency ii) Bayesian models can naturally embed information from a resourceful language and consequently improve the consistency of the discovered units. Finally, we hope this work can serve as a baseline for future research on unsupervised acoustic unit discovery in very low resource scenarios.
This work was started at JSALT 2017 in CMU, Pittsburgh, and was supported by JHU and CMU (via grants from Google, Microsoft, Amazon, Facebook, Apple), by the Czech Ministry of Education, Youth and Sports from the National Programme of Sustainability (NPU II) project "IT4Innovations excellence in science - LQ1602" and by the French ANR and the German DFG under grant ANR-14-CE35-0002 (BULB project). This work used the Extreme Science and Engineering Discovery Environment (NSF grant number OCI-1053575 and NSF award number ACI-1445606).

Figure and table captions:
Table 1: AUD results of the baseline (HMM model with uninformative prior) - Mboshi5k corpus
Table 2: Precision, Recall and F-measure on word boundaries, using different AUD methods. Segmental DTW baseline [23] gave F-score of 19.3% on the exact same corpus; dpseg [20] was also used as a word segmentation baseline and gave similar (slightly lower) F-scores to 1-best (best config with dpseg gave 42.5%) - Mboshi5k corpus
Table 3: Average duration of the (AUD) units (AUD) for the HMM and SVAE models trained with an uninformative prior. ”phones” refers to the forced aligned phone reference.
Table 4: Effect of the informative prior on AUD (phone boundary detection) - Mboshi5k corpus

Questions about this paper:
By how much they outperform the baseline?

Answers:
18.08 percent points on F-score