Multilingual and Unsupervised Subword Modeling for Zero-Resource Languages

Abstract: Unsupervised subword modeling aims to learn low-level representations of speech audio in"zero-resource"settings: that is, without using transcriptions or other resources from the target language (such as text corpora or pronunciation dictionaries). A good representation should capture phonetic content and abstract away from other types of variability, such as speaker differences and channel noise. Previous work in this area has primarily focused on learning from target language data only, and has been evaluated only intrinsically. Here we directly compare multiple methods, including some that use only target language speech data and some that use transcribed speech from other (non-target) languages, and we evaluate using two intrinsic measures as well as on a downstream unsupervised word segmentation and clustering task. We find that combining two existing target-language-only methods yields better features than either method alone. Nevertheless, even better results are obtained by extracting target language bottleneck features using a model trained on other languages. Cross-lingual training using just one other language is enough to provide this benefit, but multilingual training helps even more. In addition to these results, which hold across both intrinsic measures and the extrinsic task, we discuss the qualitative differences between the different types of learned features.

Recent years have seen increasing interest in “zero-resource” speech technology: systems developed for a target language without using transcribed data or other hand-curated resources from that language. Such systems could potentially be applied to tasks such as endangered language documentation or query-by-example search for languages without a written form. One challenge for these systems, highlighted by the zrsc shared tasks of 2015 BIBREF0 and 2017 BIBREF1 , is to improve subword modeling, i.e., to extract or learn speech features from the target language audio. Good features should be more effective at discriminating between linguistic units, e.g. words or subwords, while abstracting away from factors such as speaker identity and channel noise.
The ZRSCs were motivated largely by questions in artificial intelligence and human perceptual learning, and focused on approaches where no transcribed data from any language is used. Yet from an engineering perspective it also makes sense to explore how training data from higher-resource languages can be used to improve speech features in a zero-resource language.
This paper explores several methods for improving subword modeling in zero-resource languages, either with or without the use of labeled data from other languages. Although the individual methods are not new, our work provides a much more thorough empirical evaluation of these methods compared to the existing literature. We experiment with each method both alone and in combinations not tried before, and provide results across a range of target languages, evaluation measures, and tasks.
We start by evaluating two methods for feature extraction that are trained using (untranscribed) target language data only: traditional vtln and the more recently proposed cae BIBREF2 . The cae learns to abstract away from signal noise and variability by training on pairs of speech segments extracted using an utd system—i.e., pairs that are likely to be instances of the same word or phrase. We confirm previous work showing that cae features outperform MFCCs on a word discriminability task, although we also show that this benefit is not consistently better than that of simply applying vtln. More interestingly, however, we find that applying vtln to the input of the cae system improves the learned features considerably, leading to better performance than either method alone. These improvements indicate that cae and vtln abstract over different aspects of the signal, and suggest that vtln might also be a useful preprocessing step in other recent neural-network-based unsupervised feature-learning methods.
Next, we explore how multilingual annotated data can be used to improve feature extraction for a zero-resource target language. We train multilingual bnfs on between one and ten languages from the GlobalPhone collection and evaluate on six other languages (simulating different zero-resource targets). We show that training on more languages consistently improves performance on word discrimination, and that the improvement is not simply due to more training data: an equivalent amount of data from one language fails to give the same benefit. In fact, we observe the largest gain in performance when adding the second training language, which is already better than adding three times as much data from the same language. Moreover, when compared to our best results from training unsupervised on target language data only, we find that bnfs trained on just a single other language already outperform the target-language-only training, with multilingual bnfs doing better by a wide margin.
Although multilingual training outperforms unsupervised target-language training, it could still be possible to improve on the multilingual bnfs by target-language fine-tuning. To test this hypothesis, we tried fine-tuning the multilingual bnfs to the target language by using them as input to the cae. When trained with utd word pairs, we found no benefit to this fine-tuning. However, training with manually labeled word pairs did yield benefits, suggesting that this type of supervision can help fine-tune the bnfs if the word pairs are sufficiently high-quality.
The results above were presented as part of an earlier conference version of this paper BIBREF3 . Here, we expand upon that work in several ways. First, we include new results on the corpora and evaluation measures used in the zrsc, to allow more direct comparisons with other work. In doing so, we also provide the first set of results on identical systems evaluated using both the same-different and ABX evaluation measures. This permits the two measures themselves to be better compared. Finally, we provide both a qualitative analysis of the differences between the different features we extract, and a quantitative evaluation on the downstream target-language task of unsupervised full-coverage speech segmentation and clustering using the system of BIBREF4 . This is the first time that multilingual features are used in such a system, which performs a complete segmentation of input speech into hypothesized words. As in our intrinsic evaluations, we find that the multilingual bnfs consistently outperform the best unsupervised cae features, which in turn outperform or do similarly to MFCCs.
We start by investigating how unlabeled data from the target language alone can be used for unsupervised subword modeling. Below we first review related work and provide a brief introduction to the cae and vtln methods. We then describe our experiments directly comparing these methods, both alone and in combination.
Various approaches have been applied to the problem of unsupervised subword modeling. Some methods work in a strictly bottom-up fashion, for example by extracting posteriorgrams from a (finite or infinite) Gaussian mixture model trained on the unlabeled data BIBREF5 , BIBREF6 , BIBREF7 , or by using neural networks to learn representations using autoencoding BIBREF8 , BIBREF9 , BIBREF10 or other loss functions BIBREF11 . Other methods incorporate weak top-down supervision by first extracting pairs of similar word- or phrase-like units using unsupervised term detection, and using these to constrain the representation learning. Examples include the cae BIBREF2 and ABNet BIBREF12 . Both aim to learn representations that make similar pairs even more similar; the ABNet additionally tries to make different pairs more different.
In this work we use the cae in our experiments on unsupervised representation learning, since it performed well in the 2015 ZRSC, achieved some of the best-reported results on the same-different task (which we also consider), and has readily available code. As noted above, the cae attempts to normalize out non-linguistic factors such as speaker, channel, gender, etc., by using top-down information from pairs of similar speech segments. Extracting cae features requires three steps, as illustrated in Figure FIGREF6 . First, an utd system is applied to the target language to extract pairs of speech segments that are likely to be instances of the same word or phrase. Each pair is then aligned at the frame level using dtw, and pairs of aligned frames are presented as the input INLINEFORM0 and target output INLINEFORM1 of a dnn. After training, a middle layer INLINEFORM2 is used as the learned feature representation.
The cae and other unsupervised methods described above implicitly aim to abstract away from speaker variability, and indeed they succeed to some extent in doing so BIBREF4 . Nevertheless, they provide less explicit speaker adaptation than standard methods used in supervised ASR, such as fMLLR BIBREF13 , LHUC BIBREF14 or i-vectors BIBREF15 . Explicit speaker adaptation seems to have attracted little attention until recently BIBREF16 in the zero-resource community, perhaps because most of the standard methods assume transcribed data is available.
Nevertheless, recent work suggests that at least some of these methods may be applied effectively even in an unsupervised setting. In particular, Heck at al. BIBREF17 , BIBREF18 won the zrsc 2017 using a typical asr pipeline with speaker adaptive fMLLR and other feature transforms. They adapted these methods to the unsupervised setting by first obtaining phone-like units with the dpgmm, an unsupervised clustering technique, and then using the cluster assignments as unsupervised phone labels during asr training.
In this work we instead consider a very simple feature-space adaptation method, vtln, which normalizes a speaker's speech by warping the frequency-axis of the spectra. vtln models are trained using maximum likelihood estimation under a given acoustic model—here, an unsupervised gmm. Warp factors can then be extracted for both the training data and for unseen data.
Although VTLN has recently been used by a few zero-resource speech systems BIBREF7 , BIBREF17 , BIBREF18 , its impact in these systems is unclear because there is no comparison to a baseline without vtln. BIBREF19 did precisely such a comparison and showed that applying vtln to the input of their unsupervised feature learning method improved its results in a phoneme discrimination task, especially in the cross-speaker case. However, we don't know whether other feature learning methods are similarly benefited by vtln, nor even how vtln on its own performs in comparison to more recent methods. Thus, our first set of experiments is designed to answer these questions by evaluating the benefits of using vtln and cae learning, both on their own and in combination.
There is considerable evidence that bnfs extracted using a multilingually trained dnn can improve ASR for target languages with just a few hours of transcribed data BIBREF27 , BIBREF28 , BIBREF29 , BIBREF30 , BIBREF31 . However, there has been little work so far exploring supervised multilingual bnfs for target languages with no transcribed data at all. BIBREF32 , BIBREF23 trained monolingual BNF extractors and showed that applying them cross-lingually improves word discrimination in a zero-resource setting. BIBREF33 , BIBREF19 trained a multilingual dnn to extract BNFs for a zero-resource task, but the dnn itself was trained on untranscribed speech: an unsupervised clustering method was applied to each language to obtain phone-like units, and the dnn was trained on these unsupervised phone labels.
We know of only two previous studies of supervised multilingual BNFs for zero-resource speech tasks. In the first BIBREF25 , the authors trained bnfs on either Mandarin, Spanish or both, and used the trained dnns to extract features from English (simulating a zero-resource language). On a query-by-example task, they showed that bnfs always performed better than MFCCs, and that bilingual bnfs performed as well or better than monolingual ones. Further improvements were achieved by applying weak supervision in the target language using a cae trained on English word pairs. However, the authors did not experiment with more than two training languages, and only evaluated on English.
In the second study BIBREF34 , the authors built multilingual systems using either seven or ten high-resource languages, and evaluated on the three “development” and two “surprise” languages of the zrsc 2017. However, they included transcribed training data from four out of the five evaluation languages, so only one language's results (Wolof) were truly zero-resource.
Our experiments therefore aim to evaluate on a wider range of target languages, and to explore the effects of both the amount of labeled data, and the number of languages from which it is obtained.
We use the GlobalPhone corpus of speech read from news articles BIBREF20 . We chose 6 languages from different language families as zero-resource languages on which we evaluate the new feature representations. That means our models do not have any access to the transcriptions of the training data, although transcriptions still need to be available to run the evaluation. The selected languages and dataset sizes are shown in Table TABREF8 . Each GlobalPhone language has recordings from around 100 speakers, with 80% of these in the training sets and no speaker overlap between training, development, and test sets.
For baseline features, we use Kaldi BIBREF21 to extract MFCCs+ INLINEFORM0 + INLINEFORM1 and PLPs+ INLINEFORM2 + INLINEFORM3 with a window size of 25 ms and a shift of 10 ms, and we apply per-speaker cmn. We also evaluated MFCCs and PLPs with vtln. The acoustic model used to extract the warp factors was a diagonal-covariance gmm with 1024 components. A single GMM was trained unsupervised on each language's training data.
To train the cae, we obtained utd pairs using a freely available utd system BIBREF22 and extracted 36k word pairs for each target language. Published results with this system use PLP features as input, and indeed our preliminary experiments confirmed that MFCCs did not work as well. We therefore report results using only PLP or PLP+VTLN features as input to utd. Following BIBREF23 , BIBREF2 , we train the cae model by first pre-training an autoencoder with eight 100-dimensional layers and a final layer of size 39 layer-wise on the entire training data for 5 epochs with a learning rate of INLINEFORM0 . We then fine-tune the network with same-word pairs as weak supervision for 60 epochs with a learning rate of INLINEFORM1 . Frame pairs are presented to the cae using either MFCC, MFCC+VTLN, or BNF representation, depending on the experiment (preliminary experiments indicated that PLPs performed worse than MFCCs, so MFCCs are used as the stronger baseline). Features are extracted from the final hidden layer of the cae as shown in Figure FIGREF6 .
To provide an upper bound on cae performance, we also report results using gold standard same-word pairs for cae training. As in BIBREF2 , BIBREF24 , BIBREF25 , we force-align the target language data and extract all the same-word pairs that are at least 5 characters and 0.5 seconds long (between 89k and 102k pairs for each language).
We picked another 10 languages (different from the target languages described in Section SECREF7 ) with a combined 198.3 hours of speech from the GlobalPhone corpus. We consider these as high-resource languages, for which transcriptions are available to train a supervised asr system. The languages and dataset sizes are listed in Table TABREF16 . We also use the English wsj corpus BIBREF35 which is comparable to the GlobalPhone corpus. It contains a total of 81 hours of speech, which we either use in its entirety or from which we use a 15 hour subset; this allows us to compare the effect of increasing the amount of data for one language with training on similar amounts of data but from different languages.
Supervised models trained on these high-resource languages are evaluated on the same set of zero-resource languages as in Section SECREF2 . Transcriptions of the latter are still never used during training.
For initial monolingual training of asr systems for the high-resource languages, we follow the Kaldi recipes for the GlobalPhone and WSJ corpora and train a sgmm system for each language to get initial context-dependent state alignments; these states serve as targets for dnn training.
For multilingual training, we closely follow the existing Kaldi recipe for the Babel corpus. We train a tdnn BIBREF36 with block softmax BIBREF37 , i.e. all hidden layers are shared between languages, but there is a separate output layer for each language. For each training instance only the error at the corresponding language's output layer is used to update the weights. This architecture is illustrated in Figure FIGREF17 . The tdnn has six 625-dimensional hidden layers followed by a 39-dimensional bottleneck layer with ReLU activations and batch normalization. Each language then has its own 625-dimensional affine and a softmax layer. The inputs to the network are 40-dimensional MFCCs with all cepstral coefficients to which we append i-vectors for speaker adaptation. The network is trained with stochastic gradient descent for 2 epochs with an initial learning rate of INLINEFORM0 and a final learning rate of INLINEFORM1 .
In preliminary experiments we trained a separate i-vector extractor for each different sized subset of training languages. However, results were similar to training on the pooled set of all 10 high-resource languages, so for expedience we used the 100-dimensional i-vectors from this pooled training for all reported experiments. The i-vectors for the zero-resource languages are obtained from the same extractor. This allows us to also apply speaker adaptation in the zero-resource scenario. Including i-vectors yielded a small performance gain over not doing so; we also tried applying vtln to the MFCCs for tdnn training, but found no additional benefit.
All experiments in this section are evaluated using the same-different task BIBREF26 , which tests whether a given speech representation can correctly classify two speech segments as having the same word type or not. For each word pair in a pre-defined set INLINEFORM0 the dtw cost between the acoustic feature vectors under a given representation is computed. Two segments are then considered a match if the cost is below a threshold. Precision and recall at a given threshold INLINEFORM1 are defined as INLINEFORM2 
where INLINEFORM0 is the number of sw, swdp or all discovered matches at that threshold and INLINEFORM1 is the number of actual swdp pairs in INLINEFORM2 . We can compute a precision-recall curve by varying INLINEFORM3 . The final evaluation metric is the ap or the area under that curve. We generate evaluation sets of word pairs for the GlobalPhone development and test sets from all words that are at least 5 characters and 0.5 seconds long, except that we now also include different-word pairs.
Previous work BIBREF26 , BIBREF2 calculated recall with all sw pairs for easier computation because their test sets included a negligible number of swsp pairs. In our case the smaller number of speakers in the GlobalPhone corpora results in up to 60% of sw pairs being from the same speaker. We therefore always explicitly compute the recall only for swdp pairs to focus the evaluation of features on their speaker invariance.
Table TABREF13 shows AP results on all target languages for cae features learned using raw features as input (as in previous work) and for cae features learned using vtln-adapted features as input to either the utd system, the cae, or both. Baselines are raw MFCCs, or MFCCs with VTLN. MFCCs with VTLN have not previously been compared to more recent unsupervised subword modeling methods, but as our results show, they are a much stronger baseline than MFCCs alone. Indeed, they are nearly as good as cae features (as trained in previous work). However, we obtain much better results by applying vtln to both the cae and utd input features (MFCCs and PLPs, respectively). Individually these changes each result in substantial improvements that are consistent across all 6 languages, and applying VTLN at both stages helps further. Indeed, applying vtln is beneficial even when using gold pairs as cae input, although to a lesser degree.
So, although previous studies have indicated that cAE training and VTLN are helpful individually, our experiments provide further evidence and quantification of those results. In addition, we have shown that combining the two methods leads to further improvements, suggesting that cae training and vtln abstract over different aspects of the speech signal and should be used together. The large gains we found with VTLN, and the fact that it was part of the winning system in the 2017 ZRSC, suggest that it is also likely to help in combination with other unsupervised subword modeling methods.
As a sanity check we include word error rates (WER) for the asr systems trained on the high-resource languages. Table TABREF20 compares the WER of the monolingual sgmm systems that provide the targets for tdnn training to the WER of the final model trained on all 10 high-resource languages. The multilingual model shows small but consistent improvements for all languages except Vietnamese. Ultimately though, we are not so much interested in the performance on typical asr tasks, but in whether bnfs from this model also generalize to zero-resource applications on unseen languages.
Figure FIGREF21 shows ap on the same-different task of multilingual bnfs trained from scratch on an increasing number of languages in two randomly chosen orders. We provide two baselines for comparison, drawn from our results in Table TABREF13 . Firstly, our best cae features trained with utd pairs (row 4, Table TABREF13 ) are a reference for a fully unsupervised system. Secondly, the best cae features trained with gold standard pairs (row 6, Table TABREF13 ) give an upper bound on the cae performance.
In all 6 languages, even bnfs from a monolingual tdnn already considerably outperform the cae trained with utd pairs. Adding another language usually leads to an increase in ap, with the bnfs trained on 8–10 high-resource languages performing the best, also always beating the gold cae. The biggest performance gain is obtained from adding a second training language—further increases are mostly smaller. The order of languages has only a small effect, although for example adding other Slavic languages is generally associated with an increase in ap on Croatian. This suggests that it may be beneficial to train on languages related to the zero-resource language if possible, but further experiments need to be conducted to quantify this effect.
To determine whether these gains come from the diversity of training languages or just the larger amount of training data, we trained models on the 15 hour subset and the full 81 hours of the English wsj corpus, which corresponds to the amount of data of four GlobalPhone languages. More data does help to some degree, as Figure FIGREF21 shows. But, except for Mandarin, training on just two languages (46 hours) already works better.
Next we investigate how labeled data from high-resource languages can be used to obtain improved features on a target zero-resource language for which no labeled data is available.
In the previous experiments, we used data from GlobalPhone, which provides corpora collected and formatted similarly for a wide range of languages. However, GlobalPhone is not freely available and no previous zero-resource studies have used these corpora, so in this section we also provide results on the zrsc 2015 BIBREF0 data sets, which have been widely used in other work. The target languages are English (from the Buckeye corpus BIBREF38 ) and Xitsonga (NCHLT corpus BIBREF39 ). Table TABREF8 includes the corpus statistics. These corpora are not split into train/dev/test; since training is unsupervised, the system is simply trained directly on the unlabeled test set (which could also be done in deployment). Importantly, no hyperparameter tuning is done on the Buckeye or Xitsonga data, so these results still provide a useful test of generalization. Notably, the Buckeye English corpus contains conversational speech and is therefore different in style from the rest of our data.
For training the cae on the Buckeye English and Xitsonga corpora, we use the same sets of utd pairs as in BIBREF23 , which were discovered from fdlp features. We evaluate using both the same-different measures from above, as well as the ABX phone discriminability task BIBREF40 used in the zrsc and other recent work BIBREF0 , BIBREF1 . The ABX task evaluates phoneme discriminability using minimal pairs: sequences of three phonemes where the central phoneme differs between the two sequences INLINEFORM0 and INLINEFORM1 in the pair, such as b ih n and b eh n. Feature representations are then evaluated on how well they can identify a third triplet INLINEFORM2 as having the same phoneme sequence as either INLINEFORM3 or INLINEFORM4 . See BIBREF0 , BIBREF1 for details on how the scores are computed and averaged over speakers and phonemes to obtain the final ABX error rate. One usually distinguishes between the within-speaker error rate where all three triplets belong to the same speaker, and the cross-speaker error rate where INLINEFORM5 and INLINEFORM6 are from the same and INLINEFORM7 from a different speaker.
The ABX evaluation includes all such minimal pair phoneme triplets of the evaluation corpus. These pairs therefore rarely correspond to full words, making it a somewhat abstract task whose results may be difficult to interpret when summarizing it as a single final metric. ABX can however be very suitable for more fine-grained analysis of speech phenomena by including only specific phonetic contrasts in the evaluation BIBREF41 . In contrast, the same-different task always compares whole words and directly evaluates how good feature representations are at telling whether two utterances are the same word or not. Thus it has an immediate link to applications like spoken term detection and it allows easier error analysis. It is also faster to prepare the same-different evaluation set and run the evaluation. We wish to verify that the ABX and same-different measures correlate well, to better compare studies that use only one of them and to allow choosing the task that is more appropriate for the situation at hand.
Table TABREF22 shows results on the Xitsonga and Buckeye English corpora. Here we compare ABX error rates computed with the zrsc 2015 BIBREF0 evaluation scripts with ap on the same-different task. To the best of our knowledge, this is the first time such a comparison has been made. The results on both tasks correlate well, especially when looking at the ABX cross-speaker error rate because the same-different evaluation as described in Section SECREF11 also focuses on cross-speaker pairs. As might be expected vtln only improves cross-speaker, but not within-speaker ABX error rates.
For comparison we also include ABX results of the official zrsc 2015 topline BIBREF0 , which are posteriorgrams obtained from a supervised speech recognition system, the current state-of-the-art system BIBREF18 which even outperforms the topline for English, and the system of BIBREF42 which is the most recent form of the ABNet BIBREF12 , an architecture that is similar to our cae.
These systems score better than all of our features, but are not directly comparable for several reasons. Firstly, it is unclear how these systems were optimized, since there was no separate development set in zrsc 2015. Secondly, our features are all 39-dimensional to be directly comparable with MFCCs, whereas the other two systems have higher dimensionality (and indeed the winning system from zrsc 2017 was even greater, with more than 1000 dimensions BIBREF17 ). Such higher dimensional features may be useful in some circumstances, but lower dimensional features are often more efficient to work with and we don't know whether the competing systems would work as well with fewer dimensions.
The bnfs are in any case competitive with the higher dimensional features, and have the advantage that they can be built using standard Kaldi scripts and do not require any training on the target language, so can easily be deployed to new languages. The competitive result of BIBREF42 also shows that in general a system trained on word pairs discovered from a utd system can perform very well.
So far we have shown that multilingual bnfs work better than any of the features trained using only the target language data. However, in principle it could be possible to use the target language data to fine tune the bnfs in an unsupervised fashion, improving performance further. We explored this possibility by simply training a cae using bnfs as input rather than PLPs. That is, we trained the cae with the same word pairs as before, but replaced VTLN-adapted MFCCs with the 10-lingual bnfs as input features, without any other changes in the training procedure. Table TABREF23 (penultimate row) shows that the cae trained with utd pairs is able to slightly improve on the bnfs in some cases, but this is not consistent across all languages and for Croatian the cae features are much worse. On the other hand, when trained using gold standard pairs (final row), the resulting cae features are consistently better than the input bnfs. This indicates that bnfs can in principle be improved by target-language fine-tuning, but the top-down supervision needs to be of higher quality than the current UTD system provides.
This observation leads to a further question: could we improve the UTD pairs themselves by using our improved features (either bnfs or cae features) as input to the UTD system? If the output is a better set of UTD pairs than the original set, these could potentially be used to further improve the features, and perhaps the process could be iterated. As far as we know, no previously published work has combined unsupervised subword modeling with a utd system. However, after considerable efforts to make this work we found that the ZRTools utd system seems to be too finely tuned towards features that resemble PLPs to get good results from our new features.
To understand why the features that help with word and phone discrimination are a problem for the UTD system, we examined the similarity plots for several pairs of utterances. Figures FIGREF24 and FIGREF29 show that cae features and bnfs look quite different from PLPs. Dark areas indicate acoustic similarity and diagonal line segments therefore point to phonetically similar sequences. In Figure FIGREF24 both utterances contain the words estados unidos, but shorter and more faint lines can also be seen for rough matches like the last two syllables of servicio and visas. The ZRTools utd toolkit identifies these diagonal lines with fast computer vision techniques BIBREF22 and then runs a segmental-dtw algorithm only in the candidate regions for efficient discovery of matches.
PLPs are designed to contain fine-grained acoustic information about the speech signal and can therefore vary a lot throughout the duration of a phoneme. The diagonal lines in Figure FIGREF24 (a) are therefore very thin and there is a lot of spurious noise that does not necessarily correspond to phonetically similar units. This pattern is similar for VTLN-adapted PLPs in (b), but with less noise.
On the other hand, cae features and bnfs are trained to ignore such local variation within phonemes. This results in significantly different appearance of frame-wise cosine similarity plots of two utterances. The trained features remain more constant throughout the duration of a phoneme, resulting in wider diagonal lines in the similarity plots. Especially cae features are very good at learning phoneme-level information, indicated by the large rectangular blocks in Figure FIGREF24 (c) where phonemes of the two utterances match or are very similar. We also found the boundaries of these blocks to align well with actual phoneme boundaries provided by forced alignment. This is despite the cae not having any information about phoneme identities or boundaries during training.
While ZRTools still finds the diagonal line segments in cae features and bnfs where matches are likely to occur, the segmental dtw algorithm that then searches for exact matches finds too many of them because the lines are much wider and similarity values overall higher than for PLPs. For example Figure FIGREF29 shows a typical example of phonetically similar, but incorrect matches that are only discovered in cae features and bnfs. Although it might be possible to eventually identify a set of dtw parameters that can work with these types of features, it could be more productive to consider different approaches for features that are relatively stable within phones.
Our experiment with the UTD system was disappointing, suggesting that although cae features and bnfs improve intrinsic discriminability measures, they may not work with some downstream zero-resource tools. However, ZRTools is a single example. To further investigate the downstream effects of the learned features, we now consider the task of full-coverage speech segmentation and clustering. The aim here is to tokenize the entire speech input into hypothesized categories, potentially corresponding to words, and to do so without any form of supervision—essentially a form of unsupervised speech recognition. Such systems could prove useful from a speech technology perspective in low-resource settings, and could be useful in studying how human infants acquire language from unlabeled speech input.
Here we specifically investigate whether our BNFs improve the Bayesian embedded segmental Gaussian mixture model (BES-GMM), first proposed in BIBREF43 . This approach relies on a mapping where potential word segments (of arbitrary length) are embedded in a fixed-dimensional acoustic vector space. The model, implemented as a Gibbs sampler, builds a whole-word acoustic model in this acoustic embedding space, while jointly performing segmentation. Several acoustic word embedding methods have been considered, but here we use the very simple approach also used in BIBREF4 : any segment is uniformly downsampled so that it is represented by the same fixed number of frame-level features, which are then flattened to obtain the fixed-dimensional embedding BIBREF44 .
We retrained the cae and BNF models to return 13-dimensional features with all other parameters unchanged to be consistent with the experiments of BIBREF4 and for computational reasons. We also did not tune any hyperparameters of the BES-GMM for our new input features. Nonetheless, our baseline cae results do not exactly correspond to the ones in BIBREF4 because for example the MFCC input features have been extracted with a different toolkit and we used a slightly different training procedure.
We use several metrics to compare the resulting segmented word tokens to ground truth forced alignments of the data. By mapping every discovered word token to the ground truth word with which it overlaps most, average cluster purity can be calculated as the total proportion of correctly mapped tokens in all clusters. More than one cluster may be mapped to the same ground truth word type. In a similar way, we can calculate unsupervised word error rate (WER), which uses the same cluster-to-word mapping but also takes insertions and deletions into account. Here we consider two ways to perform the cluster mapping: many-to-one, where more than one cluster can be assigned the same word label (as in purity), or one-to-one, where at most one cluster is mapped to a ground truth word type (accomplished in a greedy fashion). We also compute the gender and speaker purity of the clusters, where we want to see clusters that are as diverse as possible on these measures, i.e., low purity. To explicitly evaluate how accurate the model performs segmentation, we compare the proposed word boundary positions to those from forced alignments of the data (falling within a single true phoneme from the boundary). We calculate boundary precision and recall, and report the resulting word boundary F-scores. We also calculate word token F-score, which requires that both boundaries from a ground truth word token be correctly predicted.
Table TABREF36 compares MFCCs, cae features (with and without vtln) and bnfs as input to the system of BIBREF4 . It shows that both vtln and bnfs help on all metrics, with improvements ranging from small to more substantial and bnfs clearly giving the most benefit. The effects of vtln are mostly confined to reducing both gender and speaker purity of the identified clusters (which is desirable) while maintaining the performance on other metrics. This means that the learned representations have become more invariant to variation in speaker and gender, which is exactly what vtln aims to do. However, this appears to be insufficient to also help other metrics, aligning with the experiments in BIBREF4 that indicate that improvements on the other metrics are hard to obtain.
On the other hand, bnfs result in better performance across all metrics. While some of these improvements are small, they are very consistent across all metrics. This shows that the bnfs are also useful for down-stream tasks in zero-resource settings. It especially demonstrates that such bnfs which are trained on high-resource languages without seeing any target language speech at all are a strong alternative to fully unsupervised features for practical scenarios or could in turn be used to improve unsupervised systems trained on the target language speech data.
bnfs cae utd
In this work we investigated different representations obtained using data from the target language alone (i.e., fully unsupervised) and from multilingual supervised systems trained on labeled data from non-target languages. We found that the cae, a recent neural approach to unsupervised subword modeling, learns complementary information to the more traditional approach of vtln. This suggests that vtln should also be considered by other researchers using neural approaches. On the other hand, our best results were achieved using multilingual bnfs. These results are competitive with state-of-the-art features learned from target language data only BIBREF17 , BIBREF18 , but have the advantage of a much smaller dimensionality. In addition, it is easy to control the dimensionality of the bnfs, unlike in the nonparametric models of BIBREF17 , BIBREF18 , and this allowed us to use them in the downstream task of word segmentation and clustering. We observed consistent improvements from bnfs across all metrics in this downstream task, and other work demonstrates that these features are also useful for downstream keyword spotting in settings with very small amounts of labeled data BIBREF45 . We also showed that it is theoretically possible to further improve bnfs with language-specific fine-tuning, and we hope to explore models that can do this more reliably than the cae in the future.
Finally, our qualitative analysis showed that both cae features and bnfs tend to vary much less over time than traditional PLPs, supporting the idea that they are better at capturing phonetic information rather than small variations in the acoustics. Although this property helps explain the better performance on intrinsic measures and the segmentation task, it harms performance for utd, where the system seems heavily tuned towards PLPs. Therefore, our work also points to the need for term discovery systems that are more robust to different types of input features.
The research was funded in part by a James S. McDonnell Foundation Scholar Award.

Figure and table captions:
Fig. 1. Correspondence autoencoder training procedure (see section II-A).
TABLE I ZERO-RESOURCE LANGUAGES, DATASET SIZES IN HOURS.
TABLE III HIGH-RESOURCE LANGUAGES, DATASET SIZES IN HOURS.
TABLE II AVERAGE PRECISION SCORES ON THE SAME-DIFFERENT TASK (DEV SETS), SHOWING THE EFFECTS OF APPLYING VTLN TO THE INPUT FEATURES FOR THE UTD AND/OR CAE SYSTEMS. CAE INPUT IS EITHER MFCC OR MFCC+VTLN. TOPLINE RESULTS (ROWS 5-6) TRAIN CAE ON GOLD STANDARD PAIRS, RATHER THAN UTD OUTPUT. BASELINE RESULTS (FINAL ROWS) DIRECTLY EVALUATE ACOUSTIC FEATURES WITHOUT UTD/CAE TRAINING. BEST UNSUPERVISED RESULT IN BOLD.
Fig. 2. Multilingual ASR training architecture. All layers are shared between languages except for the language-specific output layers at the top.
TABLE IV WORD ERROR RATES OF MONOLINGUAL SGMM AND 10-LINGUAL TDNN ASR SYSTEM EVALUATED ON THE DEVELOPMENT SETS.
Fig. 3. Same-different task evaluation on the development sets for BNFs trained on different amounts of data. We compare training on up to 10 different languages with additional data in one language (English). For multilingual training, languages were added in two different orders: FR-PT-DE-TH-PL-KO-CS-BG-RU-VI (BNFs 1) and RU-CZ-VI-PL-KO-TH-BG-PT-DE-FR (BNFs 2). Each datapoint shows the result of adding an additional language. As baselines we include the best unsupervised cAE and the cAE trained on gold standard pairs from rows 4 and 6 of Table II.
TABLE V COMPARISON OF AP ON THE SAME-DIFFERENT TASK (HIGHER IS BETTER) AND ABX CROSS-/WITHIN-SPEAKER ERROR RATES (LOWER IS BETTER) FOR THE BUCKEYE ENGLISH AND XITSONGA CORPORA.
TABLE VI AP ON THE SAME-DIFFERENT TASK WHEN TRAINING CAE ON THE 10-LINGUAL BNFS FROM ABOVE (CAE-BNF) WITH UTD AND GOLD STANDARD WORD PAIRS (TEST SET RESULTS). BASELINES ARE MFCC+VTLN AND THE CAE MODELS FROM ROWS 4 AND 6 OF TABLE II THAT USE MFCC+VTLN AS INPUT FEATURES. BEST RESULT WITHOUT TARGET LANGUAGE SUPERVISION IN BOLD.
Fig. 4. Frame-wise cosine similarity matrices for two Spanish utterances from different speakers, comparing different feature representations. Dark regions correspond to high cosine similarity and values below 0.4 are clipped. Red rectangles mark matches discovered by the UTD system and include their DTW similarity scores. In this case the match is not found with PLPs as input features.
Fig. 5. Frame-wise cosine similarity matrices for two Spanish utterances from different speakers, comparing different feature representations. Dark regions correspond to high cosine similarity and values below 0.4 are clipped. Red rectangles mark matches discovered by the UTD system and include their DTW similarity scores. The discovered matches are incorrect—although phonetically similar—and found only for cAE features and BNFs.
TABLE VII SEGMENTATION AND CLUSTERING RESULTS (LOWER SCORES ARE BETTER, EXCEPT FOR TOKEN AND BOUNDARY F-SCORE AND CLUSTER PURITY).

Questions about this paper:
With how many languages do they experiment in the multilingual setup?
Which dataset do they use?

Answers:
16
GlobalPhone
Croatian
Hausa
Mandarin
Spanish
Swedish
Turkish
ZRSC
Buckeye
Xitsonga