Abstract: Unsupervised subword modeling aims to learn low-level representations of speech audio in"zero-resource"settings: that is, without using transcriptions or other resources from the target language (such as text corpora or pronunciation dictionaries). A good representation should capture phonetic content and abstract away from other types of variability, such as speaker differences and channel noise. Previous work in this area has primarily focused on learning from target language data only, and has been evaluated only intrinsically. Here we directly compare multiple methods, including some that use only target language speech data and some that use transcribed speech from other (non-target) languages, and we evaluate using two intrinsic measures as well as on a downstream unsupervised word segmentation and clustering task. We find that combining two existing target-language-only methods yields better features than either method alone. Nevertheless, even better results are obtained by extracting target language bottleneck features using a model trained on other languages. Cross-lingual training using just one other language is enough to provide this benefit, but multilingual training helps even more. In addition to these results, which hold across both intrinsic measures and the extrinsic task, we discuss the qualitative differences between the different types of learned features.

Introduction: Recent years have seen increasing interest in “zero-resource” speech technology: systems developed for a target language without using transcribed data or other hand-curated resources from that language. Such systems could potentially be applied to tasks such as endangered language documentation or query-by-example search for languages without a written form. One challenge for these systems, highlighted by the zrsc shared tasks of 2015 BIBREF0 and 2017 BIBREF1 , is to improve subword modeling, i.e., to extract or learn speech features from the target language audio. Good features should be more effective at discriminating between linguistic units, e.g. words or subwords, while abstracting away from factors such as speaker identity and channel noise. The ZRSCs were motivated largely by questions in artificial intelligence and human perceptual learning, and focused on approaches where no transcribed data from any language is used. Yet from an engineering perspective it also makes sense to explore how training data from higher-resource languages can be used to improve speech features in a zero-resource language. This paper explores several methods for improving subword modeling in zero-resource languages, either with or without the use of labeled data from other languages. Although the individual methods are not new, our work provides a much more thorough empirical evaluation of these methods compared to the existing literature. We experiment with each method both alone and in combinations not tried before, and provide results across a range of target languages, evaluation measures, and tasks. We start by evaluating two methods for feature extraction that are trained using (untranscribed) target language data only: traditional vtln and the more recently proposed cae BIBREF2 . The cae learns to abstract away from signal noise and variability by training on pairs of speech segments extracted using an utd system—i.e., pairs that are likely to be instances of the same word or phrase. We confirm previous work showing that cae features outperform MFCCs on a word discriminability task, although we also show that this benefit is not consistently better than that of simply applying vtln. More interestingly, however, we find that applying vtln to the input of the cae system improves the learned features considerably, leading to better performance than either method alone. These improvements indicate that cae and vtln abstract over different aspects of the signal, and suggest that vtln might also be a useful preprocessing step in other recent neural-network-based unsupervised feature-learning methods. Next, we explore how multilingual annotated data can be used to improve feature extraction for a zero-resource target language. We train multilingual bnfs on between one and ten languages from the GlobalPhone collection and evaluate on six other languages (simulating different zero-resource targets). We show that training on more languages consistently improves performance on word discrimination, and that the improvement is not simply due to more training data: an equivalent amount of data from one language fails to give the same benefit. In fact, we observe the largest gain in performance when adding the second training language, which is already better than adding three times as much data from the same language. Moreover, when compared to our best results from training unsupervised on target language data only, we find that bnfs trained on just a single other language already outperform the target-language-only training, with multilingual bnfs doing better by a wide margin. Although multilingual training outperforms unsupervised target-language training, it could still be possible to improve on the multilingual bnfs by target-language fine-tuning. To test this hypothesis, we tried fine-tuning the multilingual bnfs to the target language by using them as input to the cae. When trained with utd word pairs, we found no benefit to this fine-tuning. However, training with manually labeled word pairs did yield benefits, suggesting that this type of supervision can help fine-tune the bnfs if the word pairs are sufficiently high-quality. The results above were presented as part of an earlier conference version of this paper BIBREF3 . Here, we expand upon that work in several ways. First, we include new results on the corpora and evaluation measures used in the zrsc, to allow more direct comparisons with other work. In doing so, we also provide the first set of results on identical systems evaluated using both the same-different and ABX evaluation measures. This permits the two measures themselves to be better compared. Finally, we provide both a qualitative analysis of the differences between the different features we extract, and a quantitative evaluation on the downstream target-language task of unsupervised full-coverage speech segmentation and clustering using the system of BIBREF4 . This is the first time that multilingual features are used in such a system, which performs a complete segmentation of input speech into hypothesized words. As in our intrinsic evaluations, we find that the multilingual bnfs consistently outperform the best unsupervised cae features, which in turn outperform or do similarly to MFCCs.

Question: Which dataset do they use?
Answer: GlobalPhone
Croatian
Hausa
Mandarin
Spanish
Swedish
Turkish
ZRSC
Buckeye
Xitsonga