To perform a thorough analysis of our proposed approach (which requires transcriptions), we use a corpus of South African English. We use BNFs trained on two languages and on ten languages as input features to the CNN-DTW system, and compare these to MFCCs. We also consider features from unsupervised autoencoders, trained on unlabelled datasets from five languages. We show that the 10-language BNFs work best overall, giving results that makes CNN-DTW a viable option for practical use.

Experimental results	We consider four feature extractors in our experiments: In initial experiments, we first consider the performance of these features on development data. Specifically, we use the features as representations in the DTW-based keyword spotter (DTW-KS). Results are shown in Table TABREF23 . BNFs trained on 10 languages outperform all other approaches, with speaker normalisation giving a further slight improvement. Both the stacked autoencoder and the BNFs trained on two languages perform worse than the MFCC baseline. This seems to indicate that a larger number of diverse languages is beneficial for training BNFs, and that supervised models are superior to unsupervised models when applied to an unseen target language. However, further experiments are required to verify this definitively. Based on these development experiments, we compare MFCCs and TDNN-BNF-10lang-SPN features when used for keyword spotting on evaluation data. Table TABREF24 shows the performance of the three baseline systems and CNN-DTW when using MFCCs and BNFs. In all cases except the CNN classifier, BNFs lead to improvements over MFCCs.

ASR-free CNN-DTW keyword spotting using multilingual bottleneck features for almost zero-resource languages	We consider multilingual bottleneck features (BNFs) for nearly zero-resource keyword spotting. This forms part of a United Nations effort using keyword spotting to support humanitarian relief programmes in parts of Africa where languages are severely under-resourced. We use 1920 isolated keywords (40 types, 34 minutes) as exemplars for dynamic time warping (DTW) template matching, which is performed on a much larger body of untranscribed speech. These DTW costs are used as targets for a convolutional neural network (CNN) keyword spotter, giving a much faster system than direct DTW. Here we consider how available data from well-resourced languages can improve this CNN-DTW approach. We show that multilingual BNFs trained on ten languages improve the area under the ROC curve of a CNN-DTW system by 10.9% absolute relative to the MFCC baseline. By combining low-resource DTW-based supervision with information from well-resourced languages, CNN-DTW is a competitive option for low-resource keyword spotting.

We consider two models: a multilingual TDNN trained on only two languages, and a TDNN trained on ten diverse languages. Our aim is to investigate whether it is necessary to have a large set of diverse languages, or whether it is sufficient to simply obtain features from a supervised model trained on a smaller set of languages.   An 11-layer 2-language TDNN was trained using 40-high resolution MFCC features as input on a combined set of Dutch and Frisian speech, as described in  . Speaker adaptation is used with lattice-free maximum mutual information training, based on the Kaldi Switchboard recipe  . Each layer uses ReLU activations with batch normalisation. By combining the the FAME  and CGN  corpora, the training set consists of a combined 887 hours of data in the two languages. 40-dimensional BNFs are extracted from the resulting model.  A 6-layer 10-language TDNN was trained on the GlobalPhone corpus, also using 40-high resolution MFCC features as input, as described in  . For speaker adaptation, a 100-dimensional i-vector was appended to the the MFCC input features.

Most of these neural approaches, however, relies on large amounts of training data. In this paper, we extend the ASR-free keyword spotting approach first presented in  . A small seed corpus of isolated spoken keywords is used to perform DTW template matching on a large corpus of untranscribed data from the target domain. The resulting DTW scores are used as targets for training a CNN-based keyword spotter. Hence we take advantage of DTW-based matching—which can be performed with limited data—and combine this with CNN-based searching—giving speed benefits since it does not require alignment. In our previous work, we used speech data only from the target language. Here we consider whether data available for other (potentially well-resourced) languages can be used to improve performance. Specifically, multilingual bottleneck features (BNFs) have been shown to provide improved performance by several authors  ,  ,  ,  . We investigate whether such multilingual bottleneck feature extractors (trained on completely different languages) can be used to extract features for our target data, thereby improving the overall performance of our CNN-DTW keyword spotting approach.

In this paper we consider the development of a keyword spotter without such substantial and carefully-prepared data. Instead, we rely only on a small number of isolated repetitions of keywords and a large body of untranscribed data from the target domain. The motivation for this setting is that such isolated keywords should be easier to gather, even in a crisis scenario. Several studies have attempted ASR-free keyword spotting using a query-by-example (QbyE) retrieval procedure. In QbyE, the search query is provided as audio rather than text. Dynamic time warping (DTW) is typically used to search for instances of the query in a speech collection  ,  . As an alternative, several ways of obtaining fixed-dimensional representations of input speech have been considered  . Recurrent neural networks (RNNs)  ,  , autoencoding encoder-decoder RNNs  ,  , and Siamese convolutional neural networks (CNNs)  have all been used to obtain such fixed-dimensional representations, which allow queries and search utterances to be directly compared without alignment. For keyword spotting, a variant of this approach has been used where textual and acoustic inputs are mapped into a shared space  .

Bottleneck and Autoencoder Features	Our previous work focused purely on using data from the low-resource target language. However, large annotated speech resources exist for several well-resourced languages. We investigate whether such resources can be used to improve the CNN-DTW system in the unseen low-resource language.

Bottleneck features	One way to re-use information extracted from other multilingual corpora is to use multilingual bottleneck features (BNFs), which has shown to perform well in conventional ASR as well as intrinsic evaluations  ,  ,  ,  ,  ,  . These features are typically obtained by training a deep neural network jointly on several languages for which labelled data is available. The bottom layers of the network are normally shared across all training languages. The network then splits into separate parts for each of the languages, or has a single shared output. The final output layer has phone labels or HMM states as targets. The final shared layer often has a lower dimensionality than the input layer, and is therefore referred to as a `bottleneck'. The intuition is that this layer should capture aspects that are common across all the languages. We use such features from a multilingual neural network in our CNN-DTW keyword spotting approach. The BNFs are trained on a set of well-resourced languages different from the target language. Different neural architectures can be used to obtain BNFs following the above methodology. Here we use time-delay neural networks (TDNNs)  .

The TDNN was trained with a block-softmax, with the hidden layers shared across all languages and a separate output layer for each language. Each of the six hidden layers had 625 dimensions, and was followed by a 39-dimensional bottleneck layer with ReLU activations and batch normalisation. Training was accomplished using the Kaldi Babel receipe using 198 hours of data in 10 languages (Bulgarian, Czech, French, German, Korean, Polish, Portuguese, Russian, Thai, Vietnamese) from GlobalPhone.

Conclusion	We investigated the use of multilingual bottleneck (BNFs) and autoencoder features in a CNN-DTW keyword spotter. While autoencoder features and BNFs trained on two languages did not improve performance over MFCCs, BNFs trained on a corpus of 10 languages lead to substantial improvements. We conclude that our overall CNN-DTW based approach, which combines the low-resource advantages of DTW with the speed advantages of CNNs, further benefits by incorporating labelled data from well-resourced languages through the use of BNFs when these are obtained from several diverse language. Acknowledgements: We thank the NVIDIA corporation for the donation of GPU equipment used for this research. We also gratefully acknowledge the support of Telkom South Africa.

Experimental setup	The experimental setup is similar to that of  . We consider three baseline systems: two DTW systems and a conventional CNN classifier. Our proposed approach, CNN-DTW, is supervised by the DTW-KS system. Hyper-parameters for CNN-DTW were optimized using the target loss on the development set. Hence, the SABN transcriptions are not used for training or validation. Performance is reported in terms of the area under the curve (AUC) of the receiver operating characteristic (ROC) and equal error rate (EER). The ROC is obtained by varying the detection threshold and plotting the false positive rate against the true positive rate. AUC, therefore, indicates the performance of the model independent of a threshold, with higher AUC indicating a better model. EER is the point at which the false positive rate equals the false negative rate and hence lower EER indicates a better model.

Introduction	Social media has become a popular medium for individuals to express opinions and concerns on issues impacting their lives  ,  ,  . In countries without adequate internet infrastructure, like Uganda, communities often use phone-in talk shows on local radio stations for the same purpose. In an ongoing project by the United Nations (UN), radio-browsing systems have been developed to monitor such radio shows  ,  . These systems are actively and successfully supporting UN relief and developmental programmes. The development of such systems, however, remains dependent on the availability of transcribed speech in the target languages. This dependence has proved to be a key impediment to the rapid deployment of radio-browsing systems in new languages, since skilled annotators proficient in the target languages are hard to find, especially in crisis conditions. In a conventional keyword spotting system, where the goal is to search through a speech collection for a specified set of keywords, automatic speech recognition (ASR) is typically used to generate lattices which are then searched to predict the presence or absence of keywords  ,  . State-of-the-art ASR, however, requires large amounts of transcribed speech audio  ,  .

Autoencoder features	BNFs are trained in a supervised fashion with acoustic feature presented at the input and phone targets at the outputs. A more general scenario, however, is one in which training data is unlabelled, and these targets are therefore not known. In this case, it may be possible to learn useful representations by using an unsupervised model. An autoencoder is a neural network trained to reconstruct its input. By presenting the same data at the input and the output of the network while constraining intermediate connections, the network is trained to find an internal representation that is useful for reconstruction. These internal representations can be useful as features  ,  ,  ,  ,  ,  . Like BNFs, autoencoders can be trained on languages different from the target language (often resulting in more data to train on). Here we use a stacked denoising autoencoder  . In this model, each layer is trained individually like an autoencoder with added noise to reconstruct the output of the previous layer. Once a layer has been trained, its weights are fixed and its outputs become the inputs to the next layer to be trained.

Radio browsing system	The first radio browsing systems implemented as part of the UN's humanitarian monitoring programmes rely on ASR systems  . Human analysts filter speech segments identified by the system and add these to a searchable database to support decision making. To develop the ASR system, at least a small amount of annotated speech in the target language is required  . However, the collection of even a small fully transcribed corpus has proven difficult or impossible in some settings. In recent work, we have therefore proposed an ASR-free keyword spotting system based on CNNs  . CNN classifiers typically require a large number of training examples, which are not available in our setting. Instead, we therefore use a small set of recorded isolated keywords, which are then matched against a large collection of untranscribed speech drawn from the target domain using a DTW-based approach. The resulting DTW scores are then used as targets for a CNN. The key is that it is not necessary to know whether or not the keywords do in fact occur in this untranscribed corpus; the CNN is trained simply to emulate the behaviour of the DTW. Since the CNN does not perform any alignment, it is computationally much more efficient than DTW.

Data	We use the same datasets used in our previous work  . As templates, we use a small corpus of isolated utterances of 40 keywords, each spoken twice by 24 South African speakers (12 male and 12 female). This set of 1920 labelled isolated keywords constitutes the only transcribed target-domain data that we use to train our keyword spotter. As untranscribed data, we use a corpus of South African Broadcast News (SABN). This 23-hour corpus consists of a mix of English newsreader speech, interviews, and crossings to reporters, broadcast between 1996 and 2006  . The division of the corpus into different sets is shown in Table TABREF3 . The SABN training set is used as our untranscribed data to obtain targets for the CNN. The models then perform keyword spotting on the SABN test set. Since this set is fully transcribed, performance evaluation and analysis is possible. The isolated keywords were recorded under fairly quiet conditions and there was no speaker overlap with the SABN dataset. Hence there is a definite mismatch between the datasets. This is intentional as it reflects the intended operational setting of our system.

CNN-DTW keyword spotting	Rather than using labels (as in the CNN classifier above), the CNN-DTW keyword spotting approach uses DTW to generate sufficient training data as targets for a CNN. The CNN-DTW is subsequently employed as the keyword spotter; this is computationally much more efficient than direct DTW. DTW similarity scores are computed between our small set of isolated keywords and a much larger untranscribed dataset, and these scores are subsequently used as targets to train a CNN, as shown in Figure FIGREF9 . Our contribution here over our previous work  is to use multilingual BNFs instead of MFCCs, both for performing the DTW matching and as inputs to the CNN-DTW model. In Figure FIGREF9 , the upper half shows how the supervisory signals are obtained using DTW, and the lower half shows how the CNN is trained. Equation ( EQREF8 ) shows how keyword scores are computed, resulting in a vector INLINEFORM0 for each utterance INLINEFORM1 .

Keyword Spotting Approaches	Here we describe the combined CNN-DTW keyword spotting method. We also use direct DTW and a CNN classifier as baselines, and hence these are also briefly discussed.

Dynamic time warping (DTW)	In low-resource settings with scarce training data, DTW is an attractive approach, but it can be prohibitively slow since it requires repeated alignment. We make use of a simple DTW implementation in which isolated keywords slide over search audio, with a 3-frame-skip, and a frame-wise comparison is performed while warping the time axis. From this, a normalized per-frame cosine cost is obtained, resulting in a value INLINEFORM0 , with 0 indicating a portion of speech that matches the keyword exactly. The presence or absence of the keyword is determined by applying an appropriate threshold to INLINEFORM1 .

Convolutional neural network (CNN) classifier	As a baseline, we train a CNN classifier as an end-to-end keyword spotter. This would be typical in high resource settings  ,  ,  . We perform supervised training using the 1920 recorded isolated keywords with negative examples drawn randomly from utterances in the SABN training set. For testing, a 60-frame window slides over the test utterances. The presence or absence of keyword is again based on a threshold.

Figures FIGREF25 (a-f) show ROC plots for a selection of keywords which are representative of cases with both good and bad performance. AUC improves in all cases when switching from MFCCs to BNFs, except for health, where the difference is relative small (all scores are close to chance on this keyword). In some cases, e.g. for wounded, the benefits of switching to BNFs in CNN-DTW is substantial. Interestingly, for keywords such as attack, the CNN-DTW system using BNFs actually marginally outperforms the DTW-KS system which is used to supervise it.

DISPLAYFORM0  Here, INLINEFORM0 is the sequence of speech features for the INLINEFORM1 exemplar of keyword INLINEFORM2 , INLINEFORM3 is a successive segment of utterance INLINEFORM4 , and INLINEFORM5 is the DTW alignment cost between the speech features of exemplar INLINEFORM6 and the segment INLINEFORM7 . Each value INLINEFORM8 , which is between INLINEFORM9 , is then mapped to INLINEFORM10 , with 1 indicating a perfect match and 0 indicating dissimilarity thus forming the target vector INLINEFORM11 for utterance INLINEFORM12 . A CNN is then trained using a summed cross-entropy loss (which is why the scores are mapped to the interval INLINEFORM13 ) with utterance INLINEFORM14 as input and INLINEFORM15 as target. The CNN architecture is the same as used in  . Finally, the trained CNN is applied to unseen utterances.

After all the layers are pre-trained in this fashion, the layers are stacked and fine-tuned. We use mean squared error loss and Adam optimisation  throughout. We trained a 7-layer stacked denoising autoencoder on an untranscribed dataset consisting of 160 h of Acholi, 154 h of Luganda, 9.45 h of Lugbara, 7.82 h of Rutaroo and 18 h of Somali data. We used 39-dimensional MFCCs (13 cepstra with deltas and delta-deltas) as input and extracted features from the 39-dimensional fourth layer.

Furthermore, we see that, when using BNFs, the CNN-DTW system performs almost as well as its DTW-KS counterpart. The DTW-KS system provided the targets with which the CNN-DTW system was trained, and hence represents an upper bound on the performance we can expect from the CNN-DTW wordspotter. When using BNFs, we see that the difference between the DTW-KS and CNN-DTW approaches becomes smaller compared to the difference for MFCCs. This results in the CNN-DTW system using BNFs almost achieving the performance of the DTW-KS system; the former, however, is computationally much more efficient since alignment is not required. On a conventional desktop PC with a single NVIDIA GeForce GTX 1080 GPU, CNN-DTW takes approximately 5 minutes compared to DTW-KS which takes 900 minutes on a 20-core CPU machine. Table TABREF24 shows that, in contrast to when MFCCs are used, a Gaussian noise layer (CNN-DTW-GNL) does not give further performance benefits for the BNF systems.

The resulting CNN-DTW model can therefore be used to efficiently detect the presence of keywords in new input speech. Figure FIGREF2 show the structure of this CNN-DTW radio browsing system.

Question: What languages are considered?
Answer: Bulgarian, Czech, French, German, Korean, Polish, Portuguese, Russian, Thai, Vietnamese