Multilingual Speech Recognition with Corpus Relatedness Sampling	Multilingual acoustic models have been successfully applied to low-resource speech recognition. Most existing works have combined many small corpora together and pretrained a multilingual model by sampling from each corpus uniformly. The model is eventually fine-tuned on each target corpus. This approach, however, fails to exploit the relatedness and similarity among corpora in the training set. For example, the target corpus might benefit more from a corpus in the same domain or a corpus from a close language. In this work, we propose a simple but useful sampling strategy to take advantage of this relatedness. We first compute the corpus-level embeddings and estimate the similarity between each corpus. Next, we start training the multilingual model with uniform-sampling from each corpus at first, then we gradually increase the probability to sample from related corpora based on its similarity with the target corpus. Finally, the model would be fine-tuned automatically on the target corpus. Our sampling strategy outperforms the baseline multilingual model on 16 low-resource tasks. Additionally, we demonstrate that our corpus embeddings capture the language and domain information of each corpus.

Conclusion	In this work, we propose an approach to compute corpus-level embeddings. We also introduce Corpus Relatedness Sampling approach to train multilingual speech recognition models based on those corpus embeddings. Our experiment shows that our approach outperforms the fine-tuned multilingual models in all 16 test corpora by 1.6 phone error rate on average. Additionally, we demonstrate that our corpus embeddings can capture both language and domain information of each corpus.

Introduction	In recent years, Deep Neural Networks (DNNs) have been successfully applied to Automatic Speech Recognition (ASR) for many well-resourced languages including Mandarin and English  ,  . However, only a small portion of languages have clean speech labeled corpus. As a result, there is an increasing interest in building speech recognition systems for low-resource languages. To address this issue, researchers have successfully exploited multilingual speech recognition models by taking advantage of labeled corpora in other languages  ,  . Multilingual speech recognition enables acoustic models to share parameters across multiple languages, therefore low-resource acoustic models can benefit from rich resources. While low-resource multilingual works have proposed various acoustic models, those works tend to combine several low-resource corpora together without paying attention to the variety of corpora themselves. One common training approach here is to first pretrain a multilingual model by combining all training corpora, then the pretrained model is fine-tuned on the target corpus  . During the training process, each corpus in the training set is treated equally and sampled uniformly. We argue, however, this approach does not take account of the characteristics of each corpus, therefore it fails to take advantage of the relations between them.

The results show that our approach outperforms those baselines on all corpora: it achieves 1.6% lower phone error rate on average. Additionally, we demonstrate that our corpus-level embeddings are able to capture the characteristics of each corpus, especially the language and domain information. The main contributions of this paper are as follows:

Results	Table. TABREF16 shows the results of our evaluation. We compare our approach with the baseline using all corpora. The left-most column of Table. TABREF16 shows the corpus we used for each experiment, the remaining columns are corresponding to the phone error rate of the pretrained model, the fine-tuned model and our proposed model. First, we can easily confirm that the fine-tuned model outperforms the pretrained model on all corpora. For instance, the fine-tuned model outperforms the pretrained model by 4.7% on the Amharic corpus. The result is reasonable as the pretrained model is optimized with the entire training set, while the fine-tuned model is further adapted to the target corpus. Next, the table suggests our Corpus Relatedness Sampling approach achieves better results than the fine-tuned model on all test corpora. For instance, the phone error rate is improved from 40.7% to 36.9% on Amharic and is improved from 41.9% to 40.0% on Bengali. On average, our approach outperforms the fine-tuned model by 1.6% phone error rate.

Corpus Embedding	Suppose that INLINEFORM0 is the target low-resource corpus, we are interested in optimizing the acoustic model with a much larger training corpora set INLINEFORM1 where INLINEFORM2 is the number of corpora and INLINEFORM3 . Each corpus INLINEFORM4 is a collection of INLINEFORM5 pairs where INLINEFORM6 is the input features and INLINEFORM7 is its target. Our purpose here is to compute the embedding INLINEFORM0 for each corpus INLINEFORM1 where INLINEFORM2 is expected to encode information about its corpus INLINEFORM3 . Those embeddings can be jointly trained with the standard multilingual model  . First, the embedding matrix INLINEFORM4 for all corpora is initialized, the INLINEFORM5 -th row of INLINEFORM6 is corresponding to the embedding INLINEFORM7 of the corpus INLINEFORM8 . Next, during the training phase, INLINEFORM9 can be used to bias the input feature INLINEFORM10 as follows. DISPLAYFORM0  where INLINEFORM0 is an utterance sampled randomly from INLINEFORM1 , INLINEFORM2 is its hidden features, INLINEFORM3 is the parameter of the acoustic model and Encoder is the stacked bidirectional LSTM as shown in Figure. FIGREF5 .

The results demonstrate that our sampling approach is more effective at optimizing the acoustic model on the target corpus. We also train baseline models by appending corpus embeddings to input features, but the proposed model outperforms those baselines similarly. One interesting trend we observed in the table is that the improvements differ across the target corpora. For instance, the improvement on the Dutch corpus is 3.4%, on the other hand, its improvement of 0.6% is relatively smaller on the Zulu dataset. We believe the difference in improvements can be explained by the size of each corpus. The size of Dutch corpus is very small as shown in Table. TABREF3 , therefore the fine-tuned model is prone to overfit to the dataset very quickly. In contrast, it is less likely for a larger corpus to overfit. Compared with the fine-tuned model, our approach optimizes the model by gradually changing the temperature without quick overfitting. This mechanism could be interpreted as a built-in regularization. As a result, our model can achieve much better performance in small corpora by preventing the overfitting effect.

Datasets	We first describe our corpus collection. Table. TABREF3 lists all corpora we used in the experiments. There are 16 corpora from 10 languages. To increase the variety of corpus, we selected 4 English corpora and 4 Mandarin corpora in addition to the low resource language corpora. As the target of this experiment is low resource speech recognition, we only randomly select 100,000 utterances even if there are more in each corpus. All corpora are available in LDC, voxforge, openSLR or other public websites. Each corpus is manually assigned one domain based on its speech style. Specifically, the domain candidates are telephone, read and broadcast.

Experiments	To demonstrate that our sampling approach could improve the multilingual model, we conduct experiments on 16 corpora to compare our approach with the pretrained model and fine-tuned model.

Acknowledgements	This project was sponsored by the Defense Advanced Research Projects Agency (DARPA) Information Innovation Office (I2O), program: Low Resource Languages for Emergent Incidents (LORELEI), issued by DARPA/I2O under Contract No. HR0011-15-C-0114.

Experiment Settings	We use EESEN  for the acoustic modeling and epitran  as the g2p tool in this work. Every utterance in the corpora is firstly re-sampled into 8000Hz, and then we extract 40 dimension MFCCs features from each audio. We use a recent multilingual CTC model as our acoustic architecture  : The architecture is a 6 layer bidirectional LSTM model with 320 cells in each layer. We use this architecture for both the baseline models and the proposed model. Our baseline model is the fine-tuned model: we first pretrained a model by uniformly sampling from all corpora. After the loss converges, we fine-tune the model on each of our target corpora. To compare it with our sampling approach, we first train an acoustic model to compute the embeddings of all corpora, then the embeddings are used to estimate the similarity as described in the previous section. The initial temperature INLINEFORM0 is set to 0.01, and the growth rate is INLINEFORM1 . We evaluated all models using the phone error rate (PER) instead of the word error rate (WER).

Related Work	Multilingual speech recognition has explored various models to share parameters across languages in different ways. For example, parameters can be shared by using posterior features from other languages  , applying the same GMM components across different HMM states  , training shared hidden layers in DNNs  ,  or LSTM  , using language independent bottleneck features  ,  . Some models only share their hidden layers, but use separate output layers to predict their phones  ,  . Other models have only one shared output layer to predict the universal phone set shared by all languages  ,  ,  . While those works proposed the multilingual models in different ways, few of them have explicitly exploited the relatedness across various languages and corpora. In contrast, our work computes the relatedness between different corpora using the embedding representations and exploits them efficiently. The embedding representations have been heavily used in multiple fields. In particular, embeddings of multiple granularities have been explored in many NLP tasks. To name a few, character embedding  , subword embedding  , sentence embedding  and document embedding  . However, there are few works exploring the corpus level embeddings.

To understand how our corpus embeddings contribute to our approach, we rank those embeddings and show the top-2 similar corpora for each corpus in Table. TABREF17 . We note that the target corpus itself is removed from the rankings because it is the most related corpus to itself. The results of the top half show very clearly that our embeddings can capture the language level information: For most English and Mandarin corpora, the most related corpus is another English or Mandarin corpus. Additionally, the bottom half of the table indicates that our embeddings are able to capture domain level information as well. For instance, the top 2 related corpus for Amharic is Bengali and Swahili. According to Table. TABREF3 , those three corpora belong to the telephone domain. In addition, Dutch is a read corpus, its top 2 related corpora are also from the same domain. This also explains why the 1st related corpus of Mandarin (hk) is Bengali: because both of them are from the same telephone domain. To further investigate the domain information contained in the corpus embeddings, we train the corpus embeddings with an even larger corpora collection (36 corpora) and plot all of them in Figure.

On the contrary, our approach aims to optimize the multilingual acoustic model by modifying distributions across all the training corpora.

This can be implemented by setting INLINEFORM0 to be 0. DISPLAYFORM0  On the other hand, the fine-tuned model should only consider samples from the target corpus INLINEFORM0 , while ignoring all other corpora. We argue that this condition can be approximated by setting INLINEFORM1 to a very large number. As INLINEFORM2 and INLINEFORM3 if INLINEFORM4 , we can prove the statement as follows: DISPLAYFORM0  While both the pretrained model and the fine-tuned model are special cases of our approach, we note that our approach is more flexible to sample from related corpora by interpolating between those two extreme temperatures. In practice, we would like to start with a low temperature to sample broadly in the early training phase. Then we gradually increase the temperature so that it can focus more on the related corpora. Eventually, the temperature would be high enough so that the model is automatically fine-tuned on the target corpus. Specifically, in our experiment, we start training with a very low temperature INLINEFORM0 , and increase its value every epoch INLINEFORM1 as follows.

The main reason is that the number of corpora involved in most experiments is usually limited and it is not useful to compute corpus embeddings. The only exception is the multitask learning where many tasks and corpora are combined together. For instance, the language level (corpus level) embedding can be generated along with the model in machine translation  and speech recognition  . However, those embeddings are only used as an auxiliary feature to the model, few works continue to exploit those embeddings themselves. Another important aspect of our work is that we focused on the sampling strategy for speech recognition. While most of the previous speech works mainly emphasized the acoustic modeling side, there are also some attempts focusing on the sampling strategies. For instance, curriculum learning would train the acoustic model by starting from easy training samples and increasingly adapt it to more difficult samples  ,  . Active learning is an approach trying to minimize human costs to collect transcribed speech data  . Furthermore, sampling strategies can also be helpful to speed up the training process  . However, the goals of most strategies are to improve the acoustic model by modifying the sampling distribution within a single speech corpus for a single language.

For example, a conversation corpus might be more beneficial to another conversation corpus rather than an audio book corpus. In this work, we propose an effective sampling strategy (Corpus Relatedness Sampling) to take advantage of relations among corpora. Firstly, we introduce the corpus-level embedding which can be used to compute the similarity between corpora. The embedding can be estimated by being jointly trained with the acoustic model. Next, we compute the similarity between each corpus and the target corpus, the similarity is then used to optimize the model with respect to the target corpus. During the training process, we start by uniformly sampling from each corpus, then the sampling distribution is gradually updated so that more related corpora would be sampled more frequently. Eventually, only the target corpus would be sampled from the training set as the target corpus is the most related corpus to itself. While our approach differs from the pretrained model and the fine-tuned model, we can prove that those models are special cases of our sampling strategy. To evaluate our sampling strategy, we compare it with the pretrained model and fine-tuned model on 16 different corpora.

The reason is that we mainly focus on the acoustic model in this experiment. Additionally, some corpora (e.g.: Dutch voxforge) in this experiment have very few amounts of texts, therefore it is difficult to create a reasonable language model without augmenting texts using other corpora, which is beyond the scope of this work.

Corpus Relatedness Sampling	With the embedding INLINEFORM0 of each corpus INLINEFORM1 , we can compute the similarity score between any two corpora using the cosine similarity. DISPLAYFORM0  As the similarity reflects the relatedness between corpora in the training set, we would like to sample the training set based on this similarity: those corpora which have a higher similarity with the target corpus INLINEFORM0 should be sampled more frequently. Therefore, we assume those similarity scores to be the sampling logits and they should be normalized with softmax. DISPLAYFORM0  where INLINEFORM0 is the probability to sample INLINEFORM1 from INLINEFORM2 , and INLINEFORM3 is the temperature to normalize the distribution during the training phase. We argue that different temperatures could create different training conditions. The model with a lower temperature tends to sample each corpus equally like uniform sampling. In contrast, a higher temperature means that the sampling distribution should be biased toward the target corpus like the fine-tuning. Next, we prove that both the pretrained model and the fine-tuned model can be realized with specific temperatures. In the case of the pretrained model, each corpus should be sampled equally.

Next, we apply the language specific softmax to compute logits INLINEFORM4 and optimize them with the CTC objective  . The embedding matrix INLINEFORM5 can be optimized together with the model during the training process.

FIGREF18 . To create the plot, the dimension of each corpus embedding is reduced to 2 with t-SNE  . The figure demonstrates clearly that our corpus embeddings are capable of capturing the domain information: all corpora with the same domain are clustered together. This result also means that our approach improves the model by sampling more frequently from the corpora of the same speech domain.

DISPLAYFORM0   where INLINEFORM0 is the temperature of epoch INLINEFORM1 and INLINEFORM2 is a hyperparameter to control the growth rate of the temperature.

Approach	In this section, we describe our approach to compute the corpus embedding and our Corpus Relatedness Sampling strategy.

Question: By how much do they, on average, outperform the baseline multilingual model on 16 low-resource tasks?
Answer: 1.6% lower phone error rate on average