Multilingual sequence-to-sequence speech recognition: architecture, transfer learning, and language modeling

Abstract: Sequence-to-sequence (seq2seq) approach for low-resource ASR is a relatively new direction in speech research. The approach benefits by performing model training without using lexicon and alignments. However, this poses a new problem of requiring more data compared to conventional DNN-HMM systems. In this work, we attempt to use data from 10 BABEL languages to build a multi-lingual seq2seq model as a prior model, and then port them towards 4 other BABEL languages using transfer learning approach. We also explore different architectures for improving the prior multilingual seq2seq model. The paper also discusses the effect of integrating a recurrent neural network language model (RNNLM) with a seq2seq model during decoding. Experimental results show that the transfer learning approach from the multilingual model shows substantial gains over monolingual models across all 4 BABEL languages. Incorporating an RNNLM also brings significant improvements in terms of %WER, and achieves recognition performance comparable to the models trained with twice more training data.

The sequence-to-sequence (seq2seq) model proposed in BIBREF0 , BIBREF1 , BIBREF2 is a neural architecture for performing sequence classification and later adopted to perform speech recognition in BIBREF3 , BIBREF4 , BIBREF5 . The model allows to integrate the main blocks of ASR such as acoustic model, alignment model and language model into a single framework. The recent ASR advancements in connectionist temporal classification (CTC) BIBREF5 , BIBREF4 and attention BIBREF3 , BIBREF6 based approaches has created larger interest in speech community to use seq2seq models. To leverage performance gains from this model as similar or better to conventional hybrid RNN/DNN-HMM models requires a huge amount of data BIBREF7 . Intuitively, this is due to the wide-range role of the model in performing alignment and language modeling along with acoustic to character label mapping at each iteration.
In this paper, we explore the multilingual training approaches BIBREF8 , BIBREF9 , BIBREF10 used in hybrid DNN/RNN-HMMs to incorporate them into the seq2seq models. In a context of applications of multilingual approaches towards seq2seq model, CTC is mainly used instead of the attention models. A multilingual CTC is proposed in BIBREF11 , which uses a universal phoneset, FST decoder and language model. The authors also use linear hidden unit contribution (LHUC) BIBREF12 technique to rescale the hidden unit outputs for each language as a way to adapt to a particular language. Another work BIBREF13 on multilingual CTC shows the importance of language adaptive vectors as auxiliary input to the encoder in multilingual CTC model. The decoder used here is a simple INLINEFORM0 decoder. An extensive analysis on multilingual CTC mainly focusing on improving under limited data condition is performed in BIBREF14 . Here, the authors use a word level FST decoder integrated with CTC during decoding.
On a similar front, attention models are explored within a multilingual setup in BIBREF15 , BIBREF16 based on attention-based seq2seq to build a model from multiple languages. The data is just combined together assuming the target languages are seen during the training. And, hence no special transfer learning techniques were used here to address the unseen languages during training. The main motivation and contribution behind this work is as follows:
In this work, we use the attention based approach BIBREF1 as it provides an effective methodology to perform sequence-to-sequence (seq2seq) training. Considering the limitations of attention in performing monotonic alignment BIBREF18 , BIBREF19 , we choose to use CTC loss function to aid the attention mechanism in both training and decoding. The basic network architecture is shown in Fig. FIGREF7 .
Let INLINEFORM0 be a INLINEFORM1 -length speech feature sequence and INLINEFORM2 be a INLINEFORM3 -length grapheme sequence. A multi-objective learning framework INLINEFORM4 proposed in BIBREF17 is used in this work to unify attention loss INLINEFORM5 and CTC loss INLINEFORM6 with a linear interpolation weight INLINEFORM7 , as follows: DISPLAYFORM0 
The unified model allows to obtain both monotonicity and effective sequence level training.
 INLINEFORM0 represents the posterior probability of character label sequence INLINEFORM1 w.r.t input sequence INLINEFORM2 based on the attention approach, which is decomposed with the probabilistic chain rule, as follows: DISPLAYFORM0 
where INLINEFORM0 denotes the ground truth history. Detailed explanations about the attention mechanism is described later.
Similarly, INLINEFORM0 represents the posterior probability based on the CTC approach. DISPLAYFORM0 
where INLINEFORM0 is a CTC state sequence composed of the original grapheme set and the additional blank symbol. INLINEFORM1 is a set of all possible sequences given the character sequence INLINEFORM2 .
The following paragraphs explain the encoder, attention decoder, CTC, and joint decoding used in our approach.
In this work, the experiments are conducted using the BABEL speech corpus collected from the IARPA babel program. The corpus is mainly composed of conversational telephone speech (CTS) but some scripted recordings and far field recordings are presented as well. Table TABREF14 presents the details of the languages used in this work for training and evaluation.
80 dimensional Mel-filterbank (fbank) features are then extracted from the speech samples using a sliding window of size 25 ms with 10ms stride. KALDI toolkit BIBREF24 is used to perform the feature processing. The fbank features are then fed to a seq2seq model with the following configuration:
The Bi-RNN BIBREF25 models mentioned above uses a LSTM BIBREF26 cell followed by a projection layer (BLSTMP). In our experiments below, we use only a character-level seq2seq model trained by CTC and attention decoder. Thus in the following experiments we intend to use character error rate (% CER) as a suitable measure to analyze the model performance. However, in section SECREF26 we integrate a character-level RNNLM BIBREF27 with seq2seq model externally and showcase the performance in terms of word error rate (% WER). In this case the words are obtained by concatenating the characters and the space together for scoring with reference words. All experiments are implemented in ESPnet, end-to-end speech processing toolkit BIBREF28 .
Multilingual approaches used in hybrid RNN/DNN-HMM systems BIBREF10 have been used for for tackling the problem of low-resource data condition. Some of these approaches include language adaptive training and shared layer retraining BIBREF29 . Among them, the most benefited method is the parameter sharing technique BIBREF10 . To incorporate the former approach into encoder, CTC and attention decoder model, we performed the following experiments:
In this approach, the model is first trained with 10 multiple languages as denoted in table TABREF14 approximating to 600 hours of training data. data from all languages available during training is used to build a single seq2seq model. The model is trained with a character label set composed of characters from all languages including both train and target set as mentioned in table TABREF14 . The model provides better generalization across languages. Languages with limited data when trained with other languages allows them to be robust and helps in improving the recognition performance. In spite of being simple, the model has limitations in keeping the target language data unseen during training.
Table TABREF16 shows the recognition performance of naive multilingual approach using BLSTMP and VGG model against a monolingual model trained with BLSTMP. The results clearly indicate that having a better architecture such as VGG-BLSTM helps in improving multilingual performance. Except Pashto, Georgian and Tokpisin, the multilingual VGG-BLSTM model gave 8.8 % absolute gain in average over monolingual model. In case of multilingual BLSTMP, except Pashto and Georgian an absolute gain of 5.0 % in average is observed over monolingual model. Even though the VGG-BLSTM gave improvements, we were not able to perform stage-1 and stage-2 retraining with it due to time constraints. Thus, we proceed further with multilingual BLSTMP model for retraining experiments tabulated below.
To alleviate the limitation in the previous approach, the final layer of the seq2seq model which is mainly responsible for classification is retrained to the target language.
In previous works BIBREF10 , BIBREF29 related to hybrid DNN/RNN models and CTC based models BIBREF11 , BIBREF14 the softmax layer is only adapted. However in our case, the attention decoder and CTC decoder both have to be retrained to the target language. This means the CTC and attention layers are only updated for gradients during this stage. We found using SGD optimizer with initial learning rate of INLINEFORM0 works better for retraining compared to AdaDelta.
The learning rate is decayed in this training at a factor of INLINEFORM0 if there is a drop in validation accuracy. Table TABREF20 shows the performance of simply retraining the last layer using a single target language Assamese.
Based on the observations from stage-1 model in section SECREF22 , we found that simply retraining the decoder towards a target language resulted in degrading %CER the performance from 45.6 to 61.3. This is mainly due to the difference in distribution across encoder and decoder. So, to alleviate this difference the encoder and decoder is once again retrained or fine-tuned using the model from stage-1. The optimizer used here is SGD as in stage-1, but the initial learning rate is kept to INLINEFORM0 and decayed based on validation performance. The resulting model gave an absolute gain of 1.6% when finetuned a multilingual model after 4th epoch. Also, finetuning a model after 15th epoch gave an absolute gain of 4.3%.
To further investigate the performance of this approach across different target data sizes, we split the train set into INLINEFORM0 5 hours, INLINEFORM1 10 hours, INLINEFORM2 20 hours and INLINEFORM3 full set. Since, in this approach the model is only finetuned by initializing from stage-1 model, the model architecture is fixed for all data sizes. Figure FIGREF23 shows the effectiveness of finetuning both encoder and decoder. The gains from 5 to 10 hours was more compared to 20 hours to full set.
Table TABREF25 tabulates the % CER obtained by retraining the stage-1 model with INLINEFORM0 full set of target language data. An absolute gain is observed using stage-2 retraining across all languages compared to monolingual model.
In an ASR system, a language model (LM) takes an important role by incorporating external knowledge into the system. Conventional ASR systems combine an LM with an acoustic model by FST giving a huge performance gain. This trend is also shown in general including hybrid ASR systems and neural network-based sequence-to-sequence ASR systems.
The following experiments show a benefit of using a language model in decoding with the previous stage-2 transferred models. Although the performance gains in %CER are also generally observed over all target languages, the improvement in %WER was more distinctive. The results shown in the following Fig. FIGREF27 are in %WER. “whole” in each figure means we used all the available data for the target language as full set explained before.
We used a character-level RNNLM, which was trained with 2-layer LSTM on character sequences. We use all available paired text in the corresponding target language to train the LM for the language. No external text data were used. All language models are trained separately from the seq2seq models. When building dictionary, we combined all the characters over all 15 languages mentioned in table TABREF14 to make them work with transferred models. Regardless of the amount of data used for transfer learning, the RNNLM provides consistent gains across all languages over different data sizes.
As explained already, language models were trained separately and used to decode jointly with seq2seq models. The intuition behind it is to use the separately trained language model as a complementary component that works with a implicit language model within a seq2seq decoder. The way of RNNLM assisting decoding follows the equation below: DISPLAYFORM0 
 INLINEFORM0 is a scaling factor that combines the scores from a joint decoding eq.( EQREF13 ) with RNN-LM, denoted as INLINEFORM1 . This approach is called shallow fusion.
Our experiments for target languages show that the gains from adding RNNLM are consistent regardless of the amount of data used for transfer learning. In other words, in Figure FIGREF27 , the gap between two lines are almost consistent over all languages.
Also, we observe the gain we get by adding RNN-LM in decoding is large. For example, in the case of assamese, the gain by RNN-LM in decoding with a model retrained on 5 hours of the target language data is almost comparable with the model stage-2 retrained with 20 hours of target language data. On average, absolute gain INLINEFORM0 6% is obtained across all target languages as noted in table TABREF28 .
In this work, we have shown the importance of transfer learning approach such as stage-2 multilingual retraining in a seq2seq model setting. Also, careful selection of train and target languages from BABEL provide a wide variety in recognition performance (%CER) and helps in understanding the efficacy of seq2seq model. The experiments using character-based RNNLM showed the importance of language model in boosting recognition performance (%WER) over all different hours of target data available for transfer learning.
Table TABREF25 and TABREF28 summarizes, the effect of these techniques in terms of %CER and %WER. These methods also show their flexibility in incorporating it in attention and CTC based seq2seq model without compromising loss in performance.
We could use better architectures such as VGG-BLSTM as a multilingual prior model before transferring them to a new target language by performing stage-2 retraining. The naive multilingual approach can be improved by including language vectors as input or target during training to reduce the confusions. Also, investigation of multilingual bottleneck features BIBREF30 for seq2seq model can provide better performance. Apart from using the character level language model as in this work, a word level RNNLM can be connected during decoding to further improve %WER. The attention based decoder can be aided with the help of RNNLM using cold fusion approach during training to attain a better-trained model. In near future, we will incorporate all the above techniques to get comparable performance with the state-of-the-art hybrid DNN/RNN-HMM systems.

Figure and table captions:
Fig. 1: Hybrid attention/CTC network with LM extension: the shared encoder is trained by both CTC and attention model objectives simultaneously. The joint decoder predicts an output label sequence by the CTC, attention decoder and RNN-LM.
Table 1: Details of the BABEL data used for performing the multilingual experiments
Table 2: Experiment details
Table 4: Comparison of naive approach and training only the last layer performed using the Assamese language
Table 3: Recognition performance of naive multilingual approach for eval set of 10 BABEL training languages trained with the train set of same languages
Table 5: Stage-2 retraining across all languages with full set of target language data
Fig. 2: Difference in performance for 5 hours, 10 hours, 20 hours and full set of target language data used to retrain a multilingual model from stage-1
Table 6: Recognition performance in %WER using stage-2 retraining and multilingual RNNLM
Fig. 3: Recognition performance after integrating RNNLM during decoding in %WER for different amounts of target data

Questions about this paper:
What languages do they use?

Answers:
Train languages are: Cantonese, Bengali, Pashto, Turkish, Vietnamese, Haitian, Tamil, Kurdish, Tokpisin and Georgian, while Assamese, Tagalog, Swahili, Lao are used as target languages.