Unsupervised Cross-lingual Representation Learning for Speech Recognition

1 INTRODUCTION . Cross-lingual learning aims to build models which leverage data from other languages to improve performance . This has been a long standing interest in the speech community ( Byrne et al. , 2000 ; Le & Besacier , 2009 ; Ghoshal et al. , 2013 ; Huang et al. , 2013 ; Gales et al. , 2017 ; Cho et al. , 2018 ; Seki et al. , 2018 ) which includes systems able to transcribe multiple languages ( Burget et al. , 2010 ; Bourlard et al. , 2011 ; Heigold et al. , 2013 ; Toshniwal et al. , 2018 ; Kannan et al. , 2019 ) . However , the vast majority of work in speech processing has focused on supervised cross-lingual training which requires labeled data in multiple languages . Transcribed speech is often much scarcer than unlabeled speech and requires non-trivial human annotation . Unsupervised representation learning , or pretraining , does not require labeled data and has received a lot of recent attention in computer vision ( Tian et al. , 2019 ; He et al. , 2019 ; Chen et al. , 2020 ) after much success in natural language processing ( Peters et al. , 2018 ; Devlin et al. , 2018 ) . For the latter , cross-lingual pretraining has been shown to be very effective , particularly , for low resource languages ( Lample & Conneau , 2019 ; Conneau et al. , 2019 ) . In speech processing , most work in this area has focused on monolingual unsupervised representation learning ( van den Oord et al. , 2018 ; Chung & Glass , 2018 ; Schneider et al. , 2019 ; Chung et al. , 2019 ; Baevski et al. , 2020b ; Harwath et al. , 2020 ; Jiang et al. , 2019 ; Tjandra et al. , 2019 ; Eloff et al. , 2019 ; Baevski et al. , 2020a ) . In this paper , we focus on the cross-lingual setting by learning representations on unlabeled data that generalize across languages . We build on the pretraining approach of Baevski et al . ( 2020c ) which jointly learns contextualized speech representations as well as a discrete vocabulary of latent speech representations . The latter serves to effectively train the model with a contrastive loss ( § 2 ) and the discrete speech representations are shared across languages ( Figure 1 ) . Different to recent work on unsupervised cross-lingual pretraining , we fine-tune the Transformer part of the model instead of freezing all pretrained representations ( Rivière et al. , 2020 ) or feeding them to a separate downstream model ( Kawakami et al. , 2020 ) . We extend the work of Rivière et al . ( 2020 ) by pretraining on multiple languages instead of just English and we experiment on top of a stronger baseline . We evaluate XLSR on 14 languages of the BABEL benchmark ( Gales et al. , 2014 ) which is conversational telephone data and ten languages of CommonVoice ( Ardila et al. , 2019 ) , a corpus of read speech ( § 3 ) . Multilingual pretraining outperforms monolingual pretraining in most cases , except for resource rich languages and we show that increased model capacity significantly closes the gap . We Chinese English French Multilingual quantized latent speech representations Shared Transformer encoder …… Masked Transformer CNN q q q q q L ` Contrastive loss Shared CNN encoder Speech signal In any language ( e.g . English ) Shared quantizer Figure 1 : The XLSR approach . A shared quantization module over feature encoder representations produces multilingual quantized speech units whose embeddings are then used as targets for a Transformer trained by contrastive learning . The model learns to share discrete tokens across languages , creating bridges across languages . Our approach is inspired by Devlin et al . ( 2018 ) ; Lample & Conneau ( 2019 ) and builds on top of wav2vec 2.0 ( Baevski et al. , 2020c ) . It requires only raw unlabeled speech audio in multiple languages . also demonstrate that XLSR representations can be fine-tuned simultaneously on multiple languages to obtain a multilingual speech recognition system whose performance is competitive to fine-tuning a separate model on each language § 4 ) . 2 APPROACH . Unsupervised cross-lingual representation learning has shown great success by pretraining Transformers ( Vaswani et al. , 2017 ) with multilingual masked language models ( Devlin et al. , 2018 ; Lample & Conneau , 2019 ) . In this work , we learn cross-lingual speech representations by extending wav2vec 2.0 ( Baevski et al. , 2020c ) to the cross-lingual setting . Our approach learns a single set of quantized latent speech representations which are shared across languages . Next , we outline the architecture ( § 2.1 ) , training ( § 2.2 ) and adaptations for cross-lingual training . 2.1 ARCHITECTURE . We follow the design choices described in Baevski et al . ( 2020c ) . The model contains a convolutional feature encoder f : X 7→ Z to map raw audioX to latent speech representations z1 , . . . , zT which are fed to a Transformer network g : Z 7→ C to output context representations c1 , . . . , cT ( Devlin et al. , 2018 ; Baevski et al. , 2020b ; a ) . For the purpose of training the model , feature encoder representations are discretized to q1 , . . . , qT with a quantization module Z 7→ Q to represent the targets in the self-supervised learning objective ( Figure 1 , § 2.2 ) . The quantization is based on product quantization ( Jegou et al. , 2011 ; Baevski et al. , 2020b ) by choosing quantized representations from G = 2 codebooks with V = 320 entries each . The result is concatenated to obtain q . A Gumbel softmax enables choosing discrete codebook entries in a fully differentiable way ( Jang et al. , 2016 ) . Each zt represents about 25ms of audio strided by 20ms , the context network architecture follows BERT ( Vaswani et al. , 2017 ; Devlin et al. , 2018 ) except for relative positional embeddings Mohamed et al . ( 2019 ) ; Baevski et al . ( 2020a ) . 2.2 TRAINING . The model is trained by solving a contrastive task over masked feature encoder outputs . For masking , we sample p = 0.065 of all time steps to be starting indices and mask the subsequent M = 10 time steps . The objective requires identifying the true quantized latent q̃ for a masked time-step within a set of K = 100 distractors Qt sampled from other masked time steps : − log exp ( sim ( ct , qt ) ) ∑ q̃∼Qt exp ( sim ( ct , q̃ ) ) where ct is the output of the transformer , and sim ( a , b ) denotes cosine similarity . This is augmented by a codebook diversity penalty to encourage the model to use all codebook entries ( Dieleman et al. , 2018 ) . We maximize the entropy of the averaged softmax distribution over the codebook entries for each group p̄g across a batch of utterances : 1GV ∑G g=1−H ( p̄g ) = 1 GV ∑G g=1 ∑V v=1 p̄g , v log p̄g , v . To stabilize the feature encoder we apply an L2 penalty over the outputs of the feature encoder . When pretraining on L languages , we form multilingual batches ( Devlin et al. , 2018 ; Lample & Conneau , 2019 ) by sampling speech samples from a multinomial distribution ( pl ) l=1 , ... , L where pl ∼ ( nl N ) α , nl being the number of pretraining hours of language l , N the total number of hours , and α the upsampling factor . The parameter α controls the importance given to high-resource versus low-resource languages during pretraining . 3 EXPERIMENTAL SETUP . 3.1 DATASETS . CommonVoice . The CommonVoice dataset1 is a multilingual corpus of read speech comprising more than two thousand hours of speech data in 38 languages ( Ardila et al. , 2019 ) . The amount of data per language ranges from three hours for Swedish ( `` low-resource '' ) to 353 hours for French and 1350 hours for English ( `` high-resource '' ) . Following Rivière et al . ( 2020 ) we consider ten languages : Spanish ( es ) , French ( fr ) , Italian ( it ) , Kyrgyz ( ky ) , Dutch ( du ) , Russian ( ru ) , Swedish ( sv ) , Turkish ( tr ) , Tatar ( tt ) and Chinese ( zh ) ; as well as English ( en ) for pretraining . We use the November 2019 release for training models , and for fine-tuning we use the evaluation splits of Rivière et al . ( 2020 ) which include one hour labeled data for training , 20 minutes for validation and one hour for testing . This few-shot evaluation dataset consists of phoneme sequences as output and we report phone error rate ( PER ) similar to prior work . BABEL . This dataset2 is a multilingual corpus of conversational telephone speech from IARPA , which includes Asian and African languages ( Gales et al. , 2014 ) . We adopt the same setup as Cho et al . ( 2018 ) and pretrain on ten languages : Bengali ( bn ) , Cantonese ( zh ) , Georgian ( ka ) , Haitian ( ht ) , Kurmanji ( ku ) , Pashto ( ps ) , Tamil ( ta ) , Turkish ( tr ) , Tokpisin ( tp ) , Vietnamese ( vi ) . We evaluate crosslingual transfer on four other languages ( models are not pretrained on these languages ) : Assamese ( as ) , Tagalog ( tl ) , Swahili ( sw ) , Lao ( lo ) . We train a multilingual model in ten languages and monolingual models in 14 languages . We use the same speech audio for pretraining and fine-tuning , and no unlabeled speech provided by BABEL . We use the dev folder of the BABEL dataset as our test set as `` eval '' has not been open-sourced , and use 10 % of the training set as dev data . We report character error rate ( CER ) . All audio is resampled to 16kHz . For comparison with Inaguma et al . ( 2019 ) only , we train 4-gram n-gram language models on CommonCrawl data ( Heafield et al. , 2013 ; Wenzek et al. , 2019 ) for Assamese ( 140MiB of text data ) , Swahili ( 2GiB ) , Tamil ( 4.8GiB ) and Lao ( 763MiB ) . 3.2 TRAINING DETAILS . Pretraining Models are implemented in fairseq ( Ott et al. , 2019 ) . We evaluate two architectures with the same feature encoder ( § 2.1 ) but different Transformer settings : Base with 12 blocks , model dimension 768 , inner dimension ( FFN ) 3072 and 8 attention heads ; and Large with 24 blocks , model dimension 1024 , inner dimension 4096 and 16 attention heads ; both use dropout 0.1 . For Base , we crop 250k samples , or 15.6sec of audio , and pack up to 1.4m samples on each GPU . For Large , we crop 320k samples and put up to 1.2m samples on a GPU . Batches are sampled using a factor α ∈ { 0.5 , 1 } . We use 16 GPUs for small datasets ( typically monolingual ) and 64 GPUs for large datasets ( typically multilingual ) , and use Adam ( Kingma & Ba , 2015 ) where the learning rate is warmed up for the first 10 % of updates to a peak of 1e-5 ( Base ) or 1e-3 ( Large ) , and then linearly decayed over a total of 250k updates . Fine-tuning . To fine-tune the model we add a classifier representing the output vocabulary of the respective downstream task on top of the model and train on the labeled data with a Connectionist 1 https : //voice.mozilla.org/en/languages 2 https : //catalog.ldc.upenn.edu/byyear , includes LDC2018S07 , LDC2018S13 , LDC2018S02 , LDC2017S03 , LDC2017S22 , LDC2017S08 , LDC2017S05 , LDC2017S13 , LDC2017S01 , LDC2017S19 , LDC2016S06 , LDC2016S08 , LDC2016S02 , LDC2016S12 , LDC2016S09 , LDC2016S13 , LDC2016S10 . Temporal Classification ( CTC ) loss ( Graves et al. , 2006 ; Baevski et al. , 2020a ) . Weights of the feature encoder are not updated at fine-tuning time . We determine the best learning rates setting in [ 2e-5 , 6e-5 ] based on dev set error rate . The learning rate schedule has three phases : warm up for the first 10 % of updates , keep constant for 40 % and then linearly decay for the remainder . For CommonVoice we fine-tune for 20k updates and on BABEL for 50k updates on 2 GPUs for the Base model and 4 GPUs for the Large model .

Summary: Authors extended the XLSR model from the previous mono-lingual task, where self-supervised learning was used in representation learning, to multi-lingual task. The basic idea does make a lot of sense, where all the data, irrespective of the language, is pushed through the representation learning task.  We are anyways talking about human speech, so it is reasonable to assume that learning one language can help in learning the other language.