Adapt-and-Adjust: Overcoming the Long-tail Problem of Multilingual Speech Recognition

1 INTRODUCTION . Deploying a single Automatic Speech Recognition ( ASR ) model to recognize multiple languages is highly desired but very challenging for real-world multilingual ASR scenarios due to the wellknown long-tailed distribution challenge , namely , that some resource-rich languages like English have abundant training data , while the majority low-resource languages have varying amounts of training data . The recent popular end-to-end ( E2E ) monolingual ASR architecture ( Graves et al. , 2013 ; Chan et al. , 2015 ; Vaswani et al. , 2017 ) is promising to achieve state-of-the-art performance for resource-rich languages but suffers dramatically from the long tail of low-resource languages due to the lack of training data . This paper aims to investigate an end-to-end multilingual ASR framework where a single model is trained end-to-end from a pooled dataset of all target languages to improve the overall performance of multilingual ASR tasks , especially for low-resource languages . The long-tailed data distribution problem makes building an end-to-end multilingual ASR notoriously challenging . This imbalanced data setting poses a multitude of open challenges for multi-task training because the distribution of the training data is very skewed . These challenges stem from two aspects . First , very limited audio samples are available for low-resource languages , such as Kyrgyz , Swedish , and Turkish , while simultaneously , vast amounts of data exist from high-resource languages , such as English , French , and Spanish . Second , graphemes or subword labels follow a long-tailed distribution in ASR since some labels appear significantly more frequently , even for a monolingual setting . Furthermore , a multilingual system may include languages with writing scripts other than the Latin alphabet , such as Chinese or Cyrillic , that further worsen the skewness . To further illustrate the long-tail distribution in our study , Figure 1 shows the frequencies of sentence piece tokens in the curated multilingual dataset from CommonVoice ( Ardila et al. , 2020 ) . While a standard end-to-end multilingual training approach can improve overall performance compared with monolingual end-to-end approaches , it does not address the long-tail problem explicitly . One of the key challenges is the class imbalance issue , which will bias the multilingual model towards the dominant languages . To address this , one straightforward approach is to resample the training data ( Kannan et al. , 2019 ; Pratap et al. , 2020 ) during batch assembly . However , such an ad-hoc approach does not fully resolve the underlying long-tail distribution problem , and only a marginal improvement is obtained in practice . Another challenge is how to model the languages with limited training data robustly . In this paper , the “ long-tail problem ” is twofold : 1 ) the longtailed class distribution arising from the skewed multilingual data and sentence piece distribution 2 ) the robust modelling of languages with limited training data , i.e. , tail languages . To this end , we propose the Adapt-and-Adjust ( A2 ) framework for multilingual speech recognition using a speech transformer to address the twofold long-tail problem . Firstly , for better language modeling , a distilled mBERT ( Devlin et al. , 2019 ) is converted to an autoregressive transformer decoder to jointly explore the multilingual acoustic and text space to improve the performance of low-resource languages . Secondly , to adapt the multilingual network to specific languages with minimal additional parameters , both language-specific and language-agnostic adapters are used to augment each encoder and decoder layer . While the language-specific adapters focus on adapting the shared network weights to a particular language , a common adapter is proposed to learn some shared and language-agnostic knowledge for better knowledge transfer across languages . Lastly , to increase the relative margin between logits of rare versus dominant languages , we perform class imbalance adjustments during multilingual model training or inference by revisiting the classic idea of logit adjustment ( Zhou & Liu , 2006 ) . Class imbalance adjustment ( Collell et al. , 2016 ; Cui et al. , 2019 ; Menon et al. , 2020 ) is applied by adjusting the logits of the softmax input with the class priors . We conduct experiments and establish a benchmark from the CommonVoice corpus with a realistic long-tailed distribution of different languages . The extensive experiments show that A2 significantly outperforms conventional approaches for end-to-end multilingual ASR . Our key contributions are as follows : • We propose Adapt-and-Adjust ( A2 ) , a novel end-to-end transformer-based framework for real-world multilingual speech recognition to overcome the “ long-tail problem ” ; • We demonstrate the effectiveness of utilizing a pretrained multilingual language model as a speech decoder to improve multilingual text representations and language adapters to better share the learned information across all languages . To the best of our knowledge , this work is the first to adapt a pretrained multilingual language model for multilingual ASR . • We show that incorporating class priors during training or inference is effective and essential to addressing the long-tail distribution issue in multilingual training . • We establish a reproducible multilingual speech recognition benchmark with long-tailed distributions of 11 languages from different language families for the research community . 2 ADAPT-AND-ADJUST FRAMEWORK . 2.1 OVERVIEW . Figure 2 gives an overview of the proposed A2 framework for end-to-end multilingual ASR . A2 is built on a transformer-based sequence-to-sequence model with three key novel contributions : ( 1 ) an mBERT-based decoder , ( 2 ) language adapters , and ( 3 ) class-imbalance adjustments . Firstly , the vanilla transformer decoder is replaced with mBERT for better language modeling , particularly for low-resource languages . Secondly , the common and language-specific adapters are added to each encoder and decoder layer to learn both the shared and language-specific information for better acoustic modelling . Finally , we perform class imbalance adjustments during training or inference , where the logits are adjusted with the class priors estimated from the training data . 2.2 BASE MODEL : HYBRID CTC-ATTENTION SPEECH TRANSFORMER . A sequence-to-sequence speech transformer model ( Dong et al. , 2018 ; Kim et al. , 2016 ; Karita et al. , 2019b ) based on the hybrid CTC-Attention network is used for acoustic modeling . It takes in the acoustic features x ∈ RT×F and outputs the sentence piece tokens y , where T and F denote the sequence length and feature dimension . The encoder consists of several 2D convolution layers followed by self-attention layers . The convolution layers are used to extract more robust features before they are sent to the transformer . The decoder layers have two attention mechanisms , one for self-attention and the other for the encoder output . The network is trained in an autoregressive manner by predicting the next token given the current output . In addition , the CTC layer ( Graves et al. , 2006 ) is added to the encoder output to serve as a regularizer to the attention model . Training Multi-task loss LMTL ( Watanabe et al. , 2018 ; Karita et al. , 2019b ) , combining the CTC loss ( Graves et al. , 2006 ) and attention lossLATTN , is used to train the speech transformer . The multitask loss is computed as an interpolation of the two losses with a hyper-parameter λ ( 0 ≤ λ ≤ 1 ) : LATTN = KL ( pattn||py ) , ( 1 ) LMTL = λ logpctc ( y|henc ) + ( 1− λ ) Lattn , ( 2 ) where py is the label distribution after label smoothing ( Müller et al. , 2019 ) to prevent the model from making over-confident predictions . Kullback-Leibler divergence loss ( KL ) ( Kullback & Leibler , 1951 ) , is used for the attention loss . Decoding Beam search is used to predict the sentence pieces without any additional language models . The decoding score is computed as a weighted sum of both the CTC and attention network probabilities using β as the decoding parameter to balance them ( Karita et al. , 2019a ) : ŷ = argmax y∈Y∗ { βpctc ( y|henc ) + ( 1− β ) pattn ( y|henc , y′ ) } , ( 3 ) where y′ is the decoded sequence so far . 2.3 MULTILINGUAL BERT AS TRANSFORMER DECODER . For better language modeling , especially for low-resource languages , mBERT is used as the transformer decoder . Since mBERT is pre-trained on text data , it is essential to augment a cross-attention layer to the encoder output for each mBERT layer . The cross-attention and its self-attention layers are learned to “ align ” the acoustic and text spaces for the speech recognition . This is because the text space may diverge significantly from the acoustic space of the encoder output . Autoregressive mBERT Figure 3 depicts the adaptation of mBERT as an autoregressive transformer decoder . We copy the embeddings and self-attention parameters of mBERT into the decoder layers . Let t denote the current decoding step . The autoregressive decoder takes the current input token yt to predict the next token yt+1 . The mBERT embedding layer converts the input token to a vector representation . Subsequently , the cross-attention layer takes the encoder output henc as the key and value , and the self-attention output as the query , and computes the attention output . Vocabulary Mapping The vocabulary size of the vanilla mBERT is too large ( 119,547 tokens ) for training the end-to-end speech recognition system . Therefore , vocabulary mapping is performed to reduce the number of targets for the speech transformer . In this work , sentence pieces ( SP ) ( Kudo , 2018 ) are used as the target tokens . The SP models are trained on the transcriptions with a preset vocabulary size . In this work , we use a shared set of 5,237 tokens as the multilingual system ’ s vocabulary . The minimum number in the token set for the sentence piece model is 150 for all the monolingual systems , except Chinese with 2,265 tokens . The generated sentence piece tokens are then matched against the mBERT token set . During training , the embeddings of all tokens in the mBERT vocabulary are initialized with mBERT embeddings . 2.4 LANGUAGE ADAPTERS . Similar to Kannan et al . ( 2019 ) , lightweight residual language adapters are used for better acoustic modelling with minimal language-specific parameters to increase the model robustness to languages with limited resources . As shown in Figure 4 , in addition to the language-specific adapter for capturing the language-intrinsic knowledge , a common adapter is also trained to learn language-agnostic information in the multilingual data ; we call these Dual-Adapters . The language-specific and common adapters are denoted as Alang and Acom , respectively . Each adapter of layer l consists of a down-projection layer Wld , followed by a ReLU activation function , and an up-projection layer Wlu . The adapters take h l as the input , where hl is the self attention output of layer l. We compute the output of Adapter ( hl ) as follows for both the language-specific and common adapters : Adapter ( hl ) = Wlu ( ReLU ( W l d ( LayerNorm ( h l ) ) ) ) + hl . ( 4 ) The final adapter output is computed as ol = ollang + o l com . o l is then used as the input to the next encoder or decoder layer . We create a language mask to specify the language-specific adapters . 2.5 SENTENCE PIECE CLASS IMBALANCE ADJUSTMENTS . The sentence piece class imbalance problem is addressed by incorporating the class priors during training or inference via logit adjustments . Derived from a Bayesian point of view in Menon et al . ( 2020 ) for computer vision tasks , the softmax classifier with adjusted logits as input minimizes the balanced error across all classes . A natural adjustment is to scale the logits fy ( x ) by the inverse of the corresponding class prior πy . In log domain , the adjustment can be performed as follows : f adjy ( x ) = fy ( x ) − τ · log πy , ( 5 ) where τ > 0 is a hyper-parameter . The adjustment can be viewed as applying a class-dependent offset to re-weight each logit according to its class prior . Class priors The class priors are the natural frequencies of the sentence piece tokens estimated from the multilingual training data . To form a valid prior distribution , smoothing is applied to the raw counts according to Equation 6 for zero occurrence tokens : πy = { Ci C − 1 ( N−n0 ) ×C , ci > 0 1 n0×C , otherwise , ( 6 ) whereC is the total number of counts for all labels , n0 is the number of labels with zero occurrences , N is the number of classes and ci is the raw count of class i . Training phase class imbalance adjustments To incorporate the priors during training , the logits f decyt of the last decoder layer are adjusted before softmax according to the following : f decyt = w T y · Decoder ( henc , Embedding ( yt−1 ) ) ( 7 ) f adjyt = f dec yt − τ · log πyt , ( 8 ) padjyt = exp ( f adjyt ) ∑ y′t∈ [ N ] exp ( f adjy′t ) . ( 9 ) The adjusted softmax output vector padjy of the sequence is used to compute the KL loss and perform the backward propagation to update the model . yt−1 is the previous label available only during training . To reduce the training and inference discrepancy , scheduled sampling ( Bengio et al. , 2015 ) is commonly used for sequential classification tasks like speech recognition . During later training iterations , instead of using the ground truth label yt−1 for computing the logits , y′t−1 is chosen from the maximum prediction output of the current model to simulate the inference : y′t−1 = argmax y padjyt−1 . ( 10 ) If the scheduled sampling is used , the adjusted logits at step t will have influence over all of the following tokens in the current sequence . This is a crucial difference from the image classification task in Menon et al . ( 2020 ) . If τ is set to be 1 , the training phase logit adjustment becomes similar to the widely used label smoothing technique Müller et al . ( 2019 ) . However , in conventional label smoothing , the prior πy is usually a uniform distribution that is independent of the data . The logit adjustment applies a class-specific “ smoothing ” based on the class prior , and has been shown to be superior to the baseline with the standard label smoothing . Inference phase class imbalance adjustments Alternatively , the class priors can be incorporated during inference via logit adjustments . The decoding score is computed as follows : ŷ = argmax y∈Y∗ { βpctc ( y|henc ) + ( 1− β ) padjy } . ( 11 ) During beam search , the attention decoding scores padjy are computed in the same way as the scheduled sampling from the adjusted logits .

Summary: The paper proposes three additions to improve a monolithic multilingual end-to-end ASR system. The problem of training a monolithic multilingual ASR system is that using data from multiple languages does not necessary improve over individual monolingual systems. The three additions are a large multilingual language model, the use of language adapters, and smoothing on the token probabilities. Mixing the three additions in a specific way helps improve the average word error rates.