Adapt-and-Adjust: Overcoming the Long-tail Problem of Multilingual Speech Recognition
Authors: Genta Indra Winata
Guangsen Wang
Caiming Xiong
Steven Hoi
Venue:  (2021)
Keywords: speech recognition
multilingual
long-tail
adapter
logit adjustments

Abstract: One crucial challenge of real-world multilingual speech recognition is the long-tailed distribution problem, where some resource-rich languages like English have abundant training data, but a long tail of low-resource languages have varying amounts of limited training data. To overcome the long-tail problem, in this paper, we propose Adapt-and-Adjust (A2), a transformer-based multi-task learning framework for end-to-end multilingual speech recognition. The A2 framework overcomes the long-tail problem via three techniques: (1) exploiting a pretrained multilingual language model (mBERT) to improve the performance of low-resource languages; (2) proposing dual adapters consisting of both language-specific and language-agnostic adaptation with minimal additional parameters; and (3) overcoming the class imbalance, either by imposing class priors in the loss during training or adjusting the logits of the softmax output during inference. Extensive experiments on the CommonVoice corpus show that A2 significantly outperforms conventional approaches.

Under review as a conference paper at ICLR 2021
ADAPT-AND-ADIUS OVERCOMING THE LONG-TAIL PROBLEM OF MULTILINGUAL SPEECH RECOGNITION
Anonymous authors Paper under double-blind review
ABSTRACT
One crucial challenge of real-world multilingual speech recognition is the long- tailed distribution problem, where some resource-rich languages like English have abundant training data, but long tail of low-resource languages have varying amounts of limited training data. To overcome the long-tail problem, in this pa- per, we propose Adapt-and-Adjuu ing framework for end-to-end multilingual speech recognition. The A2 frame- work overcomes the long-tail problem via three techniques: (1) exploiting a pre- trained multilingual language model (mBERT) to improve the performance low-resource languages; (2) proposing dual adapters consisting of both language- specific and language-agnostic adaptation with minimal additional parameters; and (3) overcoming the class imbalance, either by imposing class priors in the loss during training or adjusting the logits of the softmax output during inference. Extensive experiments on the Common Voice corpus show that A2 significantly outperforms conventional approaches.
(A2), transformer-basee multi-task Iearn-
INTRODUCTION
Deploying a single Automatic Speech Recognition (ASR) model to recognize multiple languages highly desired but very challenging for real-world multilingual ASR scenarios due the well- known loigetailed distribution challenge. namely, that some resource-rich languages like English have abundant training data, while the majority low-resource languages have varying amounts of training data. The recent popular end-to-end (F:2F) monolingual ASR architecture (Graves et al.) 2013 Chan al., 2013] Vaswani al. 2017) is promising to achieve state-of-the-at performance for resource-rich languages but suffers dramatically from the long tail of low-resource languages due to the lack of training data. This paper aims to investigate an end-to-end multilingual ASR framework where single model is trained end-to-end from a pooled dataset of all target languages to improve the overall performance of multilingual ASR tasks, especially for low-resource languages.
The long-tailed data distribution problem makes building an end-to-end multilingual ASR notori- ously challenging. This imbalanced data setting poses a multitude of open challenges for multi-task training because the distribution of the training data is very skewed. These challenges stem from two aspects. First, very limited audio samples are available for low-resource languages, such as Kyrgyz, Swedish, and Turkish, while simultaneously, vast amounts of data exist from high-resource languages, such as English, French, and Spanish. Second, graphemes or subword labels follow a long-tailed distribution in ASR since some labels appear significantly more frequently, even for monolingual setting. Furthermore, a multilingual system may include languages with writing scripts other than the Latin alphabet, such as Chinese or Cyrillic, that further worsen the skewness. To fur- ther illustrate the long-tail distribution in our study, Figure Ishows the frequencies of sentence piece tokens in the curated multilingual dataset from Common Voice (Ardila al.) 2020).
While a standard end-to-end multilingual training approach can improve overall performance com- pared with monolingual end-to-end approaches, it does not address the long-tail problem explicitly. One of the key challenges is the class imbalance issue, which will bias the multilingual model to- wards the dominant languages. To address this, one straightforward approach is to resample the training data (Kannan et al. 2019 Pratap al. 2020) during batch assembly. However, such an ad-hor hoc approach does not fully resolve the underlying long-tail distribution problem. and only a marginal improvement is obtained in practice. Another challenge is how to model the languages

---

Under review as a conference paper at ICLR 2021
head
long-tail tail classes only show freq P20K
word pieces
Figure 1: The long-tail distribution of sentence piece tokens in the curated multilingual dataset. The head classes are tokens with high frequency, otherwise, they are classified tail classes
with limited training data robustly. In this paper, the "long-tail problem" is twofold: 1) the long- tailed class distribution arising from the skewed multilingual data and sentence piece distribution 2) the robust modelling of languages with limited training data, i.e., tail languages.
To this end, we propose the Adapt-and-Adjust (A2) framework for multilingual speech recognition using speech transformer to address the twofold long-tail problem. Firstly, for better language modeling, a distilled mBERT (Devlin et al., 2019) is converted to an autoregressive transformer decoder to jointly explore the multilingual acoustic and text space improve the performance of low-resource languages. Secondly, to adapt the multilingual network to specific languages with minimal additional parameters, both language-specific and language-agnostic adapters are used to augment each encoder and decoder layer. While the language-specific adapters focus on adapting the shared network weights particular language, common adapter is proposed to learn some shared and Language-agnostee knowledge for better knowledge transfer across languages. Lastly, to increase the relative margin between logits rare versus dominant languages, we perform class imbalance adjustments during multilingual model training or inference by revisiting the classic idea of logit adjustment (Zhou Liu, 2006). Class imbalance adjustment (Collell et al. [016) Cui al.) 2019 Menon et al. (020) is applied by adjusting the logits of the softmax input with the class priors. We conduct experiments and establish a benchmark from the Common Voice corpus with a realistic long-tailed distribution of different languages. The extensive experiments show that A2 significantly outperforms conventional approaches for end-to-end multilingual ASR.
Our key contributions
follows:
We propose Adapt-and-Adie Adjuss (A2), a novel end-to-end vaasforrmrrbbsee framework for real-world multilingual speech recognition to overcome the "long-tail problem";
We demonstrate the effectivennss of utilizing a pretrained multilingual language model as a speech decoder to improve multilingual text representations and language adapters to better share the learned information across all languages. To the best of our knowledge, this work is the first to adapt pretrained multilingual language model for multilingual ASR.
We show that incorporating class priors during training or inference is effective and essen- tial to addressing the long-tail distribution issue in multilingual training.
We establish reproducible multilingual speech recognition benchmark with long-tailed distributions of 11 languages from different language families for the research community.
2 ADAPT-AND-ADJUST FRAMEWORK
2.1 OVERVIEW
Figure 2 gives an overview of the proposed A2 framework for end-to-end multilingual ASR. A2 requence-to-sequees
model with three key novel contributions: (1) an mBERT-based decoder, (2) language adapters, and (3) class-imbalance adjustments. Firstly, the vanilla transformer decoder is replaced with mBERT for better language modeling, particularly for low-resource languages. Secondly, the common and language-sper specccifc adapters are added to each information for better
built on transformer-basee
encoder and decoder layer to leamn both the shared and language-spectn acoustic modelling. Finally, we perform class imbalance adjustments during training or inference, where the logits adjusted with the class priors estimated from the training data.
2

---

Under review as a conference paper at ICLR 2021
Lrnal
1-2
m
Pattn
Softmax
Adjust
Byy
f (X)
M foeely--1r henc)
Feed
Feest-Forvard
Nx
Languago Adaptors
hanc
Language Adapters
XN
Self-Atzention
Cross Actention
Transfer
X
Conv2d
Self-Attention
Emhaddings
Embeddings
Audio features
Y-11
mBERTT
<ecr>
love cat
Figure 2: Overview of the Adapt-and-Adiuu framework. The layer norm is omitted to save space. Pae is the connectionist temporal classification (CTC) output, Pan is the decoder output, yr-1 is the previous token. The three key modules are (1) pre-trained mBERT-based decoding to improve language modelling; and (2) dual adapters improve acoustic modelling, all particularly for tail languages; and (3) class-imbalance adjustment of the logits face ( hene) by class priors Ty.
2.2 BASE MODEL: HYBRID CTC-ATTENTION SPEECH TRANSFORMER
A sequence-to-sequene speech transformer model (Dong et al., 2018 Kim et al. 2016 Karita al. 2019b) based on the hybrid CTC-Attention network is used for acoustic modeling. It takes in the acoustic features x E RT'F and outputs the sentence piece tokens y. where T and F denote sequence length and feature dimension. The encoder consists several 2D convolution layers followed by self-attention layers. The convolution layers used extract more robust features before they are sent to the transformer. The decoder layers have two attention mechanisms, one for self-attention and the other for the encoder output. The network is trained in autoregressive manner by predicting the next token given the current output. In addition, the CTC layer (Graves) al. 2006 is added to the encoder output serve regularizer the attention model.
Training Multi-tas task loss LMTL (Watanabe et al. 2018 Karita et al. 2019b), combining the CTC loss (Grvess et al. (2066) and attention loss CATIN, is used train the speech transformer. The multi- task loss is computed as an interpolation of the two losses with a hyper-paran parameter À (0 s A < 1):
LXITN (L(Puullly)) IPy),
LMTL A Alog (y hene) + (1 Camm
(2)
where Py is the label distribution after label smoothing (Müller et al. 2019) prevent the model from making over-confident predictions. Kullback-Leible divergence loss (KL) (Kullback &
Leibler 1951). used for the attention loss.
Decoding Beam search is used to predict the sentence pieces without any additional language models. The decoding score is computed as a weighted sum of both the CTC and attention network probabilities using 3 as the decoding parameter to balance them (Karita et al. 2019a):
y
yEy' arg max (.peac (y) /een)) + (1 - P)) (y [ene, Y')}
where y' is the decoded sequence so far.
2.3
MULTILINGUAL BERT AS TRANSFORMER DECODER
For better language modeling, especially for low-resource languages, mBERT is used as the trans- former decoder. Since mBERT is pre-trained text data, it is essential augment cross-attention
3

---

Under review as a conference paper at ICLR 2021
layer to the encoder output for each mBERT layer. The cross- and its self-attention layers are learned to "align" the acoustic and text spaces for the speech recognition. This is because the text space may diverge significantly from the acoustic space of the encoder output.
Autoregressive mBERT Figure depicts the adaptation mBERT as an autoregressive trans- former decoder. We copy the embeddings and self-attention parameters of mBERT into the decoder layers. Let denote the current decoding step. The autoregressive decoder takes the current input token Ye predict the next token M1+1- The mBERT embedding layer converts the input token to a vector representation. Subsequently, the cross-attention layer takes the encoder output hene as the key and value, and the self-attention output as the query, and computes the attention output.
outgut
output
Food Forward
Layer Norm
Cross-Atteetion
Layer Norm
N
Layer Norm
Trarsfar
Self-Aitention
SelfAAttention
Layer Norm
Embedeings
1oput
Inpot
Multiingual Dacoder
Figure 3: Parameter transfer from a pre-trained multilingual language model to speech recognition decoder. The dotted line shows the transfer direction from a specific module of the model.
Vocabulary Mapping The vocabulary size of the vanilla mBERT too large (1 19,547 tokens) for training the end-to-end speech recognition system. Therefore, vocabulary mapping is performed to reduce the number of targets for the speech transformer. In this work, sentence pieces (SP) (Kudo) 2018) are used as the target tokens. The SP models are trained the transcriptions with a preset vocabulary size. In this work, we use shared set of 5,237 tokens as the multilingual system's 's vocabulary. The minimum number in the token for the sentence piece model 150 for all the monolingual systems, except Chinese with 2,265 tokens. The generated sentence piece tokens are then matched against the mBERT token set. During training, the embeddings of all tokens in the mBERT vocabulary are initialized with mBERT embeddings.
2.4 LANGUAGE ADAPTERS
Similar to Kannan et al. (2019), lightweight residual language adapters are used for better acoustic modelling with minimal language-specifte parameters to increase the model robustness languages with limited resources.
shown in Figure addition to the language-specific adapter for captur- ing the language-ing intrinsic knowledge, a common adapter is also trained to learn language-agnostic information the multilingual data; we call these Dual-Adapters. The inguage-sppecfii and com- mon adapters are denoted as Atang and Awom. respectively. Each adapter of layer / consists of down-projeene layer W followed by ReLU activation function, and up-projection layer W! The adapters take In' the input, where hn' is the self attention output of layer l. We compute the output of Adapter(h') as follows for both the language-specift and common adapters:
Adapter(h's = w a (ReLU (Re (LayerNorm(hh))) + h
The final adapter output computed o' o ouum- o' is then used as the input to the next encoder decoder layer. We create language mask to specify the language-specte adapters.
2.5 SENTENCE PIECE CLASS IMBALANCE ADJUSTMENTS
The sentence piece class imbalance problem is addressed by incorporating the class priors during training inference via logit adjustments. Derived from Bayesian point of view in Menon et al.
4

---

Under review as a conference paper at ICLR 2021
Projection
Pnng
com
ReLU
Aarg
EN
ES
ZH
Com
Acom
Down Projection
Layer Norm
M
Figure 4: Dual-Adapters. The orange box presents the active language-spece speciffc adapter and the blue box presents the common adapter. The figure on the right shows the language adapter structure.
2020) for computer vision tasks, the softmax classifier with adjusted logits input minimizes the balanced error across all classes. A natural adjustment is to scale the logits fy (x) by the inverse of the corresponding class prior Ty. In log domain, the adjustment can be performed as follows:
full (x) fo(x) 5 log Ty.
(5)
where > O is hyper-parameter. The adjustment can be viewed as applying a class-dependent offset to re- re-weight each logit according to its class prior.
Class priors The class priors the natural frequencies of the sentence piece tokens estimated from the multilingual training data. To form a valid prior distribution, smoothing is applied to the raw counts according to Equation 6 for zero occurrence tokens:
C
(N (N-Ro)xC* > 0
Ty
(6)
otherwise,
1ig xC
where is the total number of counts for all labels, no is the number labels with zero occurrences, N is the number of classes and c; is the count of class i.
Training phase class imbalance adjustments To incorporate the priors during training, the logits plec of the last decoder layer are adjusted before softmax according the following:
dds
w Decoder(h hene
(7)
yt
(ad) pdoe
log Typ
(8)
exp(Fad)
radi)
EMElN exp(F (inl))
The adjusted softmax output vector Py adj of the sequence is used to compute the KL loss and perform the backward propagation to update the model. 8t-1 is the previous label available only during training. To reduce the training and inference discrepancy, scheduled sampling (Bengio et al. 2015) commonly used for sequential classification tasks like speech recognition. During later training iterations, instead of using the ground truth label yt-1 for computing the logits, V chosen from the maximum prediction output of the current model to simulate the inference:
V/ = argmax adj
(10)
If the scheduled sampling is used, the adjusted logits at step ł will have influence over all of the following tokens in the current sequence. This is a crucial difference from the image classification task in Menon et al. (2020). If is set to be 1, the training phase logit adjustment becomes similar to the widely used label smoothing technique Muller et al. (2019). However, in conventional label smoothing, the prior ny is usually uniform distribution that is independent of the data. The logit adjustment applics a class- slessssp "smoothing" based on the class prior, and has been shown to be superior to the baseline with the standard label smoothing.
5

---

Under review as a conference paper at ICLR 2021
Inference phase class imbalance adjustments during inference via logit adjustments. The decoding score is computed as follows:
Alternatively, the class priors can be incorporated
arg max (ppec (y (ene) + (1 3) p)
(11)
yey-
During beam search, the attention decoding scores Py are computed in the same way as the sched- uled sampling from the adjusted logits.
3
EXPERIMENTS
3.1 EXPERIMENTAL SETUP
Dataset The CommonVoice dataset (Ardila et al., 2020) is multilingual corpus collected by Mozilla. Similar toCConnaau et al. (2020a), use I1 languages: English (en), Spanish (es), French (fr), Italian (it), Kyrgyz (ky), Dutch (nl), Russian (ru), Swedish (sv), Turkish (tr), Tatar (it), and Chinese (zh). The dataset is split into training, dev, and eval sets according to the ESPNET recipe. The transcriptions are tokenized using the SentencePiece model with the unigram algorithm. We train our SentencePiece model using speech transcriptions, and the size of the vocabulary is shown in Table TO We then add special tokens, such as <unk>>, <sos>, <eos>> and a blank token for the CTC objective. The detailed data split is shown in Ta Appendix C.
Network configurations We use six transformer encoder layers with a hidden size of 2048 units and eight attention heads, each with an attention dimension of 256. For the decoder, distil- mBERT (Sanh et al. 2019) is used. The mBERT decoder consists of six transformer decoder layers with hidden size 3072 and an attention dimension of 756, and four attention heads. We train our model with batch size of 32 and accumulate the gradient in two steps to have larger batch size using a single GPU NVIDIA V100 16GB. The models are trained with the Adam optimizer with a warm-up step of 25000. In particular, for balanced sampling, we take six samples for each language and construct a balanced batch by accumulating the gradient 11 times.
Training and Evaluation We evaluate our model using beam-search with beamwidth of 10 and = 0.3 and / 0.5. The hyper-parameter T is set to 0.3 for both the training and inference phase class imbalance adjustments. The multilingual models are trained with 150K iterations. We compute the average over the last ten checkpoints as the decoding model. For the monolingual setting, we stop after 100 epochs of training. Models are evaluated using the character error rate (CER) to simplify the evaluation and to have a universal metric for all languages.
Baselines As baseline approaches, we consider the following: Monolingual: we train monolin- gual models; SMT (Standard Multilingual Training), we randomly sample the batch from the data distribution; BS (Balanced Sampling), we sample the same number utterances for each language in batch so that they have roughly equal contributions to the training; LAN-Specific Adapters: language-specifg adapters proposed by Kannan et all 2019); and LID: (language ID) conditioning with one-hot language vectors proposed by Li al. 2018).
3.2 RESULTS
In [ we present the test results on the Common Voice dataset. Compared to the monolingual models, even the SMT models improve the performance of the low-resource languages significantly. In other words, SMT is decent multilingual baseline to be compared with. We conjecture that this may be because the multilingual models can capture common sub-phonetic articulatory fea-
tures [Kirchhoff et al.) 22022] Metze] 2005 Livescu et al. 2007 Wang & Sim (014) that are shared by different languages and are beneficial for low-resource languages recognition.
Balanced Sampling We observe the same trend in Kannan et al. 2019); compared to the SMT, the tail language performance is significantly boosted. However, the performance of the head lan- guages suffers due to fewer occurrences during training. The model is clearly over-fitted to the tail languages due to up-sampling, for example, the CERs on the training set of "ky" and "sv" are sig- nificantly lower than the evaluation data (3.4% and 4.2% training vs 13.4% and 22.8% evaluation).

---

Under review as a conference paper at ICLR 2021
Consequently, the overall performance is the same SMT. In fact, after balanced sampling, the sentence piece tokens still have a long-tailes distribution, as shown in Appendix
Table 1: Test results in terms of CER (%) on the Common Voice dataset.
high-resource
intermediate
low-resource
Model
en
fr
es
ru
zh
tt
nl
tr
ky
Sv
avg
Truining hours
877
273
132
66
57
33
20
19
10
9
4
Monolingual (full)
13.3
11.5
11.1
20.7
10.3
28.2
13.6
23.0
28.0
30.0
56.1
22.3
Training hours
80
4
Monolingual
22.6
20.1
17.3
20.7
23.9
37.6
20.7
38.1
30.3
31.6
57.7
29.1
SMT
20.1
17.4
13.0
13.7
34.1
12.2
18.7
13.9
14.4
26.4
17.9
BS LAN-Specific Adapters
25.2
20.3
14.5
12.7
13.2
32.6
11.5
18.0
12.9
13.4
22.8
17.9
24.2
19.4
13.9
12.3
11.8
32.0
10.7
16.9
11.8
12.7
21.7
17.0
LID
25.7
19.2
13.7
12.0
12.0
31.6
10.8
16.4
12.0
12.5
21.8
17.1
LID + Adjust-Truee
24.7
18.5
13.0
11.3
11.6
31.2
10.4
16.4
11.6
12.4
22.0
16.6
A2 (Adjust-Inference)
22.6
17.9
12.7
11.2
11.2
30.1
10.2
15.8
11.4
12.2
21.1
16.0
A2 (Adjust-Train)
22.0
17.7
12.5
11.3
11.1
30.4
10.0
15.9
11.3
12.1
21.3
16.0
Language Adapters We next compare the language adaptation techniques, the LAN-Specine Specific Adapters (Kanman et al. 2019), the one-hot language vector (Li al. 2018), and the Dual-Adapters based on BS mBERT, which has better performance than the BS-only
Note that all adapters model. Adding the language-specing adapters without common adapters significantly outperforms the BS baseline, with 0.9% absolute performance gain. Another way of injecting language formation is to augment one-hot language vector. Interestingly, applying sentence piece class imbalance adjustment (LID Adjust- Train) to the language vector significantly improves the CER.
Sentence Piece Class Imbalance Adjustment Both the training and inference phase adjustments provide significant performance gain over the LAN-Specific Adapters, with 1 % absolute CER re- duction. The gains are mostly due to the improved performance of the head languages, although tail languages also benefit from the logit adjustments. More importantly, the gap between the mono- lingual and multilingual performance for the head languages is greatly reduced, leading to better "balanced error" performance. This strongly justifies the importance of class imbalance adjustments. Compared to BS, A2 also avoids over-fitting to the tail languages, CERs on "ky" and "sv" are 8.2% and 23.6%, much closer to evaluation CERs. Compared to SMT with random sampling, A2 has significant better averaged CER with a modest cost for the two head languages "fr" and "en". Some example transcriptions and detailed analysis using different A2 modules are given in Appendix E.
3.3
ABLATION STUDIES
3.3.1
MULTILINGUAL BERT
The effectiveness of MBER is presented in Table The performance of mBERT depends heavily on the quality of the acoustic models. Without adapters or logit adjustments, the improvement over BS is marginal, and mBERT performance is even worse for SMT. This may indicate that, with better acoustic models like A2, the text space of the vanilla mBERT is better aligned with the acoustic space, which leads to improved performance across all languages, especially for low-resource ones. is also interesting note that, even without adapters, "SMT mBERT Adjust- Train" yields the same overall CER as the best adapter system (BS mBERT Dual Dual-Adapters).
To study the impacts of the pretrained language models, more advanced XLM-R (Conneau al., 2020b) pretrained model is used in place of the distilled-mBERRT. Although XLM-- has a better multilingual language generation capability than mBERT, it does not translate to the final perfor- mance gain for the multilingual ASR task. We believe this is because it becomes more difficult for the model to align the text and acoustic space with the increased model complexities for XLM- R. XLM-R is not advised considering it has more parameters compared to Sistlleed-BBRRT and the performance gain is not significant although does improve the performance on "zh" and "fr" slightly.
7

---

Under review as a conference paper at ICLR 2021
Table 2: Ablation study on mBERT for multilingual speech recognition.
en
fr
e5
it
ru
zh
11
lr
ky
SY
avg
SMT
20.1
17.4
13.0
12.5
13.7
34.1
12.2
18.7
13.9
14.4
26.4
17.9
SMT mBERT
17.9
13.0
12.4
14.0
26.2
BS
25.2
20.3
12.7
13.2
32.6
11.5
18.0
12.9
13.4
22.8
BS MBERT
25.0
20.0
15.4
12.6
13.2
32.9
11.1
17.3
12.6
12.8
22.8
17.8
BS Dual-Adapters
18.9
12.1
12.3
31.0
10.9
16.5
12.0
21.6
BS Dual-Adapters mBERT
23,4
18.6
13.4
11.8
11.7
30.8
10.8
16.2
11.6
12.4
21.5
16.5
SMT Adjust-Tras
20.2
16.9
12.5
11.9
13.1
32.9
11.3
18.5
13.5
13.9
25.3
17.3
SMT + Adjust-Tras Train mBERT
19.4
16.5
12.1
11.7
12.2
31.1
11.0
17.5
12.7
13.2
24.6
16.5
A2 with mBERT
22.0
17.7
12.5
11.3
11.1
30.4
10.0
15,8
11.3
12.1
21.3
16.0
A2 with XLM-R
22.1
17.6
12.5
11.4
11.4
29.6
10.3
15.9
11.5
12.1
21.4
16.0
3.3.2 LANGUAGE ADAPTERS
The results and parameter sizes of different adapters are given in Table 3
Generally speaking, de- coder layer adapters are not effective in the encoder layers, indicating adaptation of the acoustic space is much more effective than of the text space. Therefore, considering the extra computation and parameters, advisable to apply only the encoder adapters.
Table 3: Ablation study on language adapters' impacts on BS mBERT models
#Params
cm
fr
it
ru
zh
tt
nl
tr
ky
SV
avg
No Adapter Decoder Adapter Only
76M
25.0
20.0
12.6
13.2
32.9
11.1
17.3
12.6
12.8
22.8
82M
23.6
19.2
12.2
11.7
32.5
10.3
16.7
11.7
12.2
21.4
16.8
Encoder Adapter Only
78M
23.1
18.5
13.3
11.
11.7
31.1
10.6
16.0
11.5
12.6
21.3
16.5
Encoder + Decoder
84M
23.4
18.6
13.4
11.
11.7
30.8
10.8
16.2
11.6
12.4
21.5
16.5
We investigate the effectiveness of the common language adapters in Table 4 outperform the language-specinn adapters significantly, by 0.5% absolute CER reduction, indicat- ing knowledge transfer with the common adapter is effective.
The Dual-Adapters
Table 4: Ablation study of language adapters.
iPaaams
ea
fr
it
xh
11
ml
tr
ky
SV
avg
LAN-Specific Adapters
24.2
12.3
10.7
16,9
21.7
17.0
Individual Dual-Adapters
84M
23,4
18.6
13.4
11.8
11.7
30,8
10.8
16.2
11.6
12.4
21.5
16,5
Langunge Group Dual Adaplers
Writen Scripts
12.1
16.3
16.8
By Language Families
78M
23.5
19.0
13.6
11.9
11.4
31.0
10.4
16.2
11.6
11.9
21.4
16.5
In addition to the individual language adapters, we also divide languages into groups to allow sharing of adapters within the same group. According the written scripts, divide the 11 languages into language groups, e.g., Latin, Chinese characters and Cyrillic scripts. They can also be groups into language families, e.g., Romance, Chinese, Turkic, Germanic. This group focuses similarities in lexica, grammars, and pronunciations, which are usually subsumed under the end-to- end multilingual architectures. According one group, languages that belong to the same cluster do not necessarily belong to the same cluster in the other group. For example, Tartar and Turkish are both Turkic languages. However, Tartar uses the Cyrillic script, and Turkish uses the Latin alphabet. All languages in the same group share the same dual-adapters, and the adapters are trained with all language members. In general, grouping by language families is better than grouping by written scripts because it is more consistent with the encoder adapters for adapting the acoustic space, which are more effective than decoder adapters Table 3 Compared individual language adapters, sharing language adapters by language families helps the low-resource languages performance, e,g. "sv" the Germanic group, "ky" and "tr" of the Turkic group because more data are used to train the group adapters. However, this also comes with a cost to the essouree-rih languages compared to "Individual Dual- Adal-ede Therefore, individual language adapters are advised considering the adapters' parameter sizes much smaller than the encoder and decoder attention weights.
the
8

---

Under review as a conference paper at ICLR 2021
3.3.3 SENTENCE PIECE
ADJUSTMENTS: TRAINING VS. INFERENCE
The logit adjustments compared in Table For the SMT systems, training phase adjustment shows a clear advantage over inference phase adjustment. Under the convex assumption, the solution of the two adjustment approaches the same. However, deep neural network optimization a non- convex problem, so they may converge to different local minima. Under SMT, the model is heavily biased towards the head classes due to random sampling. Training phase class imbalance adjustment can help the training to place more focus on the tail classes, leading to much better balanced and lower error. With better acoustic models, egg,, language adapters, the inference phase adjustment can better calibrate the raw classification scores and yield similar performance to the training phase adjustment. Lastly, we show the effects of on inference phase logit adjustment in Appendix D.
Table 5: Training and inference phase logit adjustments with different A2 models.
en
ex
ky
sv
avg
SMT + mBERT +
76M
20.4
17.9
13.9
13.0
14.0
34.1
12.4
18.9
14.0
14.3
26.2
18.1
Adjust-Inference Adjust- Train
76M
20.1
17.2
13.4
12.2
13.2
34,0
11.5
18.3
13.4
13.2
26.1
17.6
76M
19.4
16.5
12.1
11.7
12.2
31.1
11.0
17.5
12.7
13.2
24.6
16.5
mBBRTT+ +
76M
25.0
20.0
15,4
12.6
13.2
32,9
11.1
17.3
12.6
12.8
22.8
17.8
Adjust-Infernnce
76M
24.1
19.3
13.5
12.0
12.1
32.3
10.6
16.7
12.2
12.6
22.2
17.0
Adjjss- Train
76M
23,8
19.1
13.4
11.9
11.8
32.4
10.5
16.7
12.1
12.6
21.5
16.9
mBERT Dual-Adapters
84M
23.4
18.6
13.4
11.8
11.7
30.8
10.8
16.2
11.6
12.4
21.5
16.5
Adjust. Inference
84M
22.6
17.9
12.7
11.2
11.2
30.1
10.2
15.8
11.4
12.2
21.1
16.0
Adjust-Trune Trai
84M
22.0
17.7
12.5
11.3
11.1
30.4
10,0
15.9
11.3
12.1
21.3
16.0
4 RELATED WORK
Long-Tail Conventional approaches to addressing the long-tail problem are focused on data sam-
pling methods (Kubat & Matwin 1997 Chawla et al. 2002, Wallace et al. 2011). Recently, the long-tail distribution issue has regained interest for neural network models (Menon et al.) 2020). and
several approaches have been proposed, such as weight normalization (Kang et al. 2019). adaptive
margin (Cao al. 2019), and equalized loss (Tan al. 2020).
Adapters Adapters were first proposed to learn domain-specific representations in computer vi- sion in a parameter-effiiient way (Rebuffi et al., 2017). They were subsequently adopted for NLP tasks to avoid fine-tuning new model for each task by training an adapter module for each task while sharing the pre-trained language model parameters (Houlsby al. 2019 Lin et 2020). Invertible adapters were proposed in Pfeiffer et al. (2020a b) to effectively adapt an existing pre- trained multilingual model to unseen languages for multi-task cross-lingual transfer.
Multilingual ASR E2E architectures like LAS (Tohniiall et al. 2018 and the Recurrent Neural Transducer (Kannan al., 2019) have been used for building multilingual ASR system for group of Indian languages. In Kannan et al. (2019), language adapters are used to tackle the data imbalance problem, although the improvement from using language adapters marginal compared to the language vector augmentation. Acoustic vector quantization is also used in the recent work by Conneau et al. (2020a) on multilingual ASR. A massive multilingual ASR study with more than 50 languages and more than 16,000 hours of speech is presented in Pratap et al. (2020). The two main techniques are data resampling and language clusters, which bear some similarities with our balanced sampling and language adapters. Unfortunately, their datasets are not publicly available.
5 CONCLUSION
In this paper, we introduce Adapt-and-Adjust (A2), end-to-end transformer-base framework to address the crucial challenge of long-tail data distribution issues in multilingual speech recognition. A2 consists of three major components, namely. language adapters, class imbalance adjustments, and a pretrained multilingual language model. Extensive experiments on the Common Voice corpus show that A2 significantly outperforms conventional approaches for end-to-end multilingual speech recognition due to 1) the better acoustic modeling with adapters and class imbalance adjustments; and 2) the better language modeling with pretrained multilingual BERT.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. Common voice: A mussively-mulillinee speech corpus. In Proceedings The 12th Language Resources and Eval- uation Conference, pp. 4218 4222, 2020.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. CoRR, abs/1506.03099, 2015.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-dissributionaaarr margin loss. In Advances in Neural Information Processing Systems, pp. 1567-1578, 2019.
William Chan, Navdeep Jaitly, Quoe V. Le, and Oriol Vinyals. Listen, attend and spell. CoRR, htss/508.012111 2015. bttp://://aaxii org/ abs/ 110./118...00 1508.01211
Nitesh V Chawla, Kevin W Bowyer, Lawrence Hall, and Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357, 2002.
Guillem Collell, Drazen Prelec, and Kaustubh Patil. Reviving threshold-moving: a simple plug-in bagging ensemble for binary and multiclass imbalanced data. arXiv preprint arXiv: 08698, 2016.
Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, and Michael Auli. Unsupervised cross- cross-lingual representation learning for speech recognition. arXiv preprint arXiv:2006. 13979, 2020a.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pp. 8440-8451, Online, July 2020b. Association for Computational Linguistics. doi: 10. 1atp5/////220..l-mmmiaa 747. https:
//www. aclweb.ww............. org/ http://annnnooooo2oo..eeee
main. 747
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9268-9277, 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Papers), pp. 4171-4186, 2019.
I.. Dong, S. Xu, and B. Xu. Speech- treechttransformer: A no-recurrence sequence-to-sequenee model for speech recognition. In 2018 IEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5888-5888, 2018.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist tem- poral classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pp. 369-376, 2006.
Alex Graves, Abdel-rahman Mohamed, and Geoffrey E. Hinton. Speech recognition with deep recurrent neural networks. CoRR, abs/1303.5778, 2013. URL bttp://arxiv.org//ar 13C3.5778
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efffiient transfer learning for nlp. In International Conference on Machine Learning, pp. 2790-2799, 2019.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailes recognition. In International Conference Learning Representations, 2019.
10

---

Under review as a conference paper at ICLR 2021
Anjuli Kannan, Arindrima Datta, Tara N Sainath, Eugene Weinstein, Bhuvana Ramabhadran, Yonghui Wu, Ankur Bapna, Zhifeng Chen, and Seungji Lee. Large-scale multilingual speech recognition with streaming end-to-end model. Proc. Interspeech 2019, pp. 2130-2134, 2019.
Shigeki Karita, Nelson Enrique Yalta Soplin, Shinji Watanabe, Mare Delcroix, Atsumori Ogawa, and Tomohiro Nakatani. Improving transformer- mannformer-baeedd end-to-end speech recognition with con- nectionist temporal classification and language model integration. In Proc. Interspeech 2019, pp. www.doo.rooo.oo www.ddd....o........../ 1408-1412, 2019a. doi: 10.214337/mterrppecchh200 201 1938. URL http:
21437/Interspest
1011-19993
Shigeki Karita, Nelson Enrique Yalta Soplin, Shinji Watanabe, Mare Delcroix, Atsunori Ogawa, and Tomohiro Nakatani. Improving transformer-based end-to-end speech recognition with con- nectionist temporal classification and language model integration. Proc. Interspeech 2019, pp. 1408-1412, 2019b.
Suyoun Kim, Takaaki Hori, and Shinji Watanabe, Joint etc-attention based speech recog- nition using multi-task learning. CoRR, abs/1609.06773, 2016. URL http://arxiv.crr
abs/1609.065 06773
Katrin Kirchhoff, Gernot A Fink, and Gerhard Sagerer. Combining acoustic and articulatory
feature information for robust speech recognition. Speech Communicattion 37(3):303 319, 2002. ISSN 0167-6393. doi: cttp///eeaeeeearrraaae//////////
URL http:
/ //www. www.:scceeeeeiccccoo rect. wtt.
Miroslav Kubat and Stan Matwin. Addressing the curse of imbalanced training sets: One-sided selection. In ICML, 1997.
Taku Kudo. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pp. 66-75, 2018.
Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathe- matical statistics, 22(1):79-88) 86, 1951.
Bo Li, Tara N Sainath, Khe Chai Sim, Michiel Bacchiani, Eugene Weinstein, Patrick Nguyen, Zhifeng Chen, Yanghui Wu, and Kanishka Rao. Multi-dialect speech recognition with sin- gle sequence-to-ses te-seueecc model. In 2018 IEEE international conference on acousties, speech and signal processing (ICASSP), pp. 4753. IEEE, 2018.
Zhaojiang Lin, Andrea Madotto, and Pascale Fung. Exploring versatile generative language model via parameter-efficient transfer learning. arXiv preprint arXhv:2004.03829, 2020.
K. Livescu, O. Cetin, M. Hosegawa-Johnson, S. King, C. Bartels, N. Borges, A. Kantor, P. Lal, L. Yung, A. Bezman, S. Dawson Dawson-Haggerty, B. Woods, J. Frankel, M. Magimai-Dost and K. Saenko. Articulatory feature- based methods for acoustic and audio-visual speech recogni- tion: Summary from the 2006 jhu summer workshop. In 2007 IEEE International Conference on Acoustics, Speech and Signal Processing ICASSP '07, volume 4, pp. IV-621-IV-624, 2007. doi: ALL109ICCASSSP22033666898
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. arXiv preprint arXiv:2007.07314, 2020.
Metze, Articulatory features for conversational speech recognition. 2005.
Rafael Müller, Simon Korbllith, and Geoffrey E Hinton. When does label smoothing help? In Advances in Neural Information Processing Systems, pp. 4694 2019.
Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić. Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. Adapterhub: A framework for adapting transformers. arXiv preprint wxXiv:2007007777 2020a.
Jonas Pfeiffer, Ivan Vulié, Iryna Gurevych, and Sebastian Ruder. Mad-x: An adapter- based frame- work for multi-task cross-lingual transfer. arXiv preprint arXiv:2005.000) 00052, 2020b.
11

---

Under review as a conference paper at ICLR 2021
Vineel Pratap, Anuroop Sriram, Paden Tomasello, Awni Hannun, Vitaliy Liptchinsky, Gabriel Syn- naeve, and Ronan Collobert. Massively multilingual asr: 50 languages, model, billion param- eters, 2020.
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. In Advances in Neural Information Processing Systems, pp. 506-516, 2017.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:/990.01 108, 2019.
Jingru Tan, Changbao Wang, Buyu Li, Quanquan Li, Wanli Ouyang, Changqing Yin, and Junjic Yan. Equalization loss for long-tailed object recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11662-11671, 2020.
Shubham Toshniwal, Tara N. Sainath, Ron J. Weiss, Bo Li, Pedro Moreno, Eugene Weinstein, and Kanishka Rao. Multilingual speech recognition with a single end-to-end model. 2018 IEEE In- ternational Conference on Acoustics, Speech and Signal Processing (ICASSP), Apr 2018. doi: 10. (0t.9///cassp.....8880800 8461972
URL fttp://ddx doi org/ 110.1 RAA.O/IICCASPPP 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, pp. 5998-599980 2017.
Byron C Wallace, Kevin Small, Carla E. Brodley, and Thomas A Trikalinos. Class imbalance, redux. In 2011 IEEE 1 11th international conference on data mining, pp. 754-763. IEEE, 2011.
Guangsen Wang and Khe Chai Sim. Regression- based context-dependent modeling of deep neural networks for speech recognition. IEEF/ACM Trans. Audio, Speech and Lang. Proc., 22(11): 1660-1669, November 2014. ISSN 2329-9290.
Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, et al. Espnet: End-to-end speech processing toolkit. arXiv preprint
2018
Zhi-Hua Zhou and Xu- Yu-Ying Liu. Training cost-sensitive neural networks with methods addressing the class imbalance problem. IEEE Trans. on Knowl. and Data Eng., 18(1):63-77 (-776,3-77770 January 2006. ISSN 1041-4347.
12

---

Under review as a conference paper at ICLR 2021
A A2 ALGORITHM
The full algorithm of the A2 framework with training phase sentence piece class imbalance adjust- ments is outlined Algorithm I.
Algorithm Adapt-and- (A2) with Adjust-Train
Require: D: long-tailed multilingual data
Require: 0: encoder-decoder model
Require: a, A: step size hyperparameters
1: randomly initialize 0
2: copy mBERT pretrained language model to decoder parameters in 0
3: compute class priors и from DD
4: while not done do
5: Sample batch of multilingual utterances x
6:
Generate language adapter mask m using the language tag in x
7:
Compute encoder hidden states hase using x and by encoder forwarding
8:
Compute logits fdee using hene and m by decoder forwarding
9:
Adjust logit according Equation 9
10:
Compute CTC posteriors Pete (y hene))
11:
Compute attention loss CATIN in Equation
12:
Compute multi-task loss LMIL using Paw (h/e) CATIN, and À in Equation 2
13:
Update model <- 0 a VoLLL
14:
end while
B ASR MODEL STRUCTURE
The encoders and decoders of the transformer- based speech recognition model used in this study is presented here.
hene
Feed: Forward
f
Acapter
uudkal
Acapter
rosidaal
Lays Nam
Layer Norm
rrsiduul
Sali-Attn
Layer Norm
Layer Norm
hert
Figure 5: Encoder structure.
Figure Decoder structure.
Encoder Layer Figure 5 shows the structure of the encoder layer. An adapter layer is added after the layer norm and self-attention module. We also apply two residual connections after both the self-attention layer and the adapter layer:
o = SelfAtt((LayrNoom((b).)) + b
(12)
h BaaFFooward(Aaaptt(((aaaee
o),
(13)
where h!-!! is the encoder hidden states of the previous layer 1 and h/m is the output of the encoder layer.
13

---

Under review as a conference paper at ICLR 2021
Decoder Layer Figure 6 shows the structure of decoder layer with the adapter. We place the adapter layer after the cross-attention model.
0 SeffAttm(LayerNomM(Baat h + 02),
(14)
O2 TroosAAttthbaeeLLvvevvvooo/o/// +
(15)
[+1 Tecdforvan((AdapeeeLLvvveeeenn dec
(16)
where h is the decoder hidden states of the previous layer, and h is the output the current layer.
C DATASET DISTRIBUTION
Table showss the data split used for training, validation, and testing. Thooss the frequencies of the labels in the original data and estimates the balance sampling. There only few data with an occurrence of more than 50k (around I 1%). The class distribution on the balance sampling is shifted the 10k-50k range; thus, there are more labels in the middle of the distribution.
Table 6: Dataset split information
nigh-resource
intermediate
low-resource
en
fr
es
ru
ky
Train (hours)
80.0
50.0
40.0
20.0
15.0
15.0
15.0
10.0
10.0
9.0
4.0
Valid (hours)
109.0
34.0
16.5
8.3
7.3
4.2
2.5
2.5
1.3
1.1
0.5
Test (hours)
10.0
10.0
10.0
8.3
7.3
4.3
2.5
2.5
1.3
1.1
0.5
Table
Class frequency in the original data and after applying balanced sampling
100k
50k-100k
10k-50k
Ik- 10k
1k
Original data
6
9
152
1164
3905
Balance sampling
12
30
306
960
3243
Figure 7 and Figure 8 visualizes the long-tail distribution in more detailed. To better visualize them, we ignore the classes fewer than 20k.
word piaces
Figure 7: Class Frequency. We only show labels more than 20k
100
word pieces
Figure 8: Class Frequency with Balance Sampling. We only show labels more than 20k
14

---

Under review as a conference paper at ICLR 2021
D ABLATION STUDY ON INFERENCE PHASE LOGIT ADJUSTMENT
Figure 9 depicts the CERs of the systems with different 7 for inference phase logit adjustment with the best A2 configuration: balanced sampling with mBERT and Dual-Adapters. We choose one language from each group, and the other languages demonstrate the same trend. In Figure the advantage the logit adjustment clearly shown compared to the baseline without logit adjustment (T 0). All of the languages achieve their best CER with value of [0.3, 0.4]. Performance saturates with heavier smoothing (T >= 0.5).
12.0
18.5
12.8
11.5
12.6
0
11.0
122
17.0
10.5
12.0
0.1
0.2
0.3
0.4
0.5
0.1
0.2
(a) High-resource (fr)
(b) Intermediate (it)
(c) Low-resource (ky)
Figure 9: Comparison of models with different T.
TRANSCRIPTION EXAMPLES
We provide samples two languages generated from different models, eesurrce-liiiee language zh (traditional Chinese scripts and Mandarin spoken in Taiwan) and resource-rich language (English).
Table 8: Transcription of a testing traditional Chinese utterance with different models
Reference
/y
Pinyin
Kun4 Nan2 Yu3 Tiao3 Zhan4 Shi4 Jil Fat Wo3 Men2 Deo Yuan2 Dong4 Li4
Model
Hypothesis
CER
BS mBERT
P ME BL
21.4
Dual-Adupters
25 3 BLL
14.3
Dual-Adapters Adjust-Train
B BU 22
14.3
Monolingual
A
64.3
SMT
fke H
42,9
The reference in Table & roughly translates to English as "Obstacles and challenges are the sources of power that motivate us." The monolingual output is rather poor. Not only did it miss impor- tant character l for distinguishing "my" and "our", the sounding out of the sentence is also quite different from the reference (Fu4 Neng2 Yi2 Piao4 Zhan4 Shi4 Wo3 De0 Yuan2 Dong4 Li4). The standard multilingual training (SMT) improves the monolingual training by predicting the character BC (Yu3) in place of (Yi2). In addition, the output of a (Tiao2 Zhan4) sounds almost the same as the reference Ma (Tiao3 Zhan4). (The monolingual system output as fth (Piao4 Zhan4) not only does not make any sense in terms of word meaning, the pronunciation also quite different from the reference.) This shows that the multilingual training helps improve the acoustic modelling com- pared to the monolingual training if the monolingual training data are limited. The mBERT model further helps the language modeling by correcting nj B Ab (literally translates to "possibility Bd Wiit ("obstacles and challenges"). However, mBERT also re- (Jil Fal) to # (Zil Liao4) since #T ("documents/materiils is more common word than #t which is a wrong word with correct pronunciations relative to the reference. Lastly, as
and changing stations") to IN 0
places Kt fft
the adapters successfully convert #t (Zil Liao4) to Ke (Jil so that it sounds the the reference with one wrong character. This demonstrates the adapter's s capability in better acoustic modelling. Finally the errors including the wrong characters in THE (Jil and F( / (Yuan2
15

---

Under review as a conference paper at ICLR 2021
Dong4 Li4) can be easily corrected by decent external language model during decoding, e.g., RNN-LM trained on the training transcriptions.
Table 9: Transcription testing English utterance with different models
Reference
FOUND LITTLE CROWD ABOUT TWENTY PEOPLE SURROUNDING THE HUGE HOLE
Model
Hypthessss
CER
mBEKT
SOUND THE LITTLE CROWD THE ABOUT TWENTY PEOPLE SURROUNDING THE HUGE HOLE
13.6
Dual-Adapiers
SOUND LITTLE CROWD ABOUT SPENT PEOPE SURROUUDDNGG HUGE
11.9
Dual--Aaateess Adjus-Train
Monolingual
SAW THE LEVEL CROWD AS ABOUT TWENTY PEOPLE SURROUNDING THE PUISH FOR
33.9
SMT
SOUND THE LITTLE CROWD AS ABOUT TWENTY PEOPLE SU SURBOUNDING THE HUGE HOLE
13.6
Table 2 shows the transcriptions of different models applied to an English testing utterance. Dif- ferent from Table 8 where only limited training data are available. mBERT does not provide any improvement over SMT, except that makes the sentence more grammatical by adding a subject, "T" The Dual-Adapters improve the performance compared mBERT by successfully decoding the sentence piece "ENT" "SPENT", which is closer to "TWENTY" in terms of characters. Lastly, the class imbalance adjustment system yields the best CER.
F VOCABULARY COVERAGE
Table 10: Vocabulary coverage
righuresourre
intermediate
low-resource
en
fr
it
ru
zh
tt
nl
tr
ky
SV
multi
Coverage (%)
74.4
73.6
70.8
73.9
90.2
97.7
91.3
64.7
57.
91.1
53.8
54.9
Match
181
281
192
164
1.56
2,330
157
110
93
154
86
2,875
# Vocab
243
382
271
222
173
2,386
172
170
163
169
160
5,237
G Loss DYNAMICS
We plot the validation losses from A2 and the other baselines in Figures 10 and 11
NT
NT
555
WHBET Adaptars (Adjsttrent Loss)
MEBAT
31-
HeBKT Adapiars
HEBATT
A2 (Adduttmeent Luss!
30-
29
5.5-
28-
20008
33002
60000
100000
50083
60000
70:00
900.0
100000
Baratios
12a08.00 ons
Figure 10: Validation losses.
Figure 11: Closer look at validation losses.
H LANGUAGE GROUP ADAPTERS
The language groups used in the experiment shown in Table 11 We also present the detailed CER results in 12 sentence piece class imbalance adjustments. The inference phase logit adjustments is slightly better than the training phase adjustment.
16

---

Under review as a conference paper at ICLR 2021
Table 11: Language groups used in the adapters.
Groups
Languages
By Written Scripts
Latin
en, fr, es, it, tr, sv zh
Chinese
Cyrillic
ru, tt. ky
By Language Families
Germanic
en, nl
Romance
fr, it, sv
Turkic
tr, tt, ky
Chinese
zh
Table 12: Character error rate (CER) using different language group adapters.
high-resource
intermediate
low-resource
fr
it
zh
tt
ky
avg
Language Families
23.5
19.0
13.6
11.9
11.4
31.0
10.4
16.2
11.6
11.9
21.4
16.5
Adjust- Train
23.1
18.4
13.2
11.4
11.3
31.5
10.1
16.1
11.7
12.4
20.6
16.3
Adjust-Inference
22.7
18.2
12.9
11.3
11.1
30.7
10.0
15.9
11.5
11.8
20.7
16.1
Language Scripts +
24.0
19.4
13.9
12.1
11.7
32.1
10.5
16.3
12.0
12.0
21.0
Adjust- Train
23.5
18.9
13.5
11.7
11.6
31.5
10.4
16.4
11.9
12.1
21.2
16.6
Adjust-Inferenct
23.3
18.6
13.1
11.5
11.5
31.5
10.0
16.0
11.8
11.8
20.8
16.4
I FEW-SHOT SETTING
We also investigate the effectiveness of the A2 framework the few-shot setting. We take five-hour speech data of each language the training data. The test results shown in Table [3] Using mBERT as the speech decoder consistently improves the performance in the low-resource monolin- gual scenario. Interestingly, for Chinese language "zh", without using mBERT the decoder, the model seems to be unable to learn useful information and the mBERT decoder yields an absolute CER reduction of 17.2%.
For multilingual training, marginal performance gains are observed with mBERT decoder, indicat- ing the multilingual language model complementary to the shared acoustic knowledge learned from the multilingual speech data. The language adapters lead to consistent and significant perfor- mance gain across all languages. Note the sentence piece class imbalance adjustments do not yield significant performance boost in the long-tail multilingual setting in the main text. We believe this is largely due to less severe sentence piece class imbalance when the same amount of training data for each language is used.
Table 13: Character error rate (CER) five-hour Common Voice training data
high-resouree
intermediate
low-resource
en
es
it
ru
zh
tt
nl
tr
ky
sv
avg
Monolingual
Standard Training (ST)
69.6
61.1
56.5
49.8
49.6
96.4
32.8
57.5
49.6
43.7
56.1
56.5
ST mBERT
67.3
56.1
51.0
45.4
42.8
73.2
29.6
52.9
46.
39.6
55.1
50.8
Mudtilingual
SMT
41.4
33.5
25.2
22.3
21.9
43.0
15.5
24.9
18.7
18.9
28.9
26.7
SMT mBERT
40.9
32.6
24.3
21.0
19.9
42.1
14.5
23.8
17.3
17.4
26.7
25.5
mBERT + Adapters
36.7
21.3
18.3
17.2
38.2
12.9
20.9
15.1
15.1
23.9
22.6
28.4
21.0
18.0
16.9
38.0
12.7
14.8
15.0
23.7
22.3
17

---

Under review as a conference paper at ICLR 2021
J HYPER-PARAMETER SETTING
The hyper-parameters used the training are shown in Table 14
Table 14: Hyper-parameter setting for training
hyper-parameters
value
encoder-layer
encoder-units
2048
decoder-layers
decoder-units
3072
mucoder-attention-oe
256
decooder-attention--im
7.56
attention-heade headss label-smoothing batch-size
0.1
32 (for random sampling) (for balanced sampling)
maximum -entithh---nnngiinn maximum-length-out optimizer
512
noam
warmup-step
25000
accum-grad
2 (for random sampling)
11 (for balanced sampling)
grad-clip
5
18


Reviews:
[{"review_id": "um3uIS-vK0c", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims to improve multilingual speech recognition on common voice, which contains 18 languages, some of which have little data (which the authors here refer to as the long-tail languages I believe). The problem of multilingual ASR is both a practical one as well as a challenging one from the perspective of multitask learning and fairness, and I'm happy to see work in this area. \n\nThe paper proposes 3 techniques that together result in a modest improvement over the baseline on common voice. The 3 include logit re-balancing based on class priors, fusion of a BERT-based language model, and the use of a common and langauge-specific adapter layer in parallel. All of these techniques have been previously explored in slightly different forms for speech problems. They have not been combined in this way before though. To my knowledge, the logit adjustment has not been applied to the long-tail problem in speech recognition.\n\nPros\n- Addresses an important problem in ASR\n- Overall, A2 improves over the baseline of balanced sampling by an average of 1% absolute CER, or a relative improvement of 6%. That is a moderate improvement but worthwhile enough to report.\n- Introduces class-based logit adjustment to the problem of long tail\n- Introduces minor tweaks that lead to improvement, and presents ablation study\n\nCons\n- In large scale models such as this, it is improtant to report the computation requirements of the model in addition the to quality improvements, as often the quality grows with model size. Here there are no comparisons of parameter count here\n- Besides the ablation studies, there's not much to be learned on how the changes (dual adapter, logit adjustment, or the way mbert is fused) helped the quality. It would be nice to report a few failed versions that the authors tried to learn more about what works and what doesn't.\n- Overall the changes do not improve significantly over the baseline. Also there should be more competing baselines to consider, other than the adapter layers of Kannan et al. There's the multi-headed decoder approach of Pratap et al. or the language ID injection approach of Li et al. \"Multi-Dialect Speech Recognition with a Single Sequence-to-Sequence Model\". \n- It's quite unclear what the long tail refers to in this paper. Does it refer to the languages that have little data? Or does it refer to words that are rare or often misclassified? Most of the paper leads me to believe in the former, but Figures 5 and 6 in the appendix lead me to believe in the latter since the histograms are so dense.\n- There's a lack of specific examples that illustrate how the incorporation of the various techniques in this paper show an improvement in the transcription. Showing specific transcriptions would be convincing in terms how showing the wins from these techniques...\n\nOther comments:\n\nWhat is meant by the fourth bullet point in the contributions? Is there a new dataset? I do not understand the contribution \n\nThe use of previous tokens as input, i.e. not using teacher forcing, during the later stages of training (Eq. 10) is unconventional. It would be more convincing if the author discussed this a little more, including why it improves quality.\n\nIt's unclear how x_{CTC} is defined in fig 1. Is it the output of the encoder?\n\nLikewise it's unclear how the function f is defined in fig 1. Is it the same function and weights (assuming a linear transformation from the previous layer) for f(x_CTC) and f(y'_ATTN, h_enc)?\n\nFig 7 and comments to it should be moved to the main paper. It is essential for understanding of how mbert is integrated into the decoder as that is a big part of the contribution.\n\nThe grammar throughout the document is occasionally off which distracts from the content. Needs polish.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "WYGcuNmQyJv", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies multilingual ASR with a focus on the long tail problem. A new method using dual adapters is proposed. Although there are several ingredients of the method, their effectiveness are all verified in detailed ablation studies. Therefore, I believe the results shown in this paper are valuable for future work.\n\nPro:\n1. The structure of dual adapters is novel.\n2. To the best of my knowledge, this is the first work to verify the effectiveness of pretrained models in multilingual ASR.\n3. The paper contains detailed experiments.\n\nCon:\n1. The framework combines many techniques together and it is hard to tell if any one of those is the 'silver bullet'.\n2. Some design/hyperparameter choices are rather magical.\n\nQuestions:\n1. Why did you choose to use distill-mBERT over other alternatives (mBERT, XLM etc.)? Would you expect more gain if using a larger model such as XLM-R?\n2. Recent work [1] shows negative interference can impact low-resource languages in multilingual models. However, it seems like the opposite is true here: multilingual models can improve even high-resource languages (e.g. IT). Do you have any idea why?\n\n\n[1] On negative interference in multilingual models: findings and a meta-learning treatment. Wang et al., EMNLP 2020.", "extra_scores": {}, "extra_text": {}}, {"review_id": "LX2AQdUe6io", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an Adapt-and-Adjust framework to address the long-tail problem in multilingual ASR, which assembles three techniques: 1) leveraged a pre-trained model mBERT to initialize the decoder, 2)  language-specific and language-agnostic adaptors, 3) class imbalance adjustments.  Experiments on a multilingual ASR with 11 languages demonstrate the proposed method can achieve accuracy improvements.\n\nOverall this paper is clearly written and easy to follow. Each technique is presented with details and evaluated with corresponding ablation studies. It is a good paper in terms of application, experiments and systematic engineering efforts.  However, I have several concerns on the overall novelty and technical contributions: \n1) The three techniques alone are not novel enough, and each is proposed by previous works. E.g., initialized with a pre-train language model, class imbalance adjustment, and language-specific adaptors which are similar to mixture of language experts. \n2) The proposed method can hardly be called as a framework since it has not demonstrated its necessity and applicability for each component. In another view, it is more like an assemble of different improvement tricks without much centralized logic towards a dedicated and focused problem. \n3) The effectiveness of a component (mBERT) need to depend on other components, otherwise it does not work. This makes the proposed method not generalizable. Why mBERT is only effective when coupled with others? Is it necessary? Is the improvement by chance but not universal? \n4) Initializing from mBERT (trained with MLM) but adjusting to autoregressive generation would harm the model capability of mBERT. Why not initialize from GPT model or more appropriate from sequence to sequence pre-trained models with an cross-attention module such as MASS or BART? This would be more effectiveness than simply using mBERT.\n\n\n\n\n ", "extra_scores": {}, "extra_text": {}}, {"review_id": "Jo24Y4cz6eX", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses multi-lingual speech synthesis, where one ASR model is responsible for recognizing speech in multiple languages.  In this example the authors look at 11 languages with between 80 and 4 hours of training data.  The \"long-tail problem\" (which isn't clearly stated) that this work is addressing is that the discrepancy in available training data leads to a discrepancy in performance.  The paper sets out two goals 1) \"to improve the overall performance of multilingual ASR tasks\" and 2) (implicitly) to flatten the distribution across languages.\n\nA major challenge in multilingual (or multidomain or multitask) modeling like this is that improvements to the tail often come with degradation at the head.  This work demonstrates this phenomenon clearly.  On the largest languages, English performance degrades from 13.3 to 22.0 and French from 11.5 to 17.7, while on the smallest languages, Kyrghyz improves from 30.0 to 12.1 and Swedish improves from 56.1 to 21.3.  While the language average performance improves from 22.3 (monolingual) to 16.0 (proposed multilingual) it is not at all obvious that there is an application setting where this is clearly preferable.  One way to mitigate this is to pose the problem not as solving universal, multilingual speech recognition, but rather improving performance specifically on tail languages through training on higher resource languages.  If the authors were to focus on improving performance on the 8 languages with 20h or less training data, while including English (en) French (fr) and Spanish (es), but not actually caring whether the high resource languages are improved by multilingual modeling, the results here would be much more compelling.  As written the story is somewhat muddled: On average (where average is taken over language, rather than, say expected usage or the system, or population, etc.) performance improves, but the improvement to lower resource languages comes at the cost of higher resource languages.  Also A2 the proposed system on average does better than standard multilingual training, but only on the 9 lowest resource languages, on English and French A2 actually exacerbates this problem with these higher resource languages showing even larger regressions from monolingual modeling.\n\nImplicit in this approach and task is a desire for the distribution of performance across languages to be more consistent.  I would recommend making this explicit and providing some measure of variance as well as average across languages.  This could be standard deviation (if there is a belief that the performance is normally distributed) or an entropy measure.  But it would provide another dimension over which to optimize when understanding tail performance.\n\nI believe there is a typo or error in Equation 6.  First, there are mismatched subscripts for \\pi_y and c_i.  I believe this should be \\pi_i or c_y.  Second consider a distribution with three classes and label counts of c = [1, 0, 0], so C=1, n_0 = 2 and N = 3.  Equation 3 would result in \\pi = [1/1 - 1/(2*1), 1/1, 1/1] = [1/2, 1, 1] which is not a valid distribution. \n\nMinor comment: Figure 7 is mentioned in Section 2.3 but is only included in the Appendix.  It would be clearer to either describe Figure 7 where it is first mentioned, or present this information in Section 2.3 as forward referring to Appendix material.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "A50BpS-ipA0", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes three additions to improve a monolithic multilingual end-to-end ASR system. The problem of training a monolithic multilingual ASR system is that using data from multiple languages does not necessary improve over individual monolingual systems. The three additions are a large multilingual language model, the use of language adapters, and smoothing on the token probabilities. Mixing the three additions in a specific way helps improve the average word error rates.\n\nThere are two major problems in the paper. One is the imprecise use of words, and the other is the disconnect between the additions and the problems they try to solve. Details are as follows.\n\nThe paper contains a lot of imprecise use of words. The term \"long tail\" is used throughout the paper, but it is never clearly defined. The long tail of a distribution refers to a significant total amount of probability mass spread on a large support. In the context of this paper, when the paper talks about the long-tail problem, what distribution are we talking about? Is it a distribution that captures how likely a phone or a word piece is used in all of the world's languages?\n\nWhile the long-tail problem is not properly defined, the class imbalance problem more or less is. There is still a certain amount of ambiguity. For example, what are the classes? Are the classes languages, phones, or word pieces?\n\nGiven that the long-tail problem is not defined, it is hard to see why the proposed additions solve the problem. I can understand using a larger language model would help the final performance, but how does this solve the long-tail problem and the class imbalanced problem? The same applies to language adapters. The smoothing technique does have a effect on generalizing to low frequency or even unseen tokens, but the paper does not mention the connection or cite the proper papers.\n\nThe paper also ignores the relationships among languages. For example, it is obvious that none of the word pieces in Mandarin are shared with the other languages. It is also the only tonal language. As another example, Tatar is Turkic but uses the Cyrillic script; Turkish is also Turkic but it uses the Latin alphabet; Russian is not Turkic but uses the Cyrillic script. These relationships are important in interpreting the results when training multiple languages together.\n\nHere are a list of detailed comments.\n\n> x \\in R^{T,F}\n\nT,F is a rather unconventional notation. I would suggest T \\times F.\n\n> KL(y_{ATTN} || y)\n\nAre the y's labels? This is also an unconventional (if not wrong) notation. It should be the the KL of distributions, not labels. Later on, for example in equation (3), y is used as labels.\n\n> equation (3)\n\n\\mathcal{Y} is undefined.\n\n> Figure 7 depicts ...\n\nFigure 7 is in the appendix. The main content without the appendix should be as self-contained as possible.\n\n> Let t denote the current time step.\n\nThis is confusing. It's actually not the time in the actual speech, but the t-th token.\n\n> A natural adjustment is to scale the raw logits ...\n\nThe term logit is misused. Please look it up, stop misusing it, and define the symbols properly.\n\n> equation (6)\n\nThe symbol * should really be \\times.\n\n> equation (9)\n\nIt is confusing to denote the probability as y_t^{adj}. Again, because the bold face y is used as a sequence of labels else where, such as equation (11).\n\n> ... and 2 times gradient accumulation in a single GPU ...\n\nWhat does this mean exactly? Please elaborate.\n\n> This is due to the human languages share some common sub-phonetic articulatory features (Wang & Sim, 2014) ...\n\n1. This sentence is ungrammatical. 2. This is a well-known fact, and the citation cannot be this recent. 3. No evidence in this paper is shown that this is the actual cause of the improvement. Please state it clearly if this is only a speculation.\n\n> ... even MT models improve the performance of the low-resource languages significantly.\n\nThis is not exactly true. For example, the performance on Mandarin actually degrades quite significantly.\n\n> ... compared to the MT, the tail classes ... However, the head classes suffer ...\n\nAre the terms tail classes and head classes defined?\n\n> ... and possibly model overfitting to the tail classes.\n\nThis is easy to check. What's the performance on the training set?\n\n> The gains ... of the head languages, although tail languages ...\n\nAgain, what are head and tail languages?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: As one of the reviewers' comment, the paper presents "a mixed of tricks" for the multilingual speech recognition, which includes 1) the use of a pretrained mBERT, 2) dual-adapter and 3) prior adjusting. 
First, the relative gains of the pretrained mBERT is marginal (Section 3.3.1). Secondly, using 1) on top of 2) is unnecessary. 
These confuses the reader about what the conclusion of the paper is. 
It would be better if choosing one aspect of the problem and investigate it deeper. 

The decision is mainly because of the lack of novelty and clarity. 
Decision: Reject