Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper studies multilingual ASR with a focus on the long tail problem. A new method using dual adapters is proposed. Although there are several ingredients of the method, their effectiveness are all verified in detailed ablation studies. Therefore, I believe the results shown in this paper are valuable for future work.

Pro:
1. The structure of dual adapters is novel.
2. To the best of my knowledge, this is the first work to verify the effectiveness of pretrained models in multilingual ASR.
3. The paper contains detailed experiments.

Con:
1. The framework combines many techniques together and it is hard to tell if any one of those is the 'silver bullet'.
2. Some design/hyperparameter choices are rather magical.

Questions:
1. Why did you choose to use distill-mBERT over other alternatives (mBERT, XLM etc.)? Would you expect more gain if using a larger model such as XLM-R?
2. Recent work [1] shows negative interference can impact low-resource languages in multilingual models. However, it seems like the opposite is true here: multilingual models can improve even high-resource languages (e.g. IT). Do you have any idea why?


[1] On negative interference in multilingual models: findings and a meta-learning treatment. Wang et al., EMNLP 2020.
review_title: Simple method with comprehensive experiments

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper aims to improve multilingual speech recognition on common voice, which contains 18 languages, some of which have little data (which the authors here refer to as the long-tail languages I believe). The problem of multilingual ASR is both a practical one as well as a challenging one from the perspective of multitask learning and fairness, and I'm happy to see work in this area. 

The paper proposes 3 techniques that together result in a modest improvement over the baseline on common voice. The 3 include logit re-balancing based on class priors, fusion of a BERT-based language model, and the use of a common and langauge-specific adapter layer in parallel. All of these techniques have been previously explored in slightly different forms for speech problems. They have not been combined in this way before though. To my knowledge, the logit adjustment has not been applied to the long-tail problem in speech recognition.

Pros
- Addresses an important problem in ASR
- Overall, A2 improves over the baseline of balanced sampling by an average of 1% absolute CER, or a relative improvement of 6%. That is a moderate improvement but worthwhile enough to report.
- Introduces class-based logit adjustment to the problem of long tail
- Introduces minor tweaks that lead to improvement, and presents ablation study

Cons
- In large scale models such as this, it is improtant to report the computation requirements of the model in addition the to quality improvements, as often the quality grows with model size. Here there are no comparisons of parameter count here
- Besides the ablation studies, there's not much to be learned on how the changes (dual adapter, logit adjustment, or the way mbert is fused) helped the quality. It would be nice to report a few failed versions that the authors tried to learn more about what works and what doesn't.
- Overall the changes do not improve significantly over the baseline. Also there should be more competing baselines to consider, other than the adapter layers of Kannan et al. There's the multi-headed decoder approach of Pratap et al. or the language ID injection approach of Li et al. "Multi-Dialect Speech Recognition with a Single Sequence-to-Sequence Model". 
- It's quite unclear what the long tail refers to in this paper. Does it refer to the languages that have little data? Or does it refer to words that are rare or often misclassified? Most of the paper leads me to believe in the former, but Figures 5 and 6 in the appendix lead me to believe in the latter since the histograms are so dense.
- There's a lack of specific examples that illustrate how the incorporation of the various techniques in this paper show an improvement in the transcription. Showing specific transcriptions would be convincing in terms how showing the wins from these techniques...

Other comments:

What is meant by the fourth bullet point in the contributions? Is there a new dataset? I do not understand the contribution 

The use of previous tokens as input, i.e. not using teacher forcing, during the later stages of training (Eq. 10) is unconventional. It would be more convincing if the author discussed this a little more, including why it improves quality.

It's unclear how x_{CTC} is defined in fig 1. Is it the output of the encoder?

Likewise it's unclear how the function f is defined in fig 1. Is it the same function and weights (assuming a linear transformation from the previous layer) for f(x_CTC) and f(y'_ATTN, h_enc)?

Fig 7 and comments to it should be moved to the main paper. It is essential for understanding of how mbert is integrated into the decoder as that is a big part of the contribution.

The grammar throughout the document is occasionally off which distracts from the content. Needs polish.

review_title: A mix of tricks for an important problem

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper addresses multi-lingual speech synthesis, where one ASR model is responsible for recognizing speech in multiple languages.  In this example the authors look at 11 languages with between 80 and 4 hours of training data.  The "long-tail problem" (which isn't clearly stated) that this work is addressing is that the discrepancy in available training data leads to a discrepancy in performance.  The paper sets out two goals 1) "to improve the overall performance of multilingual ASR tasks" and 2) (implicitly) to flatten the distribution across languages.

A major challenge in multilingual (or multidomain or multitask) modeling like this is that improvements to the tail often come with degradation at the head.  This work demonstrates this phenomenon clearly.  On the largest languages, English performance degrades from 13.3 to 22.0 and French from 11.5 to 17.7, while on the smallest languages, Kyrghyz improves from 30.0 to 12.1 and Swedish improves from 56.1 to 21.3.  While the language average performance improves from 22.3 (monolingual) to 16.0 (proposed multilingual) it is not at all obvious that there is an application setting where this is clearly preferable.  One way to mitigate this is to pose the problem not as solving universal, multilingual speech recognition, but rather improving performance specifically on tail languages through training on higher resource languages.  If the authors were to focus on improving performance on the 8 languages with 20h or less training data, while including English (en) French (fr) and Spanish (es), but not actually caring whether the high resource languages are improved by multilingual modeling, the results here would be much more compelling.  As written the story is somewhat muddled: On average (where average is taken over language, rather than, say expected usage or the system, or population, etc.) performance improves, but the improvement to lower resource languages comes at the cost of higher resource languages.  Also A2 the proposed system on average does better than standard multilingual training, but only on the 9 lowest resource languages, on English and French A2 actually exacerbates this problem with these higher resource languages showing even larger regressions from monolingual modeling.

Implicit in this approach and task is a desire for the distribution of performance across languages to be more consistent.  I would recommend making this explicit and providing some measure of variance as well as average across languages.  This could be standard deviation (if there is a belief that the performance is normally distributed) or an entropy measure.  But it would provide another dimension over which to optimize when understanding tail performance.

I believe there is a typo or error in Equation 6.  First, there are mismatched subscripts for \pi_y and c_i.  I believe this should be \pi_i or c_y.  Second consider a distribution with three classes and label counts of c = [1, 0, 0], so C=1, n_0 = 2 and N = 3.  Equation 3 would result in \pi = [1/1 - 1/(2*1), 1/1, 1/1] = [1/2, 1, 1] which is not a valid distribution. 

Minor comment: Figure 7 is mentioned in Section 2.3 but is only included in the Appendix.  It would be clearer to either describe Figure 7 where it is first mentioned, or present this information in Section 2.3 as forward referring to Appendix material.


review_title: Reasonable approach but unconvincing

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes three additions to improve a monolithic multilingual end-to-end ASR system. The problem of training a monolithic multilingual ASR system is that using data from multiple languages does not necessary improve over individual monolingual systems. The three additions are a large multilingual language model, the use of language adapters, and smoothing on the token probabilities. Mixing the three additions in a specific way helps improve the average word error rates.

There are two major problems in the paper. One is the imprecise use of words, and the other is the disconnect between the additions and the problems they try to solve. Details are as follows.

The paper contains a lot of imprecise use of words. The term "long tail" is used throughout the paper, but it is never clearly defined. The long tail of a distribution refers to a significant total amount of probability mass spread on a large support. In the context of this paper, when the paper talks about the long-tail problem, what distribution are we talking about? Is it a distribution that captures how likely a phone or a word piece is used in all of the world's languages?

While the long-tail problem is not properly defined, the class imbalance problem more or less is. There is still a certain amount of ambiguity. For example, what are the classes? Are the classes languages, phones, or word pieces?

Given that the long-tail problem is not defined, it is hard to see why the proposed additions solve the problem. I can understand using a larger language model would help the final performance, but how does this solve the long-tail problem and the class imbalanced problem? The same applies to language adapters. The smoothing technique does have a effect on generalizing to low frequency or even unseen tokens, but the paper does not mention the connection or cite the proper papers.

The paper also ignores the relationships among languages. For example, it is obvious that none of the word pieces in Mandarin are shared with the other languages. It is also the only tonal language. As another example, Tatar is Turkic but uses the Cyrillic script; Turkish is also Turkic but it uses the Latin alphabet; Russian is not Turkic but uses the Cyrillic script. These relationships are important in interpreting the results when training multiple languages together.

Here are a list of detailed comments.

> x \in R^{T,F}

T,F is a rather unconventional notation. I would suggest T \times F.

> KL(y_{ATTN} || y)

Are the y's labels? This is also an unconventional (if not wrong) notation. It should be the the KL of distributions, not labels. Later on, for example in equation (3), y is used as labels.

> equation (3)

\mathcal{Y} is undefined.

> Figure 7 depicts ...

Figure 7 is in the appendix. The main content without the appendix should be as self-contained as possible.

> Let t denote the current time step.

This is confusing. It's actually not the time in the actual speech, but the t-th token.

> A natural adjustment is to scale the raw logits ...

The term logit is misused. Please look it up, stop misusing it, and define the symbols properly.

> equation (6)

The symbol * should really be \times.

> equation (9)

It is confusing to denote the probability as y_t^{adj}. Again, because the bold face y is used as a sequence of labels else where, such as equation (11).

> ... and 2 times gradient accumulation in a single GPU ...

What does this mean exactly? Please elaborate.

> This is due to the human languages share some common sub-phonetic articulatory features (Wang & Sim, 2014) ...

1. This sentence is ungrammatical. 2. This is a well-known fact, and the citation cannot be this recent. 3. No evidence in this paper is shown that this is the actual cause of the improvement. Please state it clearly if this is only a speculation.

> ... even MT models improve the performance of the low-resource languages significantly.

This is not exactly true. For example, the performance on Mandarin actually degrades quite significantly.

> ... compared to the MT, the tail classes ... However, the head classes suffer ...

Are the terms tail classes and head classes defined?

> ... and possibly model overfitting to the tail classes.

This is easy to check. What's the performance on the training set?

> The gains ... of the head languages, although tail languages ...

Again, what are head and tail languages?
review_title: A large disconnect between the proposed additions and the problems the paper tries to solve

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes an Adapt-and-Adjust framework to address the long-tail problem in multilingual ASR, which assembles three techniques: 1) leveraged a pre-trained model mBERT to initialize the decoder, 2)  language-specific and language-agnostic adaptors, 3) class imbalance adjustments.  Experiments on a multilingual ASR with 11 languages demonstrate the proposed method can achieve accuracy improvements.

Overall this paper is clearly written and easy to follow. Each technique is presented with details and evaluated with corresponding ablation studies. It is a good paper in terms of application, experiments and systematic engineering efforts.  However, I have several concerns on the overall novelty and technical contributions: 
1) The three techniques alone are not novel enough, and each is proposed by previous works. E.g., initialized with a pre-train language model, class imbalance adjustment, and language-specific adaptors which are similar to mixture of language experts. 
2) The proposed method can hardly be called as a framework since it has not demonstrated its necessity and applicability for each component. In another view, it is more like an assemble of different improvement tricks without much centralized logic towards a dedicated and focused problem. 
3) The effectiveness of a component (mBERT) need to depend on other components, otherwise it does not work. This makes the proposed method not generalizable. Why mBERT is only effective when coupled with others? Is it necessary? Is the improvement by chance but not universal? 
4) Initializing from mBERT (trained with MLM) but adjusting to autoregressive generation would harm the model capability of mBERT. Why not initialize from GPT model or more appropriate from sequence to sequence pre-trained models with an cross-attention module such as MASS or BART? This would be more effectiveness than simply using mBERT.




 
review_title: Review comments for paper 3487

Meta-review: As one of the reviewers' comment, the paper presents "a mixed of tricks" for the multilingual speech recognition, which includes 1) the use of a pretrained mBERT, 2) dual-adapter and 3) prior adjusting. 
First, the relative gains of the pretrained mBERT is marginal (Section 3.3.1). Secondly, using 1) on top of 2) is unnecessary. 
These confuses the reader about what the conclusion of the paper is. 
It would be better if choosing one aspect of the problem and investigate it deeper. 

The decision is mainly because of the lack of novelty and clarity. 
Decision: Reject