Unsupervised Cross-lingual Representation Learning for Speech Recognition
Authors: Alexis Conneau
Alexei Baevski
Ronan Collobert
Abdelrahman Mohamed
Michael Auli
Venue:  (2021)
Keywords: Deep learning
speech processing
multilingual modeling
cross-lingual

Abstract: This paper presents XLSR which learns cross-lingual speech representations by pretraining a single model from the raw waveform of speech in multiple languages. We build on wav2vec 2.0 which is trained by solving a contrastive task over masked latent speech representations and jointly learns a quantization of the latents shared across languages. The resulting model is fine-tuned on labeled data and experiments show that cross-lingual pretraining significantly outperforms monolingual pretraining. On the CommonVoice benchmark, XLSR shows a relative phoneme error rate reduction of 72% compared to the best known results. On BABEL, our approach improves word error rate by 16% relative compared to a comparable system. Our approach enables a single multilingual speech recognition model which is competitive to strong individual models. Analysis shows that the latent discrete speech representations are shared across languages with increased sharing for related languages.

Under review as a conference paper at ICLR 2021
UNSUPERVISED CROSS-LINGUAL REPRESENTATION LEARNING FOR SPEECH RECOGNITION
Anonymous authors Paper under double-blind review
ABSTRACT
This paper presents XLSR which learns cross-lingual speech representations by pretraining single model from the raw waveform of speech in multiple languages. We build wav2vec 2.0 which is trained by solving contrastive task over masked latent speech representations and jointly learns a quantization of the latents shared across languages. The resulting model is fine-tuned on labeled data and experi- ments show that tross-lingual pretraining significantly outperforms monolingual pretraining. On the Common Voice benchmark, XI.SR shows a relative phoneme error rate reduction of 72% compared to the best known results. On BABEL, our approach improves word error rate by 16% relative compared to a comparable system. Our approach enables single multilingual speech recognition model which is competitive to strong individual models. Analysis shows that the latent discrete speech representations are shared across languages with increased sharing for related languages.
INTRODUCTION
Cross-lingual learning aims to build models which leverage data from other languages improve
performance. This has been long standing interest in the speech community (Byrne et al. 2000
Le & Besacier 2009 Ghoshal et al., 2013 Huang et al. 2013 Gales et
2017] Cho al.
2018
Seki et al.] 2018) which includes systems able to transcribe multiple languages (Burget et al.
2010
Bourlard et al.][201 Heigold et al. 2013 Toshniwal et al.) 2018 Kannan et al.) 2019). However, the
vast majority of work in speech processing has focused on supervised cross-lingual training which
requires labeled data in multiple languages. Transcribed speech is often much searcer than unlabeled
speech and requires non-trivial human annotation.
Unsupervised representation learning, or pretraining, does not require labeled data and has received lot of recent attention in computer vision (Tian al. 2019 He et al. 2019: Chen et al. 2020 after much success in natural language processing (Peters et al. 2018 Devlin et al.) 2018). For the latter, cross-lingual pretraining has been shown to be very effective, particularly, for low resource languages (Lample & Conneau 2019: Conneau et al. 2019). In speech processing, most work in this area has focused monolingual unsupervised representation learning (van den Oord al. 2018
Chung & Glass 2018; Schneider et al.) 2019 Chung et al.) 2019 Bacyski et al. [020bb] Harwath al. 2020 Jiang et al. 2019 Tjandra et al.) 2019 Eloff al. 2019 Baevski al. 2020a).
In this paper, we focus on the cross-lingual setting by learning representations on unlabeled data that generalize across languages. We build on the pretraining approach of Baevski al. (2020c) which jointly learns contextualized speech representations representations. The latter serves effectively train the model with contrastive loss ( 2) and the discrete speech representations are shared across languages (Figure []. Different to recent work on unsupervised cross-lingual pretraining, we fine-tune the Transformer part of the model instead of freezing all pretrained representations (Rivière et al. (2200) or feeding them to separate downstream model (Kawakami et al. 2020). We extend the work of Rivière et al. (2020) by pretraining on multiple languages instead of just English and we experiment top of a stronger baseline.
well as a discrete vocabulary of latent speech
We evaluate XLSR on 14 languages of the BABEL benchmark (Gales et al. 2014) which is conver- sational telephone data and ten languages of Common Voice (Ardila et al. 2019). corpus of read speech ( 3. Multilingual pretraining outperforms monolingual pretraining in most cases, except for resource rich languages and we show that increased model capacity significantly closes the gap. We

---

Under review as a conference paper at ICLR 2021
Dentrasive losa
Multlingusl quantizzd latent speech representationss
km
Shaned Tonnloomee enooder
Transformer Mosked
Shared quarteer
2
CNN
Frenzzn
(eg.
Figure 1: The XLSR approach. shared quantization module over feature encoder representations produces multilingual quantized speech units whose embeddings are then used as targets for Transformer trained by contrastive learning. The model learns share discrete tokens across languages, creating bridges across languages. Our approach is inspired by Devlin et al.] (2018); Lample & Conneau (2019) and builds on top of wav2vec 2.0 (Baevski et al. 2020c). It requires only raw unlabeled speech audio in multiple languages.
also demonstrate that XLSR representations can be fine-tuned simultancously on multiple languages obtain a multilingual speech recognition system whose performance competitive fine-tuning a separate model on each language 4).
APPROACH
Unsupervised cross- lingual representation learning has shown great success by pretraining Transform- (Vaswani et al. 2017) with multilingual masked language models (Devlin et al.) 2018 Lample & Conneau 2019). In this work, we learn cross-lingual speech representations by extending wav2vec 2.0 (Baevski al. 2020c) to the cross-lingual setting. Our approach learns single set of quantized latent speech representations which are shared across languages. Next, we outline the architecture training (8 2.2) and adaptations for cross-lingual training.
S 2.1
2.1
ARCHITECTURE
We follow the design choices described in Baevski et al. (2020c). The model contains a convolutional feature encoder X Z map raw audio X' to latent speech representations 2ı, Zr which + C to output context representations Cı, Cr (Devlin et al. 2018 Baevski al. 2020b a}. For the purpose of training the model, feature encoder representations are discretized to qu qr with quantization module Z + self-supervised learning objective (Figure 2.2
fed to Transformer network
Q to represent the targets in the
The quantization is based on product quantization (Jegou et al. 2011 Baevski et al. 2020b) by choosing quantized representations from
codebooks with V 320 entries each. The result is concatenated to obtain q. A Gumbel softmax enables choosing discrete codebook entries in fully differentiable way (Jang et al. 2016). Each z, represents about 25ms of audio strided by 20ms, the context network architecture follows BERT (Vaswani et al. 2017 Devlin et al. 2018) except for relative positional embeddings Mohamed et al. (2019): Bacvski ct al. (020a).
2.2 TRAINING
The model is trained by solving a contrastive task over masked feature encoder outputs. For masking, we sample p = 0.065 of all time steps be starting indices and mask the subsequent M 10 time steps. The objective requires identifying the true quantized latent q for a masked time-step within
exp(sim(c..q.))
set of K 100 distractors Q sampled from other masked time steps: log Σa-q. exp(sim
where c is the output of the transformer, and sim(a, , b) denotes cosine similarity.
2

---

Under review as a conference paper at ICLR 2021
This is augmented by a codebook diversity penalty to encourage the model to use all codebook
entries (Dieleman et al.] 2018). We maximize the entropy of the averaged softmax distribution over the codebook entries for each group Pg across a batch of utterances: σV - - (P,) =
1 CV Eg= Ev=1 Py.v log Pg.c- To stabilize the feature encoder we apply an L.2 penalty over the outputs of the feature encoder.
When pretraining on L languages, we form multilingual batches (Devlin et al. 2018 Lample & Conneau 2019) by sampling speech samples from a multinomial distribution (P:)_=...... L where being the number of pretraining hours of language I, N the total number of hours, and a the upsampling factor. The parameter a controls the importance given to high-resource versus
Pt
NV
low-resource languages during pretraining.
3 EXPERIMENTAL SETUP
3.1 DATASETS
Common Voice. The Common Voice datase is a multilingual corpus of read speech comprising more than two thousand hours of speech data 38 languages (Ardila et al.) 2019). The amount of data per language ranges from three hours for Swedish ("low-resource) to 353 hours for French and 1350 hours for English ("high-resource". Following Rivière et al. (2020) we consider ten languages: Spanish (es), French (fr). Italian (it), Kyrgyz (ky). Dutch (du), Russian (ru). Swedish (sv). Turkish (tr), Tatar (tt) and Chinese (zh); as well as English (en) for pretraining. We use the November 2019 release for training models, and for fine-tuning we use the evaluation splits Rivièee et al. (2020) which include one hour labeled data for training, 20 minutes for validation and one hour for testing. This few-shot evaluation dataset consists of phoneme sequences as output and we report phone error rate (PER) similar to prior work.
BABEL. This datase is a multilingual corpus of conversational telephone speech from IARPA. which includes Asian and African languages (Gales et al., 2014). We adopt the same setup as Cho et al. (2018) and pretrain on ten languages: Bengali (bn), Cantonese (zh), Georgian (ka), Haitian (ht), Kurmanji (ku). Pashto (ps). Tamil (ta). Turkish (tr), Tokpisin (tp). Vietnamese (vi). We evaluate cross- lingual transfer on four other languages (models are not pretrained on these languages): Assamese (as), Tagalog (tl). Swahili (sw), Lao (lo). We train a multilingual model in ten languages and monolingual models in 14 languages. We use the same speech audio for pretraining and fine-tuning, and no unlabeled speech provided by BABEL. We use the dev folder of the BABEL dataset as our test set as "eval" has not been open-sources and use 10% of the training set as dev data. We report character error rate (CER). All audio is resampled to 16kHz. For comparison with Inaguma et al. (2019) only. we train 4-gram n- n-gram language models on CommonCrawl data (Heafield et al. 2013] Wenzek et al., 2019) for Assamese (140MiB of text data), Swahili (2GiB), Tamil (4.8GiB) and Lao (763MiB).
3.2 TRAINING DETAILS
Pretraining Models are implemented in fairseq (Oit et al. 2019). We evaluate two architectures with the same feature encoder (S 2. but different Transformer settings: Base with 12 blocks, model dimension 768, inner dimension (FFN) 3072 and 8 attention heads; and Large with 24 blocks, model dimension 1024, inner dimension 4096 and 16 attention heads; both use dropout 0.1. For Base, we crop 250k samples, or 15.6sec of audio, and pack up to 1. 1.4m samples on each GPU. For Large, GPU. Batches are sampled using factor
crop 320k samples and put up to 1.2m samples a E {0.5, 1}. We use 16 GPUs for small datasets (typically monolingual) and 64 GPUs for large datasets (typically multilingual), and use Adam (Kingma & Ba 2015) where the learning rate is warmed up for the first 10% of updates to a peak of le-5 (Base) or Te-3 (Large), and then linearly decayed over a total of 250k updates.
Fine-tuning. To fine-tune the model we add a classifier representing the output vocabulary of the respective downstream task on top of the model and train on the labeled data with Connectionist
httpss//voion mori /en//e// Lenguages
httpes/ /oapss///steion Log, upenn, /byyear includes 8S07 LDC20188133, LDC2018502, EDC201 LDC2017822. 7522, LDC 7508,
n 7506 L.CEL1SS 7801, IST 187666506, 1.DC301600,, 1.DC201S02, 6.122016512,, 1.DC301809, L.C20ISS1, M.C.SS......0165.05. 6.D03016510.
3

---

Under review as a conference paper at ICLR 2021
Model
#pt
#it
It
ky
n1
ru
zh
Avg
Number pretraining per language
168h
353h
29h
55h
17h
50b
793h
Number of fine-cuning hours per language
1h
1h
1h
1b
1b
1h
1h
1h
10h
Baselines from previous work
CPC! (Rivière et al.
2020
L.S
10
38.7
49.3
42.1
40.7
44.4
45.2
48.8
49.7
44.0
55.5
45.8
m-CPC" (Riviero et al.
2020
LS
10
38.0
47.1
40.5
41.2
42.5
43.7
47.5
47.3
42.0
55.0
44.5
Fer et
BBL
Our monolingual models
SR. SRR-English L.S--Monoliguul
cV
13.7
20.0
19.1
13.2
19.4
18.6
21.1
15.5
11.5
27.1
17.9
CV
6.8
10.4
10.9
29.6
37.4
11.6
63.6
44.0
21.4
31.4
26.7
Our mudllingaal models
XLSR-10 (unbalanced)
CVan
10
9.7
13.6
15.2
11.1
18.1
13.7
21.4
14.2
9.7
25.8
15.3
XLSR-10
CVan
10
1
9.4
14.2
14.1
8.4
16.1
11.0
20.7
11.2
7.6
24.0
13.66
XLSR-10 (separate vocab)
CVan
10
10
10.0
13.8
14.0
8.8
16.5
11.6
21.4
12.0
8.7
24.5
14.1
XLSR 10 (shured vocab)
CVan
10
10
9.4
13.4
13.8
8.6
16.3
11.2
21.0
11.7
8.3
24.5
13.8
Our mudiliigaal models (Large)
XLSRR-10
CVal
9.3
20.6
22.8
XLSR-10 (separate vocab)
CVall
10
10
8.1
12
11.9
7.1
13.9
9.8
21.0
10.4
7.6
22.3
12.4
XLSK-IU (shared vocab)
CVan
10
10
7.7
12.2
11.6
7.0
13.8
9.3
20.8
10.1
7.3
22.3
12.2
Table Common Voice phoneme error rate (PER). Models are pretrained on either one language (pt 1) or 10 languages (pt 10); and fine-tuned on each language (ft 1) or all languages pretraining data, LS for English LibriSpeech (100h 360h), BBL for BABEL (1070h), CV for English Common Voice (1350h), CV mo for monolingual (see number of pretraining hours per language) and CVan for multilingual (1350h). Languages can high-resource (es, fr, it) low-resource (e.g. ky, sv, tr, tt). Baseline results are from Rivière et al. 22020)
(ft 10). D indicates
Temporal Classification (CTC) loss (Graves
2006 Baevski et al. 2020a). Weights of the feature encoder are not updated at fine-tuning time. We determine the best learning rates setting in [2e-5, 6e-5] based on dev error rate. The learning rate schedule has three phases: warm up for the first 10% of updates, keep constant for 40% and then linearly decay for the remainder. For Common Voice we fine-tune for 20k updates and on BABEL for 50k updates on GPUs for the Base model and GPUs for the Large model.
3.3 PRETRAINED MODELS
We use the Base architecture unless otherwise stated. For Common Voice, we pretrain an English model on 1350h, and ten monolingual models on each pretraining set. For comparison with the English model, we train Base and Large multilingual models on 1350h of data: 793h of speech audio from the 10 evaluation languages plus 557h English audio. We upsample low-resource languages with fine-tuning, we either separate or share phoneme vocabularies across languages.
0.5 and train model with 1 for comparison (unbalanced). For multilingual
For BABEL, we train monolingual model on each of the 14 languages, well as Base and Large multilingual model on a total of 650 hours of speech audio in ten languages. Since the amount of data in each language is more balanced than for Common Voice, used for pretraining and fine-tuning and we use separate character sets for multilingual fine-tuning.
1. The same speech audio is
RESULTS
In our experiments, we first show that our approach is very effective for learning generic cross-linge mooss-lngual representations an unsupervised way. Pretraining single model on multiple languages significantly outperforms the previous state of the art on Common Voice, well our own monolingual models. Second, we demonstrate the positive impact of cross-lingual transfer on low-resource languages and provide better understanding of the trade-off between high-resource and low-resource languages. Third, by fine-tuning multilingual model on many languages at once, we show that we can obtain a single model for all languages with strong performance. Finally, analyse the impact of language similarity on cross lingual transfer, and show that, to some extent, our multilingual pretrained model implicitly learns cluster related languages.
4

---

Under review as a conference paper at ICLR 2021
4.1 EFFECTIVENESS OF UNSUPERVISED CROSS-LINGUAL REPRESENTATION LEARNING
In what follows, we compare XLSR to several baselines and show that unsupervised cross-lingual representation learning is very effective. We provide comprehensive analysis of the impact of different pretraining methods on automatic speech recognition in Table 2 and
4.1.1 MULTILINGUAL OUTPERFORMS MONOLINGUAL PRETRAINING AND PRIOR ART
We first compare monolingual (Base) ine-tuned individually on each language (ft=1). On Common Voice, XLSR- 10 obtains 13.6 PER on average (Avg), relative PER reduction of 49% compared to XLSR-Monolinge On BABEL, XLSR- 10 improves over KLSR-MMonolingua by 18% relative CER (Table ( and by more over supervised training (Training from scratch) Pretraining on multiple languages results in cross-lingual transfer and better speech representations.
to multilingual (XLSR-10) pretrained models
(Table [T]
Compared to prior work, XLSR-10 Large reduces PER by 72% relative to m-CPC (Rivière et al. 2020 on Common Voice (Table []. For BABEL, we compare to the supervised multilingual model
of Cho et al. (2018) whose setup we adopted: XLSR-10 10 Large reduces CER by 38% relative to multi- Multi-BLSTMI BLSTMP+VGGGGG (Table 2). The other most comparable work is Inaguma et al. (2019) and
Table 4 shows relative word error reduction of 16% compared to their monolingual baseline (BLSTM-HMM which outperforms their own supervised multilingual model (Multi-10).
4.1.2 MULTILINGUAL PRETRAINING OUTPERFORMS ENGLISH-ONLY TRAINING
To isolate the impact of multilingual training versus simply training on more data, we pretrain an English-only Common model XL-SR-Englis) on the same amount of data as the multilingual model (1350h) and compare the two. Table I shows that on average, improves over the monolingual models (average PER of 26.7 vs. 17.9 PER) but multilingual pretraining performs even better at 13.6 PER, a 24% relative PER reduction over XLSR-Englishe This shows that adding more training data is not the only reason for the improved accuracy: the similarity between the languages used in pretraining and fine-tuning also plays an important role.
significantly
4.1.3 LEARNED REPRESENTATION TRANSFER WELL TO UNSEEN LANGUAGES
To better assess the cross-lingual transfer of the learned representations, we evaluate the XLSRR10 BABEL model on four languages not seen during pretraining. We fine-tune this model on each language, and compare it to monolingual models pretrained specifically these languages. Table shows that a multilingual model not pretrained on any data from the four languages, still outperforms ELSR-Monolings
reducing average CER from 29 to 22.8 which compares to results from previous work of 36.8 CER (Cho et al. 2018). This further suggests that the learned representations capture generic features of the speech signal which transfer many languages.
4.2 UNDERSTANDING
TRANSFER LEARNING
In this section, we examine several properties of unsupervised cross-lingual representation learning for speech recognition. We show that it is particularly effective on low-resource languages, then describe the transfer-intefferenne trade-off which benefits low resource languages but hurts high resource languages. Finally, we show that adding capacity is important for multilingual pretraining.
4.2.1 CROSS-LINGUAL TRANSFER LEARNING IMPROVES SWW-RESOURCE LANGUAGE UNDERSTANDING
Unsupervised cross- cross-lingual representation learning and cross- lingull transfer are particularly effective on low-resource languages. On Common Voice, the separation between high-resource and low- resource languages is more salient than for BABEL. We distinguish between low-resource and high-resource based on the amount of available unlabeled speech data. For example, French and Spanish have 353h and 168h and are thus high-resource, while Swedish and Turkish have 3h and 11h and are low-resource. Monolingual models perform poorly on low-resource languages but this is
We did this setting extensively and will further in the version of the paper.
5

---

Under review as a conference paper at ICLR 2021
Model
#pt
#ft
bn
zhh
kn
ht
ku
ps
ta
tr
vi
Avg
Number of pretraining hours per language Number of fine tuning hours per language
56h
130h
46h
61h
38h
71h
63h
70h
36h
79h
650h
56h
130h
46h
61h
38h
71h
63h
70h
36h
79h
650h
Baselines from previous work
Mono- BLSTMP (Cho et al.
2018
10
43.4
37.4
35.4
39.7
55.0
37.3
55.3
50.3
32.7
54.3
44.1
Muldi BLSTMP Cho al.
2018
42.9
36.3
38.9
38.5
52.1
39.0
48.5
36.4
31.7
41.0
40.5
VGG Cho 2018
34.3
36.0
34.5
34.7
37.4
37.4
Our monolisgual models
Training from scratch
47.6
42.7
45.0
45.0
58.4
43.2
55.7
44.6
45.2
43.6
47.1
XLSR- Monolingual
31.8
28.0
30.5
27.9
46.9
25.5
36.0
26.
26.8
25.2
30.5
Our multitingual models
XLSR-10
26.6
24.7
21.8
23.2
38.2
22.6
30.5
22.3
17.3
21.7
24.9
SISR (separate vocab)
10
10
29.5
29.1
25.9
26.5
4114
25.8
33.4
24.6
19.3
24.3
27.9
Our muitilingaal models (Large)
XLSR-10
25.1
23.4
19.7
21.1
36.8
21.6
28.6
19.8
16.1
19.9
23.2
XLSR-10 (sepurate vocab)
25.0
20.7
22.0
37.2
21.2
28.9
15.9
20.7
23.7
Table 2: BABEL results using character error rate (CER) in-pretraining languages. Baseline results are from Cho et al. (2018) and use the same amount of data as our multilingual models.
where cross-lingual transfer is most effective: XLSR-10 reduces PER over XLSR-Monolingual by relative 67% on Swedish, 72% on Turkish, 72% on Kyrgyz, and 64% on Tatar.
On BABEL, the amount of monolingual data ranges between 30 hours for Swahili and 130 hours for Cantonese, with mean of 65h per language. The results (Table 2 and B show that the multilingual model outperforms the monolingual model on all languages, the biggest gains are obtained on the four lowest-resource languages: Georgian (ka), Kurmanji (ku). Tokpisin (tp) and Swahili (sw).
4.2.2 THE TRANSFER-INTERFEENE INTERFERENCE TRADE-OFF: HIGH-RESOURCE VS. LOW- NOW-RESOURCE
Per language results on Common Voice (Table show transfer-interference trade-off AAriaazhagan al. 2019; for low-resource languages (e.g. ky, nl, sv, tr, tt), multilingual models outperform monolinual models because of positive transfer, however multilingual models perform worse on high-resource languages (es, fr, it), due interference. Data from multiple languages enables better speech representations that transfer to low-resource languages but the model also needs to share its capacity across languages which degrades performance on high-resource languages.
For given model capacity, the language sampling parameter (see $ 2) controls this trade-off. Table | shows that training according to the true language distribution, XLSR-10 (unbalanced) using 1, performs less well than KLSR-10, where more capacity is allocated to low-resource languages via & = 0.5. The sole exception being French, the language with the most data. On average the unbalanced model obtains 15.3 PER while the balanced model obtains 13.6.
4.2.3 INCREASING CAPACITY FOR A MULTILINGUAL PRETRAINED MODEL
The interference problem can be alleviated by adding vazhagan et al. 2019 Conneau et al.) 2019): the gap between multilingual models and monolingual models for high-reson righuresouree languages can be reduced by increasing model capacity. In this work, only study the impact of adding more capacity to the multilingual model, by training an XLSR-10 Large model. On Common Voice, the Large model reduces PER by relative 9.6% compared to Base, reducing average PER from 13.6 to 12.3. There are no gains on very low-resource languages like Swedish but significant gains Spanish, French and Italian. On BABEL, average CER is reduced by relative 6.8%. This shows that the multilingual model benefits from more capacity overall, and in particular for high-resource languages.
capacity to the multilingual model Ari
4.3 SUPERVISED MULTILINGUAL FINE- TUNING: ONE MODEL FOR ALL LANGUAGES
When fine-tune the pretrained model each language individually, then end up with different model for each language. On the other hand, multilingual speech recognition aims to build a single model for all languages that performs well better than individual monolingual models.

---

Under review as a conference paper at ICLR 2021
Mode
Apt
An
20
2W
Avg.
Model
App
40
AVg
Narbeer lin-laning houp
Number pustraining boins
55h
53b
766
76h
30h
30h
59h
39h
22th
2208
Nander of pretnining hous
Namthrr fivettiing hours
556
780
306.
306.
50h
94h
220h
Boveliees foor pervisus wost
Bureitey fro perrloy work
Monolingral I Cua 2013
10
1
45.6
63.1
33.1
42.1
41.0
BLSTM MMME Doseine BRNN Tagni
Stag- remnitng (eboerr Viate
40.6
35.5
Our appeace (e 1.5)
Trinng from serncc
50.2
43.7
40.5
43.5
64.1
XLSB- Menelnneell
34.5
25.4
26.5
29.1
29.0
Oar opprose 19 KeaLM)
Dur
NUSR-100
10 46. 32.2
XL5R-10 XI (laggg)
10
21.8
21.0
Table 4: BABEL results on out-of-pretraining languages using word error rate (WER). XISR- 10 reduces word error rate by 16.5% com- pared to previously published results on four of the BABEL languages. We report WER with and without 4-gram KenLM language models.
Table 3: BABEL results on out-of-pretraining languages (CER). XISSR-1 provides strong representations for languages not seen during pretraining, outperforming monolingual models pretrained specifically on these languages.
Next, we investigate fine-tuning a single model on the labeled data of all languages (#ft=10) obtain single multilingual model instead of fine-tuning each language separately (#ft=1). Training batches are constructed by sampling audio samples from multiple languages (without upsampling).
For Common Voice we consider two settings since we use phonemes: separate phoneme vocabularies per languages as well as sharing phonemes across languages. A shared vocabulary reduces the number of modeled phonemes from 474 to 182 compared to separate vocabularies. shows that the Base model with monolingual fine-tuning of XLSR- obtains 13.6 average PER which compares to 14.1 PER and 13.8 PER for separate and shared vocabulary multilingual fine-tuning respectively. When increasing model capacity (Large), multilingual fine-tuning is competitive to monolingual fine-tuning: 12.3 average PER (ft=1) 12.4 and 12.2 average PER (ft=N) for separate and shared vocabularies. Multilingual fine-tuning of the Large model with a shared vocabulary achieves the best overall performance on Common Voice.
BABEL provides significantly more labeled data (650h for all languages) compared to Common Voice (10h for all languages). Performance BABEL with multilingual fine-tuning of the XLSR-10 Base model significantly decreases from 24.9 to 27.9 average CER compared to monolingual fine-tuning. However, increasing capacity helps counteract this: XLSR-10 Large achieves 23.7 avgerage CER which is much closer to monolingual fine-tuning of the Large model (23.2 avg. CER). Increasing capacity is particularly important when fine-tuning on large amounts supervised data from many languages. Multilingual fine-tuning performs competitively to monolingual fine-tuning and enables us to have a single model for many languages.
4.4 ON THE ROLE OF LANGUAGE SIMILARITY ON CROSS-LINGUAL TRANSFER
Next, we study the impact of language similarity on cross-lingual transfer and then analyze the multilingual token embedding space where we find that languages are clustered.
4.4.1
LOW-RESOURCE LANGUAGES BENEFIT MORE FROM SIMILAR LANGUAGES
We consider Italian the low-resource language for which assume only 5h of unlabeled data is available. We pretrain models on the 5h as well as 50h of unlabeled data from several other languages:
Model
#pt
#it
it
it
de
ka
zh
Number of pretraining hours
[Sh (it)
5h (it) 50h (<lang>)
XLSR-Monoling Monolingual
47.6
16.8
24.3
25.4
27
27.2
28.1
30.6
Table 5: Impact of language similarity cross-lingual transfer. We simulate low-resource language scenario by using only hours of Italian Common Voice data and add 50 hours from another language for pretraining. We fine-tune hour of Italian supervised data.
7

---

Under review as a conference paper at ICLR 2021
Italian, Spanish, German, English, Russian, Kabyle and Chinese. Finally, fine-tune each model on Ih of Italian labeled data. Table that adding more unlabeled data helps overall, but adding data from related languages gives the largest improvement, c.g., Spanish. Distant languages, e.g.. Kabyle or Chinese are less effective. In order to improve performance on a low-resource language, it is best to add unlabeled data from closely-related language.
4.4.2 ANALYZING THE SHARED DISCRETE SPEECH REPRESENTATIONS
To analyze the shared quantized latent speech representations, or discrete tokens, we train two models: one on 12 languages of Common Voice and another on 17 languages of BABEL. For each model, we run the quantizer of our model on train and dev speech samples from each language, and compute frequency vector of the discrete tokens. The resulting frequencies are normalized for each language to obtain vectors of size V x G, the number of discrete latent speech representations. The vectors represent the empirical probability distribution over the shared discrete latents. Next, we construct an symmetric similarity between
affinity matrix between languages by computing the vectors. Finally, we cluster languages using K-Means and then perform a PCA with two dimensions.
Figure 2b and 2c show the visualizations, where colors correspond to the clusters obtained by K-Means. Note, that we perform K-Means before PCA to avoid loss of information, and that PCA may make some points appear closer than they are in original vectors. We see that the model shares more discrete tokens for similar languages, e.g.. it groups Basque, Catalan, Spanish and Italian, or English, German and French, or Arabic and Kabyle (see Figure 2ap. and Mandarin (zh-CN and zh-TW), although this information is lost in the PCA visualization. Figure 2b shoss that the model may also isolate a language, such as Chinese-HongKKnn language because it shares fewer discrete tokens with other languages.
(Cantonese), which is not close to any other
For BABEL (Figure pc). we also find language groupings such as Bengali/Assamess which belong to the same family, or Zulu and Swahili which both have long vowels. However one could argue that Pashto and Kurmanji should be closer to each other since they are both Iranian languages. The purpose of this analysis is not recover full language families but to better understand how our model allocates the latent representations across languages. Interestingly, Italian closer to Spanish, the most effective language in the previous experiment (T) Future work may investigate whether encouraging shared tokens between similar languages could further help cross-lingual transfer.
Kabyle
Takgisis
Chinese-TW
Cábuano
English perman
French
Arabic
Kurmanji
Italian
French
Chinase-TW
English
German
Chinese.CM
Georgian
Basque
Spanish
Chinem-IIK
Swabhill jorhlan
Tegaing
Turkish
Assamasn
Pangali
Calatu
Janil
Pas
(a) CV-12 12 model.
(b) CV-12 model + zh-HK.
(c) BABEL-17 model.
Visualization of language similarities learned by the model Figure (a) visualizes the shared discrete latent speech representations across languages for model trained on 12 Common- Voice languages (CV-12). Figure (b) shows that adding Chinese-Hong (zh-HK) shares relatively few latents with other languages. Figure (c) is for model trained on 17 BABEL languages and illustrates that clusters can correspond to similar languages like Bengali and Assamese.
Figure 2:
CONCLUSION
In this work, we investigated unsupervised cross-lingual speech representations learned from the raw waveform. We show that pretraining data in multiple languages improves both over monolingual pretraining as well as prior work, with the largest improvements on fow-resource languages. Fine- tuning the model on multiple languages once enables single multilingual speech recognition model competitive to individually fine-tuned models. Analysis of the discrete latent speech representations reveals that the model shares capacity across languages and particularly so with related languages.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M. Tyers, and Gregor Weber. Common voice: massively- multilingual speech corpus. arXiv, 0tss/912.066700 2019.
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, et al. Massively multilingual neural machine translation in the wild: Findings and challenges. arXiv preprint arXiv:1907.0500 05099, 2019.
Alexei Baevski, Michael Auli, and Abdelrahman Mohamed. Effectiveness of self-supervised pre- training for speech recognition. In Proc. of ICASSP, 2020a.
Alexei Baevski, Steffen Schneider, and Michael Auli. vq-wav2vec: Self-supervised learning of discrete speech representations. In Proc. of ICLR, 2020b.
Alexei Baevski, Henry Zhou, AbdeIrahman Mohamed, and Michael Auli. wav2vee 2.0: A framework for self-supervised learning of speech representations. arXiv, abs/2006. 1477, 2020c.
Herve Bourlard, John Dines, Mathew Magimai-Doss, Philip Garner, David Imseng, Petr Motlicek, Hui Liang. Lakshmi Saheer, and Fabio Valente. Current trends in multilingual speech processing. Sadhana, 36, 2011.
Lukas Burget, Petr Schwarz, Mohit Agarwal, Pinar Akyazi, Feng Kai, Arnab Ghoshal, Ondrej Glembek, Nagendra Goel, Martin Karafíát, Daniel Povey, Ariya Rastrow, Richard Rose, and Samuel Thomas. Multilingual acoustic modeling for speech recognition based on subspace gaussian mixture models. In Proc. of ICASSP. 2010.
William Byrne Peter Beyerlein, Juan M. Huerta, Sanjeev Khudanpur, B. Marthi, John Morgan, Nino Peterek, Joseph Picone, Dimitra Vergyri, and T. Wang. Towards language independent acoustic modeling. In Proc. of ICASSP, 2000.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv, abs/2002.05709, 2020.
J. Cho. M. K. Baskar, R. Li, M. Wiesner, S. H. Mallidi, N. Yalta, M. Karafiát, S. Watanabe, and T. Hori. Multilingual sequence-to-sequene speech recognition: Architecture, transfer learning, and language modeling. In Proc. of IEEE SLT, 2018.
Yu-An Chung and James Glass. Speech2vec: embeddings from speech. arXiv, abs/18033,0896, 2018.
wow-ueenne-t---eueeeee framework for learning word
Yu-An Chung, Wei-Ning Hsu, Hao Tang, and James Glass. An unsupervised autoregressive model 2019.
for speech representation learning. arXiv,
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Fran- cisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. arXiv, abs/1911.02116 2019.
Jacob Devlin, Ming-Was Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv, abs/1810.04805, 2018.
Sander Dieleman, Aäron van den Oord, and Karen Simonyan. The challenge of realistic music generation: modelling raw audio at scale. arXiv, 2018
Ryan Eloff, André Nortje, Benjamin Niekerk, Avashna Govender, Leanne Nortje, Arnu Pretorius, Elan Van Biljon, Ewald van der Westhuizen, Lisa van Staden, and Herman Kamper. Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural networks. arXiv, abs/1904.07556, 2019.
Radek Fer, Pavel Matějka. František Grézl, Oldfich Plchot, Karel Veselý, and Jan Honza Černocký. Multilingually trained bottleneck features in spoken language recognition. Computer Speech & Language, 46:252-267, 2017.

---

Under review as a conference paper at ICLR 2021
Mark J. F. Gales, Kate M. Knill, Anton Ragni, and Shakti P. Rath. Speech recognition and keyword spotting for low-resource languages: Babel project research at cued. n Spoken Language Technologies for Under-Resources Languages, 2014.
Mark JF Gales, Kate M Knill, and Anton Ragni. Low-resource speech recognition and keyword- spotting. In Proc. of SPECOM, pp. 3-19. Springer, 2017.
Arnab Ghoshal, Pawel Swietojanski, and Stephen Renals. Multilingual training of deep neural networks. In Proc. of ICASSP, 2013.
Alex Graves, Santiago Fernández, and Faustino Gomez. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. In Proc. of ICML, 2006.
David Harwath, Wei-Ning Hsu, and James Glass. Learning hierarchical discrete linguistic units from visually-grounded speech. In Proc. of ICIR, 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arXiv, abs/1911.05722, 2019.
Kenneth Heafield, Ivan Pouzyrevsky. Jonathan H. Clark, and Philipp Kochn. Scalable modified Kneser-Ney language model estimation. In Proc. of ACL, 2013.
Georg Heigold, V. Vanhoucke, Andrew Senior, Phuongtrang Nguyen, M. Ranzato, M. Devin, and Jeff Dean. Multilingual acoustic models using distributed deep neural networks. In Proc. of ICASSP, 2013.
Jui-Ting Huang, Jinyu Li, Dong Yu, Li Deng, and Yifan Gong. Cross-language knowledge transfer using multilingual deep neural network with shared hidden layers. In ICASSP, 2013.
Hirofumi Inaguma, Jaejin Cho, Murali Karthick Baskar, Tatsuya Kawahara, and Shinji Watanabe. Transfer learning of ingguage-indeeender end-to-end asr with language model fusion. In Proc. of ICASSP, 2019.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv, abs/1611.01144, 2016.
Herve Jegou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE Trans. Pattern Anal. Mach. Intell., 33(1):117-128, January 2011.
Dongwei Jiang, Xiaoning Lei, Wubo Li, Ne Luo, Yuxuan Hu, Wei Zou, and Xiangang Li. Improving transformer-based speech recognition using unsupervised pre-training. arXiv, abs/1910.09932, 2019.
Anjuli Kannan, Arindrima Datta, Tara N. Sainath, Eugene Weinstein, Bhuvana Ramabhadran, Yonghui Wu, Ankur Bapna, Zhifeng Chen, and Seungji Lee. Large-scale multilingual speech recognition with streaming end-to-end model. In Proc. of Interppeech, 2019.
Kazuya Kawakami, Luyu Wang, Chris Dyer, Phil Blunsom, and Aaron van den Oord. Learning robust and multilingual speech representations. arXiv,, abs/2001.11128, 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In Proc. of ICLR, 2015.
Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. In Proc. of NeurIPS, 2019.
Viet-Bac Le and Laurent Besacier. Automatic speech recognition for under-resourced languages: Ap- plication to vietnamese language. IEEE Transactions on Audio, Speech, and Language Processing. 17, 2009.
AbdeIrahman Mohamed, Dmytro Okhonko, and Luke Zettlemoyer. Transformers with convolutional context for ASR. CoRR, abs/1 904. 11660, 2019.
10

---

Under review as a conference paper at ICLR 2021
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proc. of NAACL System Demonstrations, 2019.
Matthew E Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proc. of ACL, 2018.
Anton Ragni, Qiujia Li, Mark J. F. Gales, and Yu Wang. Confidence estimation and deletion prediction using bidirectional recurrent neural networks. In SLT, Athens, 2018.
Morgane Rivière, Armand Joulin, Pierre- Emmrenel Mazaré, and Emmanuel Dupoux. Unsupervised pretraining transfers well across languages. arXiv, abs/2002.02848, 2020.
Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vec: Unsupervised pre-training for speech recognition. In Proc. of Interspeech, 2019.
H. Seki. S. Watanabe, T. Hori, J. L. Roux, and J. R. Hershey. An end-to-end language-tracking speech recognizer for laxgud-anguagg speech. In Proc. of ICASSP, 2018
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv, 2019.
Andros Tjandra, Berrak Sisman, Mingyang Zhang. Sakriani Sakti, Haizhou Li, and Satoshi Nakamura. Vqvae unsupervised unit discovery and multi-scale code2spec inverter for zerospeech challenge 2019, arXiv, 1905.1 1449, 2019.
Shubham Toshniwal, Tara N. Sainath. Ron J. Weiss, Bo Li, Pedro Moreno, Eugene Weinstein, and Kanishka Rao. Multilingual speech recognition with single end-to-end model. In Proc. of ICASSP, 2018.
Aäron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv, abs/ Abss/187.00374 2018
Ashish Vaswani. Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Proc. of NIPS, 2017.
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Anmand Joulin, and Edouard Grave. Cenet: Extracting high quality monolingual datasets from web crawl data, 2019.


Reviews:
[{"review_id": "ywW0AjXpxrx", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to apply a state-of-the-art self-supervised training method (wav2vec 2.0) to multilingual ASR. The paper provides an intensive analysis of how we use such pre-training-fine-tuning strategies, including 1) multilingual data balance, 2) language-independent pre-trained/fine-tuning model vs. language-dependent pre-trained/fine-tuning model, use of a large model and 3) the visualization of the embedding of each language.  The paper also has some comparisons with the other reports in the same setup. This multilingual ASR scenario is one of the most straightforward applications of self-supervised training, and this paper has significant contributions to the ASR research communities. However, if we consider this paper's contributions in more broad machine learning perspectives (main target in ICLR), they are not significant. First, the method does not have the formulation or algorithm level distinction from the original wav2vec 2.0 paper. The application of this paper is also limited to ASR. The paper should address more applications. I think that we can apply the same methodology to speech translation. Such kinds of applications make the paper better shape in terms of the generalization of the proposed multilingual self-supervised training methodology.\n\nAdditional comments\n- I suggest the authors to consider to submit this paper to ICASSP or Interspeech. The paper would gain more attention in these conferences.\n- There are several complicated tweaking, and I'm concerned about the reproducibility. It would be great if the authors release the recipe for this study.\n- I could not find the detail about the decoding information, e.g., with or without beam search and language models.\n- I'm curious about the robustness (sensitivity) of the noise, recording condition, and speaking styles. For example, what happens if we mix both commonvoice and Babel data since their recording conditions and speaking styles are different. I'd like to have some analysis and visualization results.\n- Sections 2.1 and 2.2 are hard to follow, and they need more elaborated explanations. For example, this part is not very self-consistent, and we need to check the original wav2vec 2.0 paper to understand it (even high-level). No explicit definition about $T$, $G$, $V$, and $p$. \n- Section 3: It is not easy to follow wow to use the data for each condition. I suggest the authors summarize such information by using a table.\n- Many experimental results are very informative, but for me, the most informative result is that if we use a large model, we could use a single multilingual model. I think this finding would be beneficial for our practical multilingual ASR development.", "extra_scores": {}, "extra_text": {}}, {"review_id": "eFL9hSnwxKT", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an unsupervised cross-lingual speech representation learning algorithm for multi-lingual automatic speech recognition. The main building block is based on an existing study (wav2vec series). The idea of this study is to apply previously (successful) unsupervised speech representation learning scheme to cross-lingual scenario to overcome the data sparsity problem of ASR for low-resource language. In this way, the experiments done in this paper are very meaningful and could shed light on future work on low-resource speech processing. The evaluations are intensive, and the results support most of the main claims well.\n\nCons:\n1. Since the building block is from an existing study: wav2vec 2.0 and there is no improvement to it, this paper lacks of novelty in theoretical part.\n2. The experimental design for the investigation on transfer-interference trade-off mainly comes from existing studies (Conneau et al., 2019). No further insight is given. Actually, less analysis is done compared to (Conneau et al., 2019): will the performance on low-resource languages keep increasing if more languages are included.\n3. There doesn't exist a clear definition of \"similar languages\" in section 4.4. Thus the results in section 4.4.2 could be confusing: not sure how these languages are clustered together, because they are overlapping on vowel/consonant space?\n\nQuestions and comments:\n1. The authors mentioned \"Weights of the feature encoder are not updated at fine-tuning time\". Is there a specific reason? Is it because after updating the encoder parameters the performance gets worse?\n2. A follow-up. \"The purpose of this analysis is not to recover full language families but to better understand how our model allocates the latent representations across languages. \". It could be very interesting if some connections can be found between phonological similarities among languages (https://wals.info/feature) and their latent speech representations. For example, how come the black languages (Chinese) are closer to purple languages than blue languages in figure 2 (b)\n3.  Wondering why the authors don't use tsne for visualization. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "C2rsGoRsGfg", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper applied self-supervised learning (wav2vec 2.0) to multilingual speech recognition task. Promising results demonstrated in Babel and Mozilla tasks.\n\nAll the key points studied in this paper has being shown in previous paper, e.g. wav2vec 2.0 shows the effectiveness of self-supervise learning to speech, a long history of speech papers shows multilingual pretrain is good for mono lingual, on low resource task for both e2e or hybrid system. Combining all of these are new, but this is more like a system paper and fit better to a speech conference.\n\nSection 4.4.2 is interesting, but given https://arxiv.org/pdf/1901.08810.pdf has similar analysis (in their case, quantized unit related to phonemes), this analysis method also has being studied. Moreover, applying latent output from a multilingual ASR system to language identification task has being studied before and works very well.\n\nI'd assume the paper give very good results. however, it ignores almost all previous publications on Babel, e.g. there is no comparison with a hybrid model. Table 2 is self-contained, but it would be great also include some previous Babel number so we can see how e2e compare with traditional hybrid model.\n\nThis is a well written and interesting paper. However, there isn't much add to wav2vec 2 and multilingual (transfer) learning methodology. As a milestone paper, for example, super good performance, these task are not representative enough. So I don't think it fit to ICLR and suggest it resubmit to a speech conference.", "extra_scores": {}, "extra_text": {}}, {"review_id": "-1K4cvoCT1U", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors extended the XLSR model from the previous mono-lingual task, where self-supervised learning was used in representation learning, to multi-lingual task. The basic idea does make a lot of sense, where all the data, irrespective of the language, is pushed through the representation learning task.  We are anyways talking about human speech, so it is reasonable to assume that learning one language can help in learning the other language. \n\nMy main criticism is that technical novelty appears to be limited. The core idea appears to be just the use of multi-lingual data instead of mono-lingual data. Authors could emphasize the technical novelty in the Introduction if I am not correct in this assessment. On the other hand, the results authors obtain are excellent so there is merit in publishing them. \n\nOther issue I have is that BABEL has been used in speech community quite a bit, so I am not completely sure whether authors baseline results are the current SoTA. Especially, I find a bit dubious that  (Fer et al., 2017) is used as a baseline, where that paper is dated and deals with language identification and not multi-lingual phone recognition! Actually, these results are not even available from (Fer et al., 2017). In speech community, WER results are typically reported and not PERs, this might make comparison to SoTA a bit more difficult as authors do not seem to use language model at all. So apples-to-apples comparison might be harder. However, I ask authors to make sure the baselines are good enough. \n\nThe other thing is that unsupervised ASR has seen wealth of papers recently, some have used GAN type approaches such as https://arxiv.org/abs/1904.04100 authors could also comment a bit about these approaches. Note in that one PER is also used as a measurement. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work mainly applies wav2vec 2.0 to multilingual speech recognition and lacks of novelty. 
The various pre-training and fine-tuning mix-match are specific to the speech recognition task. As suggested by reviewers, it is recommended to resubmit to a speech conference. 
Also the paper lacks comparisons to SOTA on one of the well studied task (i.e. BABEL) in the speech field.

The main factor for the decision is lack of novelty.
Decision: Reject