Fleurs-SLU: A Massively Multilingual Benchmark for Spoken Language Understanding
Authors: Fabian David Schmidt
Ivan Vulić
Goran Glavaš
David Ifeoluwa Adelani
Venue: COLM 2025 (2025)
Keywords: spoken language understanding
multilingual benchmarks
multilingual evaluation
TL;DR: A massively multilingual benchmark for topical utterance classification and textual multiple-choice QA from spoken paragraphs

Abstract: Spoken language understanding (SLU) is indispensable for half of all living languages that lack a formal writing system, since these languages cannot pair automatic speech recognition (ASR) with language models to benefit from language technology. Even if low-resource languages possess a writing system, ASR for these languages remains unreliable due to limited bimodal speech and text training data. However, the evaluation of multilingual SLU remains limited to shallow tasks such as intent classification or language identification. To address this, we present Fleurs-SLU, a multilingual SLU benchmark that encompasses (i) 692 hours of speech for topical utterance classification in 102 languages and (ii) multiple-choice question answering through listening comprehension spanning 944 hours of speech across 92 languages. We extensively evaluate both end-to-end speech classification models and cascaded systems that combine speech-to-text transcription with subsequent classification by large language models on Fleurs-SLU. Our results show that cascaded systems exhibit greater robustness in multilingual SLU tasks, though speech encoders can achieve competitive performance in topical speech classification when appropriately pre-trained. We further find a strong correlation between robust multilingual ASR, effective speech-to-text translation, and strong multilingual SLU, highlighting the mutual benefits between acoustic and semantic speech representations.

Published as a conference paper at COLM 2025
Fleurs-SLU: A Massively Multilingual Benchmark for Spoken Language Understanding
Fabian David Schmidt', Ivan Vulie², Goran Glavaš', David Ifeoluwa Adelani
1 Center For Artificial Intelligence and Data Science, University of Würzburg, Germany
2 Language Technology Lab, University of Cambridge, United Kingdom
3 Mila, McGill University and Canada CIFAR Al Chair
Abstract
Spoken language understanding (SLU) is indispensable for half of all living languages that lack a formal writing system. Unlike for high-resource lan- guages, for these languages, we cannot offload semantic understanding of speech to the cascade of automatic speech recognition (ASR) and text-based large language models (LLMs). Even if low-resource languages possess a writing system, ASR for these languages remains unreliable due to lim- ited bimodal speech and text training data. Nonetheless, the evaluation of multilingual SLU limited to shallow tasks such as intent classification or language identification. This is why we present Fleurs-SLU, multilingual SLÚ benchmark that encompasses (i) 692 hours of speech for topical utter- ance classification in 102 languages and (ii) multiple-choice question an- swering via listening comprehension spanning 944 hours of speech across 92 languages. We extensively evaluate end-to-end speech classification models, cascaded systems that combine speech-to-text transcription with subsequent L.L--based classification, and multimodal LLessh-LLLMs on Fleurs-SLU, Our results show that cascaded systems are more robust in multilingual SLU, though well-pretrained speech encoders can perform competitively in topical speech classification. Closed-soure speech-LLMs match or surpass the performance of cascaded systems. We observe a strong correlation between robust multilingual ASR, effective speech-to- text translation, and strong multilingual SLU, indicating mutual benefits between acoustic and semantic speech representations.
1 Introduction
Only about half of the world's living languages possess formal writing system, under- scoring the need for massively multilingual speech technology (Ethnologue, 2017). Spoken language understanding (SLÚ) is hence critical feature of inclusive technology for these languages, which cannot rely on combining automatic speech recognition (ASR) with lan- guage models to offload semantic speech understanding. However, the datasets currently available for evaluating SLU in truly relevant languages suffer from sigmificant limitations. For instance, the Minds14 benchmark assesses SLU on 14 exclusively high-resource lan- guages for intent classification, task that often requires only shallow semantic processing, such as detecting specific keywords (Gerz et al., 2021). Likewise, while SpeechTaxi focuses on spoken topical classification of Bible verses in 28 diverse languages, SL. of Bible verse arguably does not reflect real-world usage (Keller & Glavaš, 2025).
Moreover, even when low-resource language has a writing system, multilingual ASK often struggles to reliably transcribe that language due to the limited availability of bimodal speech and text data. Pre-trained on the self-supervised wav2vec-BERT objective (Baevski et al., 2020), these models embed acoustic features that capture phonetic rather than semantic information (Choi et al., 2024). Conversely, multilingual speech models that excel in semantic
'The datasets per-language results are available Huggingface: SIB-Fleurs; Belebele-Fleurs.
1

---

Published as a conference paper at COLM 2025
encoding tend to be more robust in ASR tasks by, e.g., leveraging context-related cues or cross-lingual similarities such as cognates or similar syntactic structures (cf. $5.1). This underscores that robust SLU is instrumental for genuinely inclusive speech technology.
To address the shortcomings of multilingual SLU evaluation of prior benchmarks, we compile Fleurs-SLU by realigning datasets derived from Flores (Team et al., 2022), dachine-translltion benchmark with parallel sentences for over 200 languages. We first filter out silent instances from Fleurs (Conneau et al., 2022) and then map the remaining data back to Flores. The resulting dataset is then merged with SIB-200 (Adelani et al., 2024) and Belebele (Bandarkar al., 2023), that both are based on Flores, to create topical utterance classification benchmark for 102 languages, and dataset for multiple-choice question answering (QA) on spoken paragraphs for 92 languages, respectively.
To the best of our knowledge, we are the first to present multilingual SLU benchmark that spans over 100 languages and enables end-to-end SLU evaluation of multilingual speech encoders (mSE). SIB-Fleurs supports utterance classification for 7 topics with 692 hours of speech spanning 102 languages, over 70 more additional languages than existing benchmarks. Belebele-Fleurs provides textual multiple-choice QA from spoken paragraphs, totaling 944 hours of speech across 92 languages and exceeding concurrent work by 18 languages (Costa-jussà et al., 2024). 2) We extensively benchmark state-of-the- art speech models on SIB-Fleurs and Belebele-Fleurs. Like prior work, we observe that Cascaded Systems (CS) remain more robust than mSE, enabling SLU competitive with text-based language understanding (Keller & Glavaš, 2025). We additionally benchmark state-of-the-ard speech-LLMs on multilingual SLU and find that often perform on par or better than CS. We are the first to show that mSE pre-trained on language understanding objectives can yield performance competitive to that of CS on speech classification. We further isolate utterance quality as an important factor in multilingual SLU and show that, in zero-shot cross- cross-lingus lingual transfer (ZS-XLT), worse utterance quality can substantially deteriorate transfer performance. 3) We empirically demonstrate that strong multilingual SLU coincides with much more robust multilingual ASR and higher-quality speech-to- English-text translation (S2FTT). This suggests that the pre-training of multilingual speech models may take SLU into account to make multilingual ASR more robust.
Contributions. 1)
2 Related Work
Multilingual Speech Representation Learning. Modern speech representation models are pre-trained with self-supervised objectives that are optionally followed-up by ASR and speech-to-ter translation (S2TT) training. Among these models, mHubert employs the self-supervised wav2vec 2.0 objective on 90K hours of speech across 147 languages by predicting pseudo-labels derived from clustering raw speech features (Boito et al., 2024). Mb is also pretrained with wav2vec 2.0 on a large corpus of 491K hours of speech spanning 1,406 languages (Pratap et al., 2024). Whisper-v3 is a Transformer encoder-decoder that has been multi-task pre-trained on multilingual ASR, (S2ETT), spoken language identification, and Voice activity detection on 680k hours of audio (Radford et al., 2022). SeamlessM4TT22-Larg
models
neex-ttranslttion
is multilingual multimodal translation model (Seamless Communication et al., 2023). It combines a texi-to-text translation (T2TT) model pre-trained on 105 languages and a Conformer speech encoder pre-trained with w2v-BERT 2.0 on 4.5M hours of audio. The model is then trained on T2TT, S2TT, ASR, and knowledge distillation (KD) objectives. The authors perform KD from the text encoder to the speech encoder by minimizing the KL-divergence between the decoder's 's token output distributions on bi-modal speech and text data.
Multilingual SLU. The evaluation of multilingual SLU is constrained to limited set of tasks. SLU has been significantly shaped by task-oriented dialogue (ToD), with datasets cre- ated focusing on ToD-specific tasks such as intent classification and slot filling. These tasks frequently require only basic semantic understanding, often reducing to merely detecting specific keywords. Additionally, commonly used utterance-level SLÚ tasks like language identification (LID) and sentiment classification do not assess content-based understanding but instead rely on phonetic or prosodic features of speech encodings. As result, the
2

---

Published as a conference paper at COLM 2025
majority of SLU datasets are predominantly in English. Multilingual exceptions other- wise limited. The Mindsl4 dataset for intent classification only includes 14 high-resource languages (Gerz et al., 2021). SpeechTaxi offers spoken topical classification for Bible verses in 28 diverse languages, which however do not adequately represent real-world domains (Keller & Glavaš, 2025).
Concurrent Work. Costa-jussà et al. (2024) concurrently released the multilingual SLU benchmark BM-BELEBELE for 74 languages. In this dataset, the authors first extend Fleurs by approx. 20% by incorporating human recordings for sentences that are part of Flores but were missing in Fleurs, as well as the questions and answers from Belebele. Costa-jussà et al. (2024) find that a CS-based approach on transcribing speech with Whisper-v3-Lard and subsequently prompting Llama 70B with the transcription trails prompting the LLM on clean text by approx. 8 percentage points (pp).
3 Fleurs-SLU
We create Fleurs-SLU, a massively multilingual SLU benchmark for speech classification and multiple-choice QA from spoken paragraphs, from datasets that are based on Flores.
3.1 Core Datasets
Flores consists of professionally translated 3,001 sentences of English Wikipedia paragraphs to evaluate machine translation (Team et al., 2022).2 2 Fleurs comprises 2.3 spoken utterances, on average, per sentence from the DEV and DEVTEST splits of 102 languages in Flores (Conneau et al., 2022). 3 Fleurs is used to evaluate multilingual ASR, LID, well as speech- to-text and text-to-speech tanslation in all language directions. SIB-200 refined the topical metadata annotations of sentences in the DEV and DEVTEST splits of Flores into categories topical classification benchmark for 205 language variants. Belebele is multiple-choice reading comprehension benchmark for 122 languages (Bandarkar et al., 2024). The authors reconstruct paragraphs from Flores sentences. They generate 1-2 questions per English paragraph, which are professionally translated into 121 languages. Belebele comprises 900 questions that span across 488 passages.
(Adelani et al., 2024). 4 The resulting SIB-200
3.2 Benchmark Creation
We compile Fleurs-SLU by carefully aligming data from the above benchmarks as follows.
1) Merging Fleurs& & Flores. We begin by removing silent instances from Fleurs.5 5 We first normalize the loudness to a target RMS level of -25 dB. We next apply voice activity detection using Silero-VAD (Team, 2024). 6 Samples are deemed silent if speech is detected in less than 5% of their duration.? Lastly, we verify our approach on 50 randomly sampled predicted silent instances. We find only one borderline misclassified example, which is noisy but comprehensible. We then conservatively merge Fleurs and Flores by matching instances first on exact string match and then by Levenshtein distance of 3 on normalized strings. 8
2a) SIB-Fleurs. For each language, we pool instances from the training, validation, and test splits of SIB-200 (Adelani et al., 2024) and align the data with our merged Fleurs-Flores dataset with the same string alignment procedure as before. The data is then regrouped
Flores comprises 3,001 sentences divided into DEV (997 sentences), DEVTEST (1,012 sentences), and TEST (992 sentences) sets. The authors did not release the TEST set.
3 Almost all languages part of Fleurs however are missing few hundred sentences from Flores. 4" "science/ /technologg"" "travel", "politics", , "sports", "health", "entertainment", "ggogrrphy"..
5Counts of removed examples are listed Appendix A.6. hemm//aaers//sllerorror
/ Phttps://gittee
7The Silero-Ve VAD pipeline also frequently removes inaudibly noisy samples as a side effect.
Normalized strings remove characters that are Unicode punctuation codepoints.
3

---

Published as a conference paper at COLM 2025
into the training, validation, and test splits of the original Fleurs dataset. This segmentation ensures compatibility for speech models that may be trained on ASR using the training set of Fleurs prior to SIB-Fleurs evaluation. This also ensures that the speakers in the training set are different from those in the validation and test sets. Table lists aggregated statistics the instance- and the utterance-level for SIB-Fleurs. Appendix A.2 provides list of samples by split per language.
Utterance
Multiple-Choice QA
Classification
Spoken Paragraphs
Languages
102
Languages
92
Classes (Topics)
7
Answer Choices
4
Utterances per sample
2.2 (2.0)
Questions per paragraph
1.8 (1.8)
Duration per utterance (s)
12.9 (11.6)
Sentences per paragraph
3.6 (4.0)
Total audio (hr)
692
Utterances per sentence
2.0 (2.0)
Samples by Split
Duration per utterance (s)
(12.0)
Training
696 (728)
Duration per paragraph (s)
47.0 (43.4)
Validation
66 (70)
Total audio (hr) Samples (Paragraphs)
944
Test
163 (174)
709 (771)
Table 1: Statistics of SIB-Fleurs. Utterance- level metrics are aggregated by language and then pooled over languages. Metrics: either sums or averages (median).
Table 2: Statistics of Belebele-Fleurs. Met- rics are aggregated by language and then pooled over languages. Metrics: either sums or averages (median).
2b) Belebele-Fleurs. We merge our Fleurs-Flores sentences with Belebele paragraphs by intersecting the URLs of the texts. We discard all paragraphs that are not complete in Fleurs- Flores. We verify our reconstructed paragraphs against the original Belebele by ensuring that the Levenshtein distance for strings with removed punctuation is negligibly small (less than characters). Table 2 provides summary statistics on both the paragraph- and the sentence-level for Belebele-Fie Fleurs, while Appendix A.3 lists the samples per language.
4 Experimental Setup
4.1 Tasks and Languages
SIB-Fleurs. We train mSE on the utterances and CS on the transcriptions of the English training set. For both mSE and CS, we feed the sequence-level representation pooled from token or speech frame embeddings into a classification head. Roberta-Large uses the [CLS] token sequence-level embedding. All other models average the token speech frame output embeddings.
Belebele-Fleurs. We train and validate CS models on the English training and dev sets of Belebele (Bandarkar et al., 2024), respectively. We jointly embed the paragraph, question, and choices with text encoders. We then average the token encodings of each choice c; C and project the choice embedding via head DD to logit lq We minimize the cross-entropy [Cl of the concatenated choice logits { the label choice."
4.2 Cross-Lingual Transfer Setups
We experiment two commonly used cross-lingual transfer (XLT) paradigms. Zero-shot cross-lingual transfer (ZS-XLI) and Translate-Tess (TTEST) allow us to evaluate XLT without requiring additional annotation for any target language. In 7S-XLT, gual model on the English source-language data (cf. $4.1) and then directly run inference on the larget-language test instances. In TTEST, the model is also first fine-tuned on labeled English source-language data. At test time, the target-language examples are translated
first train multilin-
We do not evaluate speech LLMs existing models are based on Whisper-v3 which is limited to 30 seconds of audio input. We leave such evaluation future work.
4

---

Published as a conference paper at COLM 2025
to the source language prior to inference, which enables XLT with monolingual LLMs. We additionally evaluate zero-shot prompting of speech-LLMs. Here, we provide the context of the task in English and all relevant input in-language in the respective modality, ie. text or utterances, to the speech-LLMMs (cf. $A.1).
Speech Classification. We evaluate state-of- ofte----ttthhttttte mSE for speech classification (cf. $2). We include MMS-1B without fine-tuning ('MMS-1B"), with ASR fine-tuning on Fleurs ('MMS- 1B-Fleurs'), and with ASR fine-tuning on multilingual datasets ('MMS-IB-all'), allowing us to analyze the impact of ASR fine-tuning on cross-lingual SLU. For ASR fine-tuning, Pratap et al. (2024) train language adapters and language-specific decoding heads while keeping all other parameters frozen. For task fine-tuning, we freeze the adapters to facilitate ZS-XLT. We further evaluate mHubert and the speech encoders of both Whisper-v3-Large and SeamlessMM4TV2-Large (cf. $2).
Cascading & Text. Cascaded systems (CS) perform XLT in two steps: (i) an ASR model tran- scribes speech into text, and (ii) text encoder processes the transcription via classification head. We consider two transcription targets: the target language (in-language) and English Oppeechtto-English translation, S2ETT). S2ETT corresponds to TTEST, enabling XLT with monolingual LLMs. We use SeanlessM4IIV-LLrge and Whisper-v3-Large as ASR backends. For languages not supported by the model, we manually select the closest available lan- guage for in-language transcription. All decoding is greedy, as more complex decoding strategies do not improve XLT performance (Ebing & Glavaš, 2024). We additionally evaluate on the original text (TEXT) of SIB-200 and Belebele. For TTEST, we translate this text to English using SeamlessMHTv2-LLne
10 For classification, we evaluate three text encoders: Roberta-Large (Liu et al., 2019), LLM2Vec (BehnamGhader et al., 2024), and NLLB-LLM2Ves (Schmidt et al., 2024). LLM2Vec is sequence encoder based Llama 3 8B (AI@Meta, 2024), trained with bidirectional attention on masked next-token prediction and SimCSE (Gao et al., 2021). NLLB-LLM2Ve extends LLM2Vec with the encoder of the NLLB translation model, covering 200+ languages for robust multilingual NLU (Schmidt et al., 2024).
Speech-LLMs. We further evaluate two multimodal speech-LLMs, Qwen 2.5 7B-Omni (Xu et al., 2025) and Gemini 2.0 Flash in zero-shot prompting setup. 12 For each task, we provide an English task description and supply the task-specific input (e-g., question, paragraph, or utterance) in the target language, using the appropriate modality (text or speech). To ensure consistent audio quality across languages, we normalize all utterances to a root mean square (RMS) level of 0.07. For Belebele-Fleurs, where inputs consist of multiple utterances, we normalize the concatenated sequence. Appendix $A.1 details the prompt formats and input preprocessing used for each task and modality.
4.3 Further Details
Hyperparameters. We train all mSE and CS models with AdamW (Loshchilov & Hutter, 2019), weight decay of 0.01, and with 10% linear warm-up and followed by linear decay on an effective batch size of 32. For SIB-Fleurs, we run a grid search over the learning rates (1,2,..., 9, 10 since suitable hyperparameters have not yet been extensively studied for mSE on such downstream task. For Belebele-Fleurs, we fine-tune the LLMs on the learning rates (1, with LoRAs of rank 1- and alpha a x=32 attached onto all linear layers, as full fine-tuning is prohibitively expensive. We train models for 20 and 3 epochs for SIB-Fleurs and Belebele-Fleurr Fleurs, respectively, and validate at every 10% of training steps. Experimental results are averaged across random seeds. We report results for runs on the learning rate that performs best, on average, on the English validation sets. For experiments with Qwen 2.5 7B-Omni and Gemini 2-Flash, we perform greedy decoding. We report further experimental details of our prompting experiments in SA.11.
We use the gold Q& in English for TTEST. Otherwise, MT models would need to be combined, e.g., STETT of paragraphs with Whisper, and translation of textual Q&A with SeamlessM4Tv2. 11 htthtt//////l
flash Note that the audio encoder of Qwen 2.5 7B-Omni is initialized with Whisper-v3-Large prior to further cross-modal instruction tuning.
woogle.com/ver google. com/ver ai/-anerggtnttte camerative-aili httol////ooes///eeeeeeeoeeeeee/e
5

---

Published as a conference paper at COLM 2025
Text vs. Utterance Quality. We evaluate models trained on the original text (TEXT) as well as on speech data of varying quality, Each sentence in Fleurs is associated with one or more utterances. To quantify speech quality, we compute the character error rate (CER) between the reference Flores sentence and its transcriptions produced by Whisper-v3-Laa SeamlessM4Tv2-L.an
and Based on these CER scores, we construct two utterance subsets: one comprising the lowest-CER utterances (best quality), and one with the highest-CEi, utterances (worst quality). We fine-tune the mSE and CS models and evaluate all models (incl. speech-LLMss) separately on each subset to isolate the effect of utterance quality on downstream performance. This setup also balances the training data to enable fair comparison between mSE, CS, and TEXT.
Results
Table summarizes the main results for both tasks by approach across our XLT setups. We dissect the results along several axes of analysis.
English. The first column presents the in-language English performance by task, modality (TEXT vs. mSE, CS, and speech-LLM), and utterance quality by model.
Text, CS, and Speech-LLMs, English ASR performance is strong for both transcription models (cf. Figure 1), and CS effectively leverages the NLU capabilities of LLMs, thanks to their extensive pre-training on both tasks. CS are even on par with and sometimes outperform models trained on gold text, underscoring the quality of ASR on English. The slight outper- formance of CS over TEXT in SIB-Fleurs results from model selection on comparatively small validation splits (cf. Table 1). As result, CS outperforms all mSE models SIB-Fleurs. On Belebele-Fleurs, both CS models backed by LLM2Vec and NLLB-LLM2Ve perform com- parably. In sum, the transcription model has no clear impact on English performance. The speech-LLMs perform on par or better on Belebele-Fleurs while slightly trailing the best fine- tuned models on both CS and TEXT on SIB-Fleurs. Gemini 2.0 Flash consistently performs better than Qwen 2.5B 7B-Omni. Notably, their cross-modal gap (TEXT vs. speech-LLM) is small (approx. -15%)..
mSE. The English results SIB-Fleurs suggest that the SLU capabilities of mSE models are strongly shaped by their pre-training curriculum. mHubert (41.1%) and MMS-1B (44.6%), which are pre-trained solely in self-supervised manner, underperform other mSE on SIB- Fleurs. Fine-tuning MMS-1B 1B on ASR, either on Fleurs (MMS- B--Feurss or Fleurs combined with additional data (MMS-1B-AII results in significant improvements on SIB-Fleurs of +1 +10-20 percentage points (pp). Notably, MMS-IB-Fleurs outperforms MMS-1B-All (+9.6 pp), suggesting that the broader domain mixture in the MMS-1B-All training set negatively affects this task. Whisper-v3-Large surpasses all MMS-1B variants (78.3%). The large- scale pre-training of Whisper-v3 on both multilingual ASR and S2ETT enhances its SLU performance. SeamlessM4TTV2-Lagg is the best performing mSE on the English test set of SIB-Fleurs (87.4%), with only a slight performance gap compared to CS (approx. -3 pp). The joint pre-training multilingual ASR well as multilingual and cross-modal MT with text-to-speech knowledge distillation (KD) significantly enriches the semantics in speech representations of SeamlessMMTT L-Laage. The results particularly highlight that (i) multilingual ASR, (ii) S2ETT pre-training, and (iii) text-to-speech KD are crucial training objectives for enabling speech encoders to acquire strong SLU capabilities.
Utterance Quality. The quality of English utterances does not affect performance of the models across both tasks. This can be attributed to the large-scale training on English ASR data of all models. This is why CS are also on par with TEXT.
ZS-XLT. The right-hand side of Table 3 reports ZS-XLT performance. For each task, we group the languages into (i) languages supported by Whisper-v3, (ii) languages supported by SeamlessM4TV2, and (iii) languages unsupported by either model. Whisper-v3 and SeamlessM4TV2 overlap in their support for 81 Fleurs languages." 13
For unsupported languages, hand-select the clasest supported language transcribe into.

---

Published as a conference paper at COLM 2025
English
Non-English
Utterance Quality (Best; Worst)
W
W
Language Groups (Size)
Setup
EN
Whisper
S4T
Unsup.
Non-EN
SIB SIB-Fleurs
(1)
(85)
(90)
(101)
MHUBERT
X
41.1
39.7
27.3
19.1
27.2
19.1
26.5
16.3
26.9
18.6
MMS-1B
46.9
19.9
13.6
19.8
13.5
20.8
13.3
19.8
13.4
mSE
X
64.8
64.2
25.2
20.8
24.7
20.3
27.1
22.1
25.0
20.6
MMS-1B-ALL
X
55.2
53.1
21.7
18.1
21.4
17.9
24.9
21.0
21.7
18.2
WHISPER-V3-L
X
78.3
76.3
46.0
42.0
45.5
41.3
39.4
38.4
44.7
40.8
SEAMLESMATV2-L
X
87.4
88.5
82.3
79.3
82.2
79.1
55.7
52.9
79.0
75.8
ROBERTAL ARGE WII-EN
90.4
91.7
75.8
72.7
73.6
70.8
52.0
48.7
71.5
68.4
ROBERTAL ARG S4T-EN
91.7
90.8
86.2
84.5
85.9
84.2
56.3
55.2
82.3
80.6
CS
LLM2VEC-WH
91.1
92.3
76.6
75.3
74.5
73.4
53.0
53.2
72.4
71.1
T.L.2VC--SATE S4T-EN EN
93.0
91.3
87.1
85.2
86.9
84.9
58.1
36.6
83.4
81.4
NLLB- LLLB-L WH
92.5
91.0
77.5
73.9
75.4
72.2
54.4
53.2
73.7
70.4
NLLB- WLLB-LLM2VER LLM2VEC-S4T
94.0
92.7
84.9
85.3
84.9
62.3
82.3
81.9
Speech- LLM
OWEN 2.5 7B-OMNI
P
87.0
87.6
59.5
58.4
58.3
57.3
45.6
45.6
56.
55.9
GEMINI 2.0 FLASH
87.6
88.7
81.4
80.8
80.0
59.4
78.6
ROBERTAL ARGI SRGFFS S4T-EN LLM2VEC-S4T-DE NLLB-LLM2VER OWEN 2.5 7B-OMNI GEMINI 2.0 FLASH
87.4
55.3
85.2
T
91.0
86.4
87.9
55.6
84.1
TEXT
92.3
88.1
17.99
80.7
87.2
P
88.7
76.4
75.1
53.9
73.0
90.4
77.2
Belehele-Fleurs
(1)
(79)
(84)
(91)
ROBERTAL ARGE WH-EN ROBERTAL LEEEAALLAGEEAAAA ARGE-S4T-EN
81.3
56.2
55.2
55.5
82.0
79.4
67.5
66.4
67.0
65.8
47.2
46.2
65.2
64.1
CS
LLMZVEC-S4T-EN NLLBBLLL2VEE--M
LLM2VEC-WI-EII
95.5
94.9
94.4
94.4
84.0
74.6
83.1
73.5
73.6
83.5
72.6
82.5
64.7
63.5
63.6
72.7
81.8
80.9
94.8
93.5
56.2
56.8
40.9
55.5
53.8
N...B-II. M2BBE...MVVE--SS
X
94.9
93.5
61.3
60.2
60.8
59.6
43.9
41.1
59.2
57.8
Speech- LLM
QWEN 2.5 7B-OBBn 18-ONNI GEMINI 2.0 FLASH
P
91.6
91.0
4
50.8
50.4
49.7
34.4
34.4
49.0
48 4
94.1
93.4
84.4
83.5
$2.9
82.0
59.7
81.1
80.2
ROBERTAL ARGI ARGE-SSHHH S4T-EN SANE NILBBLLL
83.3
72.1
72.6
48.3
70.5
95.3
87.8
88.3
66.0
86.3
TEXT
95.1
65.6
55.2
63.8
OWEN 2.5 7B-OMNI GEMINI 2.0 FLASH
P
94.1
69.1
67.4
36.6
64.9
95.9
90.8
82.0
89.1
Table 3: ZS-XLT with mSE, CS, and TEXT and zero-shot prompting with speech-LLMs
(cf. $4), We report accuracy averaged over 3 seeds on checkpoints that maximize perf. on English validation sets. CS: The suffixes WH and S4T denote Transcription & Translation (incl. -EN:
respectively. Numbers in paren- theses denominate size of group (e.g., Whisper-v3 supports 84 languages of SIB-Fleurs). X=Zero-shot Cross- Cross-Lingual Taansfer----aanslltt--e-s-------
Prompting (cf. Unsup=UUssupported. The (second-)ber model in each column bold (underline).
Setup:
$4). Abbreviations:
Text, CS, and Speech-LLMs. Across both tasks and all language groups, models trained on transcriptions of SeamlessM4TV2-Larg consistently outperform models fine-tuned on transcriptions of Whisper-v3-Lange. The performance gap grows Whisper's support decreases or becomes unavailable for the target languages (cf. 'S4T' and 'Unsup.' language groups in Table 3). Nevertheless, while CS are mostly competitive on SIB-Fleurs, they trail models evaluated on ground-truth paragraphs more significantly on Belebele-Fleuurs (approx. -5 pp). We presume that transcription and S2ETT errors propagate more severely in multiple-choice QA. Moreover, the best speech-LLM, Gemini 2.0 Flash, performs highly competitively across tasks, boasting on par or better performance than other models.
For SIB-Fleurs, all CS with Seamless mance all 101 target language, on average, relative to English (approx. -10 pp). Most of this XLT gap comes from the 7 target languages that Seamless (approx. -35 pp on average). In contrast, pairing LMs with Whisper-v3-Large causes more pronounced drops on languages that Whisper supports (approx. -13 pp) and again larger deficit on languages neither model supports (approx. -32pp). CS are highly competitive to TEXT (approx. -0.7 pp). Only NLLB-LLMP Vec evaluated on gold text tremendously outperforms all other models on unsupported languages, since they are supported by NLLB (Team et al., 2022).
deteriorate only slightly in XLT perfor-
does not support
7

---

Published as a conference paper at COLM 2025
For Belebele-Fleurs, S2ETT paired with LLM2Vec (i.e., TTEST) outperforms ZS-XLI cascading on in-language transcriptions and NLLB-LLM2Vec, except for unsupported languages. This suggests that S2ETT of both Whisper-v3 and SeamlessM4TV2 sufficiently translates the target languages into English for successful NL.U. On unsupported languages, however, NLL.B-LLLMVec performs better for likely two reasons. First, for languages with low S2ETT quality, in-language transcriptions to closely related languages likely better preserve the core meanings of input sequences. Second, in addition translation, NLLB was pre-trained with denoising autoencoding, making NLLB-LLM2Ved more resistant to noisy inputs than LLM2Vec. Overall, the findings suggest that SeamlessM4Tv2-Large is more robust for both in-language ASR as well as S2ETT (cf. Figure 1).
The performance of speech-I.LMMs varies substantially across models and language groups. Gemini 2.0 Flash strongly outperforms Qwen 2.5 7B-Omni on both tasks in non-English settings. Qwen exhibits marked performance degradation when transitioning from English to non-English inputs, dropping by approx. 30 pp in TEXT, and even more steeply in the speech modality, where the cross-modal gap widens from approx. 3% (English) to 18% (non-English). In contrast, Gemini maintains strong performance inputs and shows only a moderate cross-modal drop of approx. 9%. This however exceeds the gap for the best fine-tuned CS models their TEXT counterparts (approx. 3%). as its audio encoder, making it sensitive to Whisper's
pp
non-English TEXT
Qwen relies on Whisper-v3-Lanee transcription limitations. Moreover, its instruction tuning possibly is focused on a small subset of high-resource languages, potentially limiting its generalization to languages not seen during alignment fine-tuning (Xu et al., 2025). This consistent with its sharp drop in performance on unsupported languages. By contrast, Gemini's closed nature precludes detailed analysis, but its robustness across modalities and languages suggests more extensive and diverse instruction tuning setup. It may also benefit from large-scale multilingual pretraining, enhanced alignment objectives, or broader training coverage.
mSE. The ZS-XLT performance of mSE on SIB-Fleurs mirrors the trends we observed for English performance. When trained solely with the wav2vec 2.0 objective, MMS-1B (19%) only slightly surpasses random performance (14%) in 75-XLT for the highest-quality utter- ances, regardless of the target language group. In contrast to English, post-hoc modular ASR fine-tuning of MMS-1B with language adapters and language-specific decoding heads, whether on Fleurs alone (MMS- yields much smaller gains in ZS-XLT (approx. p1) Furthermore, despite employing fully parameter-shared multilingual ASR and S2ETT pre-training, Whisper-v3-Large fails to transfer strong English in-language performance other languages effectively, with the XLT gap ranging from -32.3 pp for supported languages to -38.9 pp for unseen languages. In contrast, only M4a
or on Fleurs with additional data 1MMS-1B-All
achieves performance comparable to English on sup- ported languages (-5.2 pp), though also deflates in performance unsupported languages (55.7%). Overall, these findings nevertheless indicate that multilingual cross- cross-modal transla- tion and multilingual sot-to--spechh distillation align and semantically enrich multilingual speech representations to enhance cross-lingen cross-lingual SLU.
Utterance Quality. The utterance quality consistently affects performance on both tasks for all model configurations evaluated on Eng-Esh languages. Notably, CS and speech-LLMs seem to be less affected than mSE. We attribute this to the much more sizable pre-training on text of diverse quality of LLMs compared to speech-only models. For SIB-Fleurs, the magnitude of the performance gaps between best and worst quality utterances in ZS-XLT mimics how well the speech model backbones perform across both tasks in both CS and end-to-end speech classification. Meamless than Whisper-v3-Large whereas MMS variants suffer from the largest drops (approx. 5 pp).
more robust to noisy utterances
5.1 Further Analyses and Discussion
ASR & Translation Performance. To further understand the underlying factors behind our main results (cf. Table 3), we benchmark Whisper-v3-Large and SeamlessM4TV2--arrg ASR and -eatttt--nnniss-e----eee
(S2ETT) on all 102 Fleurs languages in Figure 1. We first compute CER and SaC between the Flores sentences and the in-language
8

---

Published as a conference paper at COLM 2025
Automatic Speech Recognition
Machine Translation
Wan
Win
CER Hover & Anted)
sacreBLEI EU ii
60
30
45
22
30
15
15
EN
Whisper (85)
S4T
Unsupported
Not-EN (101)
Whisper
S4T
Unsupported Not-EN
Lang, Group
(No. langs)
(90)
(7)
(85)
(90)
(7)
(101)
Whisper-1arge-v3
SeamlessM4TV2-Larg
Figure ASR & Speech-to-English-Ters Translation (S2ETT). CER and sacreBLEU scores for ASR and S2ETT outputs on Fleurs utterances, evaluated against original Flores sentences and pooled across all splits. Numbers parentheses indicate the number of supported languages per model; e.g., Whisper-v3 (SeamlessM4Tv22 supports 85 (90) of the 101 non- English Fleurs languages. The two models combined do not support 7 languages. 'Win Ratio' denotes the number of languages which model outperforms the other. Abbreviations: Whisper = Whisper-v3-Large) S4T SeamlessM4TTV2-Larg
transcriptions and S2ETT transcriptions of utterances in Fleurs pooled across all splits, respectively.4 14 We 'macro-average" the metrics over all languages.
ASR. Whisper-v3-Large outperforms SeamlessM4Tv2-Large for ASR on 58 out of 102 Fleurs languages (ef. 'win ratio'). Nevertheless, SeamlessMMATv2-Largee is the overall more robust transcription model: in transcription for languages other than English, the CER of Whisper-v3- large more than twice high the CER of Seamless per-language results (cf. Appendix A.5)) confirm that SeamlessMHTV22LLarg exhibits much better ASR support for low resource languages, while being competitive in transcription quality for higher-resource languages.
on average. The full
S2ETT. Sww.Seealllee122-aarre. outperforms Whisper-v3-large across the board in speech-to- English-text--rrasslaion and is favored for all but 2 languages. For both groups of languages supported by Whisper-v3-Large and SeamlessM4Tv22LL. Large achieves on average about 8 higher sacreBLEU than Whisper-v3-Large. The gap from SeamlessM4T-Largge to Whisper-v3-Lard reduces to about 5 sacreBLEU for unsup- ported languages, This supports the notion that SeamlessM4TTV2-Large is more robust multilingual speech encoder.
respectively, SeamlessM4TV2
The more robust ASR and much stronger translation performance of SeamlessM+TVV2 results stem from its pre-training. SeamlessM4TV2 first initializes a text encoder and decoder with weights from pre-trained NLLB translation model (Team et al., 2022) and speech encoder pre-trrined on 4.5M hours of self-supervised training on w2v-BERT 2.0 objective. The model is then trained translation objectives from text and speech to text between backbone is
any two languages in both translation directions. The text encoder-decodee used to train the speech encoder with token-level knowledge distillation objectives on decoder output representations. On the contrary, Whisper-v3 trains models from scratch on, among others, in-language ASR and S2ETT (cf. S4). Consequently, the mixture of strong initialization from existing MT backbones, text-to-speech knowledge distillation, and multimodal translation objectives result in much stronger translation performance.
The main results (cf. Table 3), together with our ASR and S2ETT analyses, underscore that sizable text and pre-training coupled with cross-modal and multilingual translation and text-to-speed knowledge distillation infuses rich semantic knowledge ledge into mSE, as witnessed by the SLU performance SeamlessMMIIV--based models on Fleurs-SLU.
1eWe verify that there is no leakage of dev and test splits of Fleurs in SeamlessM4IV--Large and Whisper-v3-Large. We observe that the averages and the standard deviations CER and sacreBLEU by task and language for either model are highly comparable across splits. We trust that the models have not been trained the dev and test set of Fleurs.

---

Published as a conference paper at COLM 2025
ZS-XLT
EN (1)
S4T (89)
N/A (12)
AVG (101)
S4Tv2-L
Size
B
W
B
W
B
W
W
INCL. LA
635M
87.4
88.5
82.2
79.1
55.7
52.9
79.0
75.8
EXCL. LA
588M
88.5
86.4
80.9
77.5
55.6
52.0
77.8
74.4
47M
+1.1
-2.1
-1.3
-2.6
-0.1
-0.9
-1.2
-1.4
Table 4: Ablation of Length Adaptor. We benchmark SeamlessM4Tv2-Largg with and without the Length Adaptor on SIB-Fleurs. See Table 3 for further details.
Length Adaptor. The sequence length of encoded speech typically far exceeds the length of embedded tokenized text for the same input. The mSE of Seamless SeamlessMHTv2-Largg thus ap- pends temporal convolution, 'length adaptor', as its final layer to reduce the resolution of speech frames by a factor of 8 and to better align the modalities (Seamless Commu- nication et al., 2023). 15 This begs the question whether a length adaptor is essential for pooling semantics from speech output tokens. We hence compare SeamlesM4TTV--arrge performance on SIB-Fleurs with and without the length adaptor.
Table 4 presents the inconclusive results for both variants. For English, performance im- proves on high-quality utterances but decreases on lower-quality ones. In contrast, per- formance most pronouncedly declines for low-quality utterances in ZS-XLT to supported languages (-2.6%). The ZS-XLT unsupported languages is not affected for high-quality utterances (-0.1%) and only slightly for low-quality ones (-0.9%). Three factors may explain this modest gap. First, removing the length adaptor reduces model size by 47M parameters (-7.4 pp), which were part of substantial pre-training Second, the length adaptor, as the final layer of the mSE, is most explicitly trained to embed speech into shared multilingual space, attended to by the text decoder at every layer. Finally, the length adaptor might filter noisy frames through temporal downsampling to improve the robustness of speech encodings. While a modest gap persists across setups, we cannot decisively infer whether appending length adaptor is, opposed to replacing model capacity with other layers of equal parameter size during pre-training, crucial for improved multilingual SLU. We conclude that the pre-training regime more vital for multilingual SLU capabilities of mSE than nuanced architectural design choices. We leave a more detailed investigation into the utility of length adaptors in SLU to future work.
capacit
Conclusion
We introduce Fleurs-SLU, a multilingual SLU benchmark for semantic speech classification across 102 languages and multiple-choice question answering from spoken paragraphs in 92 languages. Using Fleurs-SLU, we evaluate massively multilingual speech models in both end-to-end speech classification and a cascaded approach that combines initial speech-to- text transcription and subsequent text-based classification with LLMs.. Our findings indicate that, while cascaded systems remain the most robust option, multilingual speech encoders can achieve competitive performance when adequately pre-trained. Moreover, speech-LLMs can yield oftte-of-the-art performance on par with the best targeted SLU pipelines when appropriately aligned for multilingual spoken language instruction following, Moreover, speech-LLMs can achieve state-of-the-art performance on par or better than the best tar- geted SLU pipelines when properly aligned for multilingual spoken language instruction following. Furthermore, we observe strong correlation between strong multilingual SLU and both the robustness of multilingual ASR and the effectiveness of cross- modal speech translation to English text. This suggests that multilingual SLU and multilingual ASR can be mutually beneficial. We hope that our findings inspire future work towards developing more efficient multilingual speech encoders that are jointly pre-trained for both multilingual ASR and SLU to close the performance gap between end-to-end speech classification and cascaded approaches.
For architectural details, refer to the original paper (Communication 2023).
10

---

Published as a conference paper at COLM 2025
References
David Ifeoluwa Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba O. Alabi, Yanke Mao, Haonan Gao, and En-Shiun Ánnie Lee. STB-200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects. In Yvette Graham and Matthew Purver (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 226- 245, St. Julian's, Malta, March 2024. Association for Computational Linguistics. URL 14/.
https: / http:///aaaaaalooooooooooo www.hhoooooyo.o/22244444 wrg/2024..../.....
Al@Meta. Introducing meta llama 3, 2024. URL https://all www.mett....omo com/bblog/mettt-llam----
Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: framework for self-supervised learning of speech representations. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian parallel reading comprehension dataset in 122
Khabsa. The belebele benchmark: pa language variants. arXiv preprint ar Xiv:2230. 16884, 2023.
Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. The belebele benchmark: parallel reading comprehension dataset in 122 language variants. In Lun-We Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meefing of the Association for Computational Linguistics (Volume I: Long Papers), pp. 749-775, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653 00.16633v1/2024..cm /120044.cll-oong.. 44. URL https: http::::///aaaahhooogl/l/// org 2024. acl acl-long.444 44/.
Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. LLM2Vec: Large language models are secretly powerful text encoders. In First Conference on Language Modeling, 2024. URL forum?id=IW1PR7/VBBB
net/
Marcely Zanon Boito, Vivek lyer, Nikolaos Lagos, Laurent Besacier, and Ioan Calapodescu. mHuBERT-147: A Compact Multilingual HuBERT Model. In Interspeech 2024, 2024.
Kwanghee Choi, Ankita Pasad, Tomohiko Nakamura, Satoru Fukayama, Karen Livescu, and Shinji Watanabe. Self-supervised speech representations are more phonetic than semantic, 2024. URL https://arxixs org/ abs/2406. 08619.
Seamless Communication, Loic Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, Christopher Klaiber, Pengwei Li, Daniel Licht, Jean Maillard, Alice Rako- toarison, Kaushik Ram Sadagopan, Guillaume Wenzek, Ethan Ye, Bapi Akula, Peng-Jen Chen, Naji El Hachem, Brian Ellis, Gabriel Mejia Gonzalez, Justin Haaheim, Prangthip Hansanti, Russ Howes, Bernie Huang, Min-Jae Hwang, Hirofumi Inaguma, Somya Jain, Elahe Kalbassi, Amanda Kallet, Ilia Kulikov, Janice Lam, Daniel Li, Xutai Ma, Ruslan Mavlyutov, Benjamin Peloquin, Mohamed Ramadan, Abinesh Ramakrishnan, Anna Sun, Kevin Tran, Tuan Tran, Igor Tufanov, Vish Vogeti, Carleigh Wood, Yilin Yang, Bokai Yu, Pierre Andrews, Can Balioglu, Marta R. Costa-jusså, Onur Celebi, Maha Elbayad, Cynthia Gao, Francisco Guzmán, Justine Kao, Ann Lee, Alexandre Mourachko, Juan Pino, Sravya Popuri, Christophe Ropers, Sahiyyah Saleem, Holger Schwenk, Paden Tomasello, Changhan Wang, Jeff Wang, and Skyler Wang. Seamlessm4t: Massively multilingual & multimodal machine translation, 2023. URL httpss:/arxiv.org///
11596.
Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. Fleurs: Few-shot learning evaluation of universal representations of speech. Xiv preprint arXiw:2205.12444 2022. URL https: //arxiv. larxiv.orgg/as//2255.1 12446 12446.
11

---

Published as a conference paper at COLM 2025
Marta R. Costa-jussà, Bokai Yu, Pierre Andrews, Belen Alastruey, Necati Cihan Camgoz, Joe Chuang, Jean Maillard, Christophe Ropers, Arina Turkantenko, and Carleigh Wood. 2m-belebele: Highly multilingual speech and american sign language comprehension dataset, 2024. URL https://aaxis: www.:orgggaaaoooooe ///g//////// 0827448274.
Benedikt Ebing and Goran Glavaš. To translate or not to translate: A systematic investi- gation of translation-based cross-lingual transfer to low-resource languages. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapler of the Association for Compulational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 5325-5344, Mexico City, Mexico, June 2024. As- sociation for Computational Linguistics. doi: nttps://aclanthoo www.aannnhooooooooo/2222444 naa.l- long. 298/
298. URL
Ethnologue. How many languages in the world are unwritten? web reb.archive.org. ethnologue. com/ https:/ /web. archive http://wwbbarcchiie..ro///////// woregggeseeewt----------rrrtttee 0, 2017.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pp.
Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 00.18633/v1/2000 ht/66///1/200..eempmmm....33.3/ mmill- URL https: //acs: anthologyyyy.... anntholoy org. //tg//2211.222 emn.p-main. 552.m///................
Daniela Gerz, Pei-Hao Su, Razvan Kusztos, Avishek Mondal, Michat Lis, Eshan Singhal, Nikola Mrkšić, Tsung-Hsien Wen, and Ivan Vulić. Multilinggua and cross-lingual intent detection from spoken data, 2021. URL htttps://arriiti wttp://wwwwg////////ee//// 08524.
Lennart Keller and Goran Glavaš. Speechtaxi: On multilingual semantic speech classification. In Proceedings of the 2025 International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pp. to appear, 2025.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach, 2019. URL https://arxivv wwtp://wwogg////////a///ooosse/s 197. 116922
llya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https:///penrevieww.ee wtw.i/wwwwwetoorrmmmmimm
Bkg6RiCqY7.
Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, Alexei Baevski, Yossi Adi, Xiaohui Zhang, Wei-Ning Hsu, Alexis Conneau, and Michael Auli. Scaling speech technology to 1,000+ languages. Journal of Machine Learning Research, 25(9)):1-52, 2024. htthtt/:/ http:///mmmm..orr/pppppss/////// 11115///3311111.111 13m..
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision, 2022. URL
04356.
Fabian David Schmidt, Philipp Borchert, Ivan Vulić, and Goran Glavaš. Self-distillation for model stacking unlocks cross-lingual NLU in 200+ languages. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 6724-6743, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1 22024..iniiggs-ema //aclanthologyoorg org/2024 mendingg--emlp... 394/.
URL https:
Loic Barrault Seamless Communication, Yu-An Chung, Mariano Coria Meglioli, David Dale, Ning Dong, Mark Duppenthaler, Paul-Ambroise Duquenne, Brian Ellis, Hady Elsahar, Justin Haaheim, John Hoffman, Min-Jae Hwang, Hirofumi Inaguma, Christopher Klaiber, Ilia Kulikov, Pengwei Li, Daniel Licht, Jean Maillard, Ruslan Mavlyutov, Alice Rakotoari- son, Kaushik Ram Sadagopan, Abinesh Ramakrishnan, Tuan Tran, Guillaume Wenzek,
12

---

Published as a conference paper at COLM 2025
Yilin Yang, Ethan Ye, Ivan Evtimov, Pierre Fernandez, Cynthia Gao, Prangthip Hansanti, Elahe Kalbassi, Amanda Kallet, Artyom Kozhevnikov, Gabriel Mejia, Robin San Roman, Christophe Touret, Corinne Wong, Carleigh Wood, Bokai Yu, Pierre Andrews, Can Bali- oglu, Peng-Jen Chen, Marta R. Costa-jussà, Maha Elbayad, Hongyu Gong, Francisco Guzmán, Kevin Heffernan, Somya Jain, Justine Kao, Ann Lee, Xutai Ma, Alex Mourachko, Benjamin Peloquin, Juan Pino, Sravya Popuri, Christophe Ropers, Saftyyah Saleem, Hol- ger Schwenk, Anna Sun, Paden Tomasello, Changhan Wang, Jeff Wang, Skyler Wang, and Mary Williamson. Seamless: Multilingual expressive and streaming speech translation. 2023.
NLLB Team, Marta Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Bar- rault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. No language left behind: Scaling human-centered machine translation, 2022.
Silero Team. Silero vad: pre-trained dnterprise-grade voice activity detector (vad), number detector and language classifier. https://gitee thub. com/
2024.
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. Qwen2.5-omni technical report. ar preprint arXiv:2503.20215 2025.
A Appendix
A.1 Further Experimental Details
on a single Nvidia L40S 48GB or A100 80GB, respectively. We estimate that the total compute budget accumulates to about 2K GPU hours. Training runs on SIB-Fleurs require about 30-45 minutes, while fine-tuning LLMs for Belebele require roughly 9 hours per run. Evaluation, per checkpoint, across all languages supported by the corresponding benchmark, requires about hour due to the comprehensive setups (e.g., original text and two types of transcriptions for speech recordings).
Compute infrastructure. All experiments
Prompting Qwen 2.5 7B-Omni & Gemini 2-Flash.
For the multimodal speech-LLMs Qwen 2.5 7B-Omni and Gemini 2-Flash, we apply the following utterance pre-processing. Each audio segment is normalized to target root mean square (RMS) level of 0.07 to ensure consistent loudness across samples and reduce variabil- ity due to recording conditions. Specifically, we first normalize the loudness of individual utterances; for Delebele-Fleurs, we additionally normalize the loudness after concatenation. The models are then prompted using the task- and modality-specific templates detailed below.
13

---

Published as a conference paper at COLM 2025
SIB-Fleurs
Speech.
The utterance belongs to of the following topics. Utterance: [IN-LANGUAGE UTTERANCE]
Text.
The sentence belongs one of the following topics. Sentence: [IN-LANGUAGE SENTENCE]]
Topics:
1. Entertainment
2. Geography
3. Health
4. Politics
5. Science and Technology
6. Sports
7. Travel
Please respond with only the number of the correct answer 2,, 6, 7).
Relebele-Fleurg
Speech.
Listen to the audio passage. Based on the audio, answer the following multiple-choice question.
Passage:
[IN-LANGUAGE CONCATENATED UTTERANCES]
Text.
Given the paragraph, answer the following multiple-choice question. Paragraph: [IN-LANGUAGE PARAGRAPH]
Question:
[IN-LANGUAGE QUESTION]
Options:
1. [IN-LANGUAGE AULE 1]
2. [IN-LANGUAGE MULTIPLE-CHOICE-NNSWEE 2]
3. [IN-LANGUAGE MULTTPLE-CHOCCE-ANSWEE 3]
4. [IN-LANGUAGE MULTTPLE-CHOICEE-NNWEE 4]
Please respond with only the number of the correct answer 2, or 4).
14

---

Published as a conference paper at COLM 2025
A.2 SIB-Fleurs
Language
Language Name
Language Code
Language Name
Train
Validation
Test
Pth]
afr Latn
Afrtkaans
Amhatic
752
406
86
54
149
95
Lag
1tz Latn
Luxembourgish
Ganda
703
71
70
173
176
az1_lata
asi Beng
Ara_
ast. Lath
Modern Standard
Asturian
Assamese
Azerbaijani
379
730
69
64
176
133
177
Iva_lato
lut_Latn
ral_ Mlyn
Deva
Malayalam
Marathi
Standard Latvian
749
698
725
634
39
68
69
71
174
177
174
98
bel_Syr]
Belarusian
rkd_Cyrl
Macedonian
691
71
127
ben_Beng
Bengali
Rit_
Maltese
231
71
176
tri_Latn
Mauci
749
71
176
bul_Oyr]
Bulgarian
70
kya_
Burmese
746
71
175
cat
Cehuano
149
att
m14_ atn
Dutch
Norwegian Bokmal
724
723
51
127
123
ces_Lath
Czech
68
172
apt_
Nepali
734
70
175
ckb_Arab
Central Kurdish
70
176
nsc Latn
Northern Sotho
433
70
eyt_latn
Welsh
739
71
127
nya_latn
Nyanja
720
dan_latn
Danish
696
70
127
oci_lato
Ocetan
deu Latn
German
736
69
125
ory_Drya
Grek
Greek
750
67
168
Junufuru
Easter Panjabi
Lat-Latt
English
738
71
127
Arab
Lath
Estonian
176
Arab
Latn
Finnish
175
Latn
Lath
French
753
161
Portuguese
Fat_
Nigerian Fulfuide
166
Latn
West Central Oroano
17
Russann
733
71
173
glo_Latn
Latn
Slovak
628
71
169
Latn
sly_Latn
Slovenian
704
71
174
go3_Gujr
Gujorati
sat_
Shona
689
71
176
snd_Arab
Sindhi
749
71
177
Hehrew
70
175
500 atn
Somali
746
70
177
heva
Hindi
60
132
apa_late
Spanish
676
71
177
latn
Creatian
756
176
srp_Cyrl
Serbian
730
63
164
hun Latn
Hungarian
750
177
sav_Latn
Swedishh
696
71
168
Armenian
741
71
127
swh Latn
Swahil
745
85
154
1ho_ Jatn
Igho
757
71
177
tan Tanl
Tamil
895
71
169
aatn
Indonesian
728
69
167
tel Telu
Telugu
638
66
153
ist_
Ieelandic
381
18
23
tak Cyr1
Taak
690
69
163
ita_latn
Italian
69
175
Lat_ Latn
Taçalog
604
71
176
Jat_latn
Javanese
171
tha That
That
710
71
176
Spn_
lapanese
tur Latn
Turkish
692
87
164
Karmada
ukr Cyr]
unb Latn
Umbundu
L'rrainia
732
473
87
39
169
108
khk_Cyr1
Halh
Kazaki
Kabuverdianu
743
71
70
71
175
176
177
vis
urd_Ardb
Nol Latn
uzn Lzn_
Wolof
Northern Uzbek
Undu
Vietnamese
737
636
734
613
69
65
70
175
176
120
khr
Khimer
388
69
168
Latn
Xhosa
756
kir_0yr1
Kyngyz
729
71
177
yor Latn
Yoruba
kor Hang
Korean
869
61
181
Niass
Chirese
lin_lath
1it_latn
Laoo
Lithuanian
Lingala
lao
253
230
391
132
178
139
zutn
hant
2stllate
Zulu
Standard Malay
Chirese
Table 5: Number of samples by split and language in SIB-Fleurs.
15

---

Published as a conference paper at COLM 2025
A.3
Belebele-Fleurs
Language
Name
Samples
Language Code
Language Name
Samples
mar __vaa
Marathi
869
afr Latn
Afrikaans
309
mkd_Cyri
Macedonian
667
anh_Ethi
Amharic
782
mlt_Latn
Maltese
816
arb_Arab
Modern Standard
387
mri_Latn
Maori
877
ase_ asn_Beng
Assamese
824
mya_Mymr
Burmese
864
azj_
ben_ Beng
North Azerbaijani
Bengali
759
nid_Latn
Dutch
674
bul_Cyrl
Bulgarian
873
nob_Latn
Norwegian Bokmál
635
cat_Latn
Catalan
652
npi_Deva
Nepali
876
ceb_Latn
Cebuano
783
nso_Latn
Northern Sotho
569
ces_Latn
Czech
802
nya_Latn
Nyanja
752
ckb_Arab
Central Kurdish
842
ory_Orya
Odia
220
dan_Latn
Danish
696
pan_Guru
Eastern Panjabi
396
ell_Grek
deu_Latn
German
Greek
837
804
pbt_Arab
Southern Pashto
628
eng_Latn
English
844
pes_Arab
Western Persian
673
est_Latn
Estonian
736
pol_Latn
Polish
765
fin_Latn
Finnish
826
por_Latn
Portuguese
791
fra_Latn
French
839
ron_Latn
Romanian
815
fuv_Latn
Nigerian Fulfulde
848
rus_Cyrl
Russian
819
gat_Latn
West Central Oromo
slk_Latn
Slovak
513
guj_Gujr
Gujarati
880
slv_Latn
Slovenian
724
Latn
Hausa
heb_Hebr
Hebrew
878
sna_Latn
Shona
735
hin_Dewe
Hindi
515
snd_Arab
Sindhi
878
hrv_Latn
Croatian
896
som_Latn
Somali
874
hun_Latn
Hungarian
879
spa_Latn
Spanish
659
hye_Armi
Armenian
861
srp_Cyrl
Serbian
766
ind_Latn
ibo ibo_Latn
Igbo
Indonesian
783
838
swe_Latn
Swedish
681
ist_Latn
Ieelandic
81
swh_Latn
Swahili
780
ita_
Italian
851
tam_Taml
Tamil
714
jav_Latn
Javanese
835
tel_
Telugu
567
jpn_Jpan
Japanese
tgk_
Tajik
632
kan_Knda
Kannada
606
tgl_Latn
Tagalog
505
kaz_Cyrl
keo_
Georgian
Kazakh
870
372
tha_Thai
Thai
745
kea_Latn
Kabuverdianu
770
tur
Turkish
706
khk_Cyr1
Halh
869
ukr_Cyrl
Ukrainian
773
khm Khmr
Khmer
439
urd_Arab
Urdu
482
kir_Cyrl
Kyrgyz
811
uzn_Latn
Northern Uzbek
812
kor _aa_
Korean
535
vie_Latn
Vietnamese
847
lin_Latn
lao_Laoo
Lingala
778
346
wol_Latn
Wolof
495
lit_Latr
Lithuanian
xho_Latn
Xhosa
900
lug_Latn
Ganda
703
yor_Latn
Yoruba
652
luo_Late
Luo
512
zho_Hans
Chinese
888
lvs_Latn
Standard Latvian
555
zan_Hant
Chinese
527
mal_Mlym
Malayalam
809
zsm_Latn
Standard Malay
749
zul_Latn
Zulu
838
Table 6: Number of samples per language in Belebele-Fleurs.
16

---

Published as a conference paper at COLM 2025
A.4 Analysis of Tee Taataiiiissteee--Taaataattoo Performance
Language
Whisper-v3-Large
SeamlessMHIv2-Large
Language
Whispervå-large
Seamles:
AVG
13.4
23.6
Itz_Latn
16.0
17.6
32.9
42.5
Latn
0.7
18.3
anh_Ethi
luo_Latn
1.3
Arab
19.3
33.4
Ivs_Latn
13.3
29.0
asm Bsn_Beng
2.3
21.9
mal_Miym
9.9
25.0
Lat__atn
25.7
27.7
mar_Deva
27.2
LzinLatn
nkd_Cyr]
bel Cyrl
mat_
11.1
40.1
ben_Beng
8.1
27.2
ar mri_Latn
6.5
1.3
bos_Latn
27.6
35.6
mya_Mymir
0.4
19.4
bul_Cyri
26.4
33.2
nid_Latn
22.1
25.8
cat_Latn
32.5
nob_latt
29.7
34.7
ceb_latn
6.2
9.1
npi_Deva
10.6
94
ces ces_latn
24.5
33.1
nst_Latn
0.7
2.6
ckb_Arab
1.6
23.2
nya_Latn
0.9
19.3
cyn Latn
oci_Latn
17.6
23.5
Latn
32.6
38.9
ory_Orya
4.8
26.8
Latn
32.6
36.8
pan. pan_ curu
12.9
30.2
el1_Grak
20.5
27.7
Ara_
1.7
18.2
15.3
30.4
16.0
30.5
fin_Latn
pol_Latn
20.7
24.2
fra_Lain
34.0
36.1
por_Latn
37.5
38.9
Lat_Latn
0.2
0.9
ron rot_Latn
30.6
36.5
Latn
0.3
0.7
rus_Cyrl
26.2
30.3
gle_latn
1.5
18.6
25.1
33.6
glg_latn
slv_Latn
18.0
27.4
gu] guj_Gujr
10.7
30.2
sna
1.0
3.8
hau
0.5
1.3
and_Arab
3.6
8.7
heb heb_Hebr
16.1
31.3
0.4
18.1
spa spa_Latn
22.0
hrv_Lain
24.9
31.7
srp_Cyr1
29.8
37.3
hat_Latn
17.5
27.9
swe Latn
34.3
38.5
hye_Arm
9.0
31.2
Sah_
5.6
32.4
1bo
26
Taml
22.5
tel
Lssn
7.5
25.7
Cyrl
8.7
26.9
ita_Latn
23.8
27.6
[g] Lain
21.2
25.6
jav_Latn
4.1
23.6
tha_ Thai
13.4
23.5
jpn_Jpan
22.2
30.1
kan_
ukr_Cyr]
27.6
32.8
kan Knda
6.7
24.9
umb_latn
0.2
1.1
kat Gear
2.3
21.7
urd_Arab
15.9
25.5
kaz_Cyr1
3,8
25.0
uzn_Latn
5.0
25.5
26.5
19.2
26.4
khk_Cyr1
wol Latn
1.2
1.6
khn_
4.5
22.3
xho xhtn
0.8
7.6
kir-Cyrl
2.4
19.0
yat_Latn
0.6
14.6
kar_Hang
19.0
22.7
Hans
14.6
22.4
Lao_
7.1
zant
18.5
lin_latn
0.5
1.5
25n Latn
24.7
31.0
lit_latn
12.7
24.3
zul zut_Latn
0.6
12.4
Table 7: Per-language average sacreBLEU for utterances to their original English sentences, computed over the pooled train, dev, and test splits. Results are shown for Whisper-v3-Large and SeamlessM4Tv22Larre (cf. 54).
of Fleurs
17

---

Published as a conference paper at COLM 2025
A.5
Analysis of ASR Performance
Language
Whisper-v--Larg
SeamlessMTTv2-Larg
Language
Whispen-v3-Large
SeamlessM&Tv2--arggg
Latn
13.1gs
12.613.6
Itz_Latn
29.5m6
40.1150
anth
207 8q7
23.9
Lutn
44.9888
14
arb.Arab
7.678
8.0
luo_Latn
42.7
58.89p5
asn_Beng
99.0349
15.
Ivt_Latn
7.3ga8
8,2
ast__latn
15.5a8
19,815.
mal_Mlmm
104.3714
14.2
Lzj_Latn
7.2g4
8,3%,6
mar
24.314.
10.7a5
bel_Cyr1
11.98
6,920.
mkd_Cyr1
5.8
8.6123
ben_Beng
33.9hL
9.0%.
Latn
26.0
12 11.3
bas Lain
4.652
7,8
ari_Latn
13.4
54.7444
bul_Cyri
5.25.
9.613.7
mya_Myur
132.9%
22.2as
cat_Latn
3.346
5,3g4
nid_Latn
3.138
6.877
ceb Latn
17.477
19.011.5
nab_Latn
4,657
9.0ha2
catn
4.230.0
8.0
npi_Deva
25.6110
74 152
ckb_Arab
98.31007
13.9
95.7
66.933
eyn
15.8
14.4
nya_Latn
35.0%68
124
Lain
4.822
10.0
oci_Latn
25.9138
33.8185
deu Latn
2223
6.5g7
ory_ ory_
93.8
11.47
ell_Grek
6474
10.8
pan curu
44,2401
10.3g4
3.149
6.2g.1
pbt_Arab
36.8139
21.6533
eat_
7.367
11.0156
Ara_
8,672
7.788
Fin_Latn
3.243
10.2,3.
pol
2.338
7.935
fra_Latn
28
6.0g.9
por por_Latn
288
7.6004
fut_Latn
194.3%80
57.0g8.7
ron_Latn
3.2
6.111
Latn
35.8117
47.8ga
rus_Cyr1
2.64
6.107
gle_Latn
85.1
21.415.
slk_Latn
3.94
7.0
gagn
4.340
6.5m0
slv_latn
6.074
42
guj_Gujr
21.3171
10.3g4
saa_Latn
26.7342
327
Latn
33.8pu
51.8
sna_Arab
94.5%.4
33.5178
12.2119
14.515.
son_latn
34.9287
179
hin_Deva
11,7153
10.210.3
spa_Latt
2.256
6.3
bry hrt_Lattn
4.5122
8.5
srp_Cyr]
76.6%6
7.6120
hun_Latn
5,2
8,5
swe_Latn
3.757
9.21133
hye
18.0a24
9.1m0.
11.0
8.9
Latn
42.9t64
59.352.
tan. Tam]
13.3152
11.6132
ind_latn
4.0s8
9,2
tel_Tell
90.1919
13.314.3
ist_Latn
16.0u6
10.312.
tgk_
28.8822
8.8
ita_Latn
2.129
5.2g3
[gl 1gi_Lain
5.0
12.0pas
jav_Latn
26.3523
11.311.0
Thai
9.8116
11.4125
jpn_
6.312.1
16.410.5
tur Lat_
6.225.
7.5g0
kan_Latn
45.1847
53.9
ukr Cyrl
3.0
8.9
kan_Knda
20.1265
10.992
Lnb_latn
156.7982
55.571.
kat_Geor
19.8339
7.10
urd_Arab
9.4g9
kaz_Cyrl
8.9
9.7
uzn_Lain
25.0
8.411.4
kea kea_Latn
35.8172
38.113.
Vit_Latn
4.756
6.7.3
<hk_Cyri
37.6168
14.622.9
wot_Latn
140 140.62088
47.9k2
khn Khmr
144.0.1.7
29.811.2
kht_
48 48.1gn1
32.3
kir_Cyrl
28.311
8.8
yor_latn
49.3182
33.2
kar Har_Hang
7.4on
10.3
zho_Hans
16.2
19.0134
lao
109.745.
30.311.7
Hant
30.641.
1188
lin_
22.0122
58.7
Lat_
3.4g0
8.5
lit_latn
8.3106
11.9
zut_Latn
42.6%
18.4,5.
Table 8: Per-language average CER for transcribing Fleurs utterances to their original sentences, computed over the pooled train, dev, and test splits. Results are shown for Whisper-v3-Large and SeamlessMATTv-LLar are transcribed using the closest manually selected supported language.
(cf. $4). Utterances in unsupported languages
18

---

Published as a conference paper at COLM 2025
A.6 Silent Fleurs Examples
Language
Split
Count
nb_no
train
497
e5_
train
490
cy_gb
394
sd_in
train
307
train
15
ckb_1q
ny_m
lest
train
nso_za
lest
dev
ur_pk
ps_af
train
fa_ir
train
so
train
ceb_ph
train
1g_ug
train
kea_cv
train
bg_bg
train
bn_in
cy cy_sb
test
ff_sn
train
hr_hr
train
hy_an
train
ur_pk
dev
train
da_dk
test
da_dk
train
ff sn_sn
dev
train
he il
he_il
test
ig_ng
train
kan_ke
train
kn_in
kn_in
test
train
mi_nz
dev
train
ms_my
train
or or_in
train
dev
sk_sk
test
lest
ta_in
te_in
lest
train
unb_ao
train
Table 9: Number of silent examples in Fleurs by language and split.
19


Reviews:
[{"review_id": "EgY2djXuVo", "reviewer": "Reviewer_KQ4z", "summary": "This paper proposes a multilingual benchmark for topic classification and multiple choice QA.\nThe dataset is derived from aligning existing datasets Flores, Fleurs, SIB-200 and Belebele.\nCascade and end to end models are compared on the proposed dataset and models are further analyzed on separate tasks (ASR, MT) in order to understand performance on the original tasks.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* massive multilingual benchmark datasets are critical for the community\n* the experimentation is thorough, the results convincing and the paper is well written", "reasons_to_reject": "I don't see a specific reason to reject", "questions_to_authors": "* Abstract: the motivation is languages without writing system but all experiments compare cascade and end-to-end systems. Since cascade systems rely on a writing system, the motivation is not fully realized.\n* 99: \"Concurrent Work. Costa-jussà et al. (2024)\" I am not aware of what qualifies as concurrent work at COLM (and the reviewing guidelines do not specify this) but noting that https://arxiv.org/abs/2412.08274 shows a first submission on 11 Dec 2024.\n* 147-148: \"We train mSE on the utterances and CS on the transcriptions of the English training set.\" It would be interesting to see what happens in an \"unconstrained\" setting where both mSE and CS have access to additional data.\n* 37: \"wav2vec-BERT objective\" → wav2vec or wav2vec 2.0. w2v-BERT refers to a different paper: https://ieeexplore.ieee.org/abstract/document/9688253. It would be good to go through the paper and fix all instances (note that for SeamlessM4T v2 it should be \"w2v-BERT 2.0\") \n* Table 1: it was unclear to me what the numbers in parenthesis represent", "extra_scores": {}}, {"review_id": "I7lAQsWmdp", "reviewer": "Reviewer_TY1K", "summary": "This paper introduces Fluers-SLU benchmark for massive multilingual spoken language understanding, The benchmark consists of 692 hours of speech for topical classification in 102 languages (SIB-Fleurs) and 944 hours of speech for multi-choice question answering task in 92 languages (Belebele-Fleurs). The authors did extensive experiments for benchmarking models on the datasets, including end-to-end multilingual speech encoders (mSE) and cascaded systems (CS). The results show that the cascaded systems remain the most robust option, however the mSE can achieve competitive performance with adequate pre-training. The authors also find correlation between ASR and speech-to-English translation and SLU.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The benchmark increased the coverage of languages to 102 for topical classification adn 92 for multi-choice question answering.\n- Comprehensive experiments have been conducted and results are very insightful.\n- The code will be released, which can help the research community to verify and research on top of the benchmark.", "reasons_to_reject": "NA", "questions_to_authors": "For the length adaptor section, it may not be conclusive because the SeamlessM4Tv2 model is pre-trained on massive amount of data. I would suggest the authors choose a model baseline and train the model from scratch with and without the length adaptor to verify if it is helpful to model performance.", "extra_scores": {}}, {"review_id": "HKMKHdspJU", "reviewer": "Reviewer_t3tg", "summary": "In this work, a multilingual spoken language understanding (SLU) benchmark, Fleurs-SLU, is proposed. It is derived from Fleurs, Flores and its derivatives (SIB-200 and Belebele) spanning over 100 languages. The SLU tasks include speech classification and question answering. \nDetailed evaluation is also presented for both end-to-end and cascaded systems. It would be a valuable benchmark for the SLU tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- A multilingual SLU benchmark is provided for SLU tasks\n- Extensive experiments are conducted on the top of the proposed benchmark.", "reasons_to_reject": "- The description of experiments are not clear and hard to follow.", "questions_to_authors": "- As described in section 4.1, the speech encoder and cascaded systems are trained with English training set. I assume it is English data from Fleurs. However, I also notice Table 3 presents both MMS-1B-FLEURS and MMS-1B-ALL. It is quite confusion and no clear explanation about \"ALL\" training data.\n- \"zero-shot cross-lingual transfer (ZS-XLT)\" mentioned in section 4.2 is not clear. As mentioned in section 4.1, the model is trained with English data only. So the source language is English, is it correct?\n- In Table 3, there are 7 languages are not supported by Whisper and SeamlessM4T, how do we generate the corresponding transcription with Whisper or SeamlessM4T models?", "extra_scores": {}}, {"review_id": "1yyEATlKNw", "reviewer": "Reviewer_VcwH", "summary": "This paper proposes two multilingual benchmark datasets for spoken language understanding (SLU): SIB-Fleurs and Belebele-Fleurs. Based on these datasets, it evaluates several end-to-end and cascaded models. Results shows correlations between SLU performance and ASR/speech translation performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed benchmark datasets are novel and carefully created. They are valuable assets to the community.", "reasons_to_reject": "- This paper conducted many experiments and provided many data points. But no conclusive or convincing insights came out. (1) The tested methods were all based on pooled embedding + a classification layer. This was a weak method comparing with contextual generative models nowadays. (2) Many conclusions were drawn from comparing two independent models, such as Whisper vs. SeamlessM4T. Without variable-controlled ablation study, it's not convincing to conclude that a certain training receipt difference was causing the performance difference. (3) Correlation was not causation. We could not conclude that we should jointly train ASR and SLU, because ASR and SLU were mutually beneficial, because ASR scores correlated with SLU scores.", "questions_to_authors": "- What does \"ca.\" mean in Line 101 and 105?\n- Line 167: What are the multilingual datasets? Why do we need to present this method as it never works?\n- Figure 1: Why does Whisper wins more but in all language groups it gets higher CER? If it often wins by a small margin but losses by a large margin, then win ratio is not a good metric.", "extra_scores": {}}]

Author rebuttal:

Meta-review: Paper summary:

This paper presents a dataset for spoken language understanding (SLU) with a focus on low-resource languages and comparing the performance of cascaded (ASR + NLU) systems vs. end-to-end systems. The dataset is comprised of two parts: (1) 692 hours of speech across 102 languages annotated at the utterance level with topic classification labels and (2) 944 hours of speech across 94 languages accompanied by multiple-choice questions that assess a model's listening comprehension.

Extensive experiments are performed comparing cascaded vs. E2E systems. The paper finds that in general cascaded systems are still more robust than E2E systems, but this performance depends on how accurate the component systems are (ASR and MT), and with adequate pre-training the E2E systems can often be competitive to the cascaded systems.

Paper significance:

The paper curates a new medium-sized (~1600 hour) multilingual (~100 languages) dataset that is annotated for spoken language understanding. Such datasets with this degree of language coverage and annotation are hard to come by and very valuable for continuing work in multilingual SLU, so this contribution will be extremely valuable for the research community.

Pros (as raised by reviewers):

All reviewers noted that the dataset introduced by this paper will be very useful for the multilingual SLU research community (KQ4z, TY1K, t3tg, VcwH)

The experiments presented by the paper comparing E2E vs. cascaded SLU systems are very thorough and informative (KQ4z, TY1K, t3tg)

Cons (as raised by reviewers):

Two reviewers indicated that there were no reasons to reject the paper (KQ4z, TY1K)

One reviewer (t3tg) found the experimental description hard to follow, but still noted positively that the experiments were very thorough.

One reviewer (VcwH) found the introduced dataset to be valuable, but questioned whether the experiments led to any meaningful insights

Cons address by discussion between authors and reviewers:

Regarding the clarity issues raised by t3tg, the authors replied to the reviewer clarifying their questions, and the reviewer acknowledged their response and did not follow up with additional questions.

Regarding the significance of the experiments (VcwH), in my view the authors adequately made the case for their experiments (e.g. being the first to rigorously examine SLU on 100+ languages), and the reviewer noted that they believe the paper should be accepted on the basis of the dataset.

[Automatically added comment] At least one review was discounted during the decision process due to quality]
Decision: Accept