OWLS: Scaling Laws for Multilingual Speech Recognition and Translation Models
Authors: William Chen
Jinchuan Tian
Yifan Peng
Brian Yan
Chao-Han Huck Yang
Shinji Watanabe
Venue: ICML 2025 poster (2025)
Keywords: Automatic Speech Recognition
Scaling Laws
Speech Translation
Multilingual
Primary area: applications->language_speech_and_dialog
TL;DR: Scaling laws for multilingual ASR and ST models. Largest model is 18B parameters trained on 360K hours of ASR/ST data.

Abstract: Neural scaling laws offer valuable insights for designing robust sequence processing architectures. While these laws have been extensively characterized in other modalities, their behavior in speech remains comparatively underexplored. In this work, we introduce OWLS, an open-access, reproducible suite of multilingual speech recognition and translation models spanning 0.25B to 18B parameters, with the 18B version being the largest speech model, to the best of our knowledge. OWLS leverages up to 360K hours of public speech data across 150 languages, enabling a systematic investigation into how data, model, and compute scaling each influence performance in multilingual speech tasks. We use OWLS to derive neural scaling laws, showing how final performance can be reliably predicted when scaling. Scaling to larger models can improve ASR performance across the board, in both low and high resource languages, improving the accessibility of speech technologies. Finally, we show how OWLS can be used to power new research directions by discovering emergent abilities in large-scale speech models. Model checkpoints will be released on https://huggingface.co/collections/espnet/owls-scaling-laws-for-speech-recognition-and-translation-67ab7f991c194065f057ce8d for future studies.

OWLS: Scaling Laws for Multilingual Speech Recognition and Translation Models
William Chen
Jinchuan Tian
Yifan Peng
Brian Yan
Chao-Han Huck Yang? Shinji Watanahe
Abstract
Wriseer olsS
Total Training Hours
0ls9 v3.
10
"OWLS"" Canary
Neural scaling laws offer valuable insights for designing robust sequence processing architec- tures. While these laws have been extensively characterized other modalities, their behavior in speech remains comparatively underexplored. In this work, we introduce OWL.S, an open-access, reproducible suite of multilingual speech recog- nition and translation models spanning 0.25B to 18B parameters, with the 18B version being the largest speech model, to the best of our knowl- edge. OWLS leverages up to 360K hours of pub- lic speech data across 150 languages, enabling systematic investigation into how data, model, and compute scaling each influence performance in multilingual speech tasks. We use OWLS to derive neural scaling laws, showing how final per- formance can be reliably predicted when scaling. Scaling to larger models can improve ASR per- formance across the board, in both low and high resource languages, improving the accessibility of speech technologies. Finally, we show how OWLS can be used to power research direc- tions by discovering emergent abilities in large- scale speech models. Model checkpoints will be released on huggingface for future studies.
10*
101
10a Trainable Params. of Muitilingual Speech Models
10a
10
Figure 1. Comparison of previous open models and OWLS
models (red) by parameter count and training dataset size. Whisper (Radford al., 2023) and Canary (Puvvada et al., 2024) are trained on undisclosed data, while OWSM (Peng et al., 2023b) and the presented OWLS use public data.
2021), Transformer-basee (Vaswani et al., 2017) models have dominated the fields of Automatic Speech Recognition (ASR) and Speech Translation (ST).
The state-of-the-ar (SOTA) in ASR/ST has now progressed to not only scaling terms of model and data size, but also tasks and languages. In recent years, there has been signifi- cant interest in developing massively multilingual models that perform ASR/ST for hundreds, if not thousands, of diverse spoken languages (Chen et al., 2023b; Pratap et al., 2023; Babu et al., 2022; Yu et al., 2023; Chen et al., 2024; Zhang et al., 2023), with the goal of having single model that can universally convert multilingual speech into text.
However, the architecture of these massively multilingual models is complex, and their scaling properties pose signifi- cant challenges for both experimental designs in advancing speech science. This challenge is further exacerbated by the multi-modal nature of spoken language systems, which must handle the complexities of both multilingual text and speech. Prior art on the scaling laws neural models devi- ates significantly from the goal of SOTA universal systems. The majority study single-task and single-modality systems (Biderman et al., 2023; Ghorbani et al., 2022; Zheng et al., 2022), while multilingual work concentrates only on set- tings where a few languages are supported (Fernandes et al., 2023; Yang et al., 2023; Li et al., 2021).
1. Introduction
Neural acoustic models have shown robust performance in processing human speech information and have demon- strated remarkable capabilities in spoken language tasks (Radford al., 2023; Peng al., 2023b; Barrault et al., 2023a). Powered by large-scale training (Baevski et al., 2020; Zhang et al., 2023; Chen et al., 2024; 2022; Li et al.,
'Carnegie Mellon University 'NVIDIA. Correspondence to: William Chen <williamchen& wwm...dd.... Chao- Han Huck Yang <hucky @ @nvidia.com>>, Shinji Watanabe <shinjiw @ @rcc..org>.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
To address this, we present OWLS, a Open Whisper-style

---

OWLs: Scaling Laws for Multilingual Speech Recognition and Translation Models
Large-scale neural model Suite for Speech Recognition and Translation. OWLS contains 13 fully transparent' speech foundation models for ASR/ST, pre-trained on up to 360K hours of multilingual data across 150 languages, with each model ranging from 0.25B to 18B parameters (Figure 1). We experiment with scaling in terms of both model and data size, and analyze the change in downstream ASR/ST per- formance. Through these investigations, we derive a neural scaling law to predict the change in model performance for each task and language. We also evaluate test-time capa- bilities of large-scale ASR/ST models, studying how new abilities emerge at scale and showing how speech model scaling can be benefits to new languages with in-context learning. Our contributions are summarized as follows:
a are thus the empirically learned variables of the power law. Varying the value of x: allows a practitioner to estimate the scaling behavior in different settings. When x 22, for example, the power law models the data-rich (T + oc) and compute-rich (B - oo) setting. Previous work (Gu et al., 2023) in language model re-scoring has shown that the Word Error Rate (WER) can also be modeled as a power law function of z. We can thus modify Equation as follows:
WER(2) = Bgx0
We empirically show that this power law can also generalize to the multi-modal task of ASR (Figures 3 and 9), allowing true downstream performance to be easily predicted when x = NN B. Furthermore, we also observe that it can be applied to ST (via BLEU(z) = f=,2%=) and thus extends our findings to more tasks (Figures and 6).
We open- spen-source OWLS, a collection of 13 Whisper- style ASR/ST models trained on up to 360K hours of publicly available data and 150 languages. We will also release all model training code, training logs, and intermediate checkpoints.
2.2. Scaling Laws for text and vision
The impact of scaling neural models has been thoroughly studied in the domains of text and vision. Early studies in scaling text models focused on supervised tasks such as ma- chine translation (MT) (Gordon et al., 2021; Ghorbani et al., 2022). The most relevant work to ours is from Fernandes et al. (2023), who devised scaling laws for multilingual MT models. However, these are only trained on two translation tasks/languages. In comparison, our work evaluates on over 100 languages and tasks.
We train and release an OWLS model with 18B total parameters, which makes it the largest of all publicly known ASR/ST models and nearly double that of prior work (Zheng et al., 2022).
We systemically evaluate the effects of model and data scaling on ASR and ST, developing the first set of neural scaling laws for these tasks. We not only mea- sure the usefulness of model scaling, but also identify failure cases that it is not able to overcome.
Later studies focused instead on scaling self-supervised LLMs (Biderman et al., 2023; Tay et al., 2023; Kaplan et al., 2020). Kaplan et al. (2020) empirically showed that language modeling obeys power law w.r.t B. Biderman et al. (2023) released suite of open-access LLMs, and showed how they can be used to understand scaling behaviors on downstream tasks. Our research can be viewed as a combination of these works, albeit applied to speech: we introduce a suite of open-access large ASR/ST models and also derive scaling laws for downstream tasks.
We evaluate the test-time capabilities of frozen large- scale speech foundation models via in-context learning, and discover several new emergent abilities present in large models that are absent in smaller ones.
N,T, and
2. Background and Related Work
2.1. Neural Scaling Laws
Previous research has shown that the performance of Transformer-based (Vaswani et al., 2017) models at scale can be empirically predicted with three fundamental vari- ables: the model size N, the training data size T, and the compute budget (Hestness et al., 2017; Rosenfeld al., 2020; Kaplan al., 2020; Hernandez et al., 2021; Ghorbani et al., 2022; Fernandes et al., 2023). This can be summa- rized by modeling the change in the cross-entropy loss L,
In vision, there is existing literature the scalability of vision encoders on image classification tasks (Zhai et al., 2022). However, these tasks do not require multi-modal understanding. Our work is thus most similar to those on cexl-t-imaae/immeeot--eex tasks (Henighan et al., 2020). However, we focus on the speech modality while also con- sidering multi-tasking and zero-shot behaviors.
when varying each variable independently:
2.3. Multilingual Processing and Scaling in Speech Multilingual ASR is the concept of having a single model that can recognize speech many languages (Watanabe
[(xx= = Togo + Bg.
where & E (N, B), I(x) is the reducible loss that obeys the power-scaling law, and Lox is irreducible loss. B and
"W assume that the model parameters are equally distributed between the encoder and decoder for encoder-decode architec- tures. Otherwise, the law can also be formulated as bivariate function w.r.t. to the encoder parameters N. and decoder parame- ters Na (Fernandes et al., 2023; Ghorbani et al., 2022)
We follow the definition of "transpareney" (Dabbish et al., 2012) on open-source, open-data, and open transcripts.
2

---

OWLs: Scaling Laws for Multilingual Speech Recognition and Translation Models
100
Hungarian
(Rt: 0.37)
0.5B (Rt: 0.52)
Javanese
1.0B (R": 0.50)
80
2:0B (Rt: 0.49)
4:08 (R!: 0.48)
9:0B (R": 0.50)
Icelandic
(R": 0.51)
Korean
60
WERICER
Bengali
Oriya
French
Chinese
Filipino
Russian
English
Galician
Swahili
Italian
Spanish
German
Japanese
10'
10
10"
10
10"
Training Hours
Figure 2. The effect of scaling model size on the 102 FLEURS languages, plotted as WER (or CER) versus available training data. Although WER/CER generally decreases with more training data, the relationship is only moderately correlated, as indicated by the R values in the legend. Model performance is also influenced by domain alignment and orthographic transparency: for instance, more transparent languages (c.g., Spamish, Italian) often achieve lower error rates with less data than opaque languages (c.g., English, French),
et al., 2017a). While initial investigations focused on only combining a few languages together (Conneau et al., 2021), I), modern multilingual ASR models are capable of handling hundreds, if not thousands, languages (Zhang et al., 2023; Pratap et al., 2023; Chen et al., 2024; Radford et al., 2023; Li et al., 2022). Recent SOTA multilingual speech models have begun supporting tasks in addition to ASR. Joint language prediction and speech recognition is now common method of developing multilingual ASR models (Chen et al., 2023b; Radford et al., 2023). Whisper-style models (Radford et al., 2023; Peng et al., 2023b) use system of language and task prompts also perform language identification, speech translation, and timestamp prediction. On the other hand, the Seamless family (Barrault et al., 2023a;b) leverages task decomposition to perform ASR within a speech-to-speech translation framework. Our work focuses on Whisper-style models, as their use of task prompts allow us to easily eval- uate the effects of scale on zero/few-shot performance.
3. The OWL Suite
3.1. Dataset
We largely rely on the OWSM v3.2 (Tian et al., 2024) dataset for our experiments. It consists of 180K hours of ASR/ST data gathered across 25 public corpora, covering 150 unique languages. For our experiments on scaling up the training data size beyond 180K hours, we also include an additional 180K hours from a cleaned subset of YODAS (Li al., 2023) from Peng et al. (2025), for total of 360K hours. Note that this YODAS data is only used to train two models (OWLS 1B 360K and OWLS 18B v2). More details about the dataset can be found in Section A in the Appendix.
3.2. Training Details
All OWLS models follow a Transformer (Vaswani et al., 2017) encoder-decoder architecture trained using hybrid CTC/attention (Graves et al., 2006; Watanabe et al., 2017b) loss. The inputs to the Transformer are 80-dimension log- Mel filterbanks extracted with a frame shift of 10ms, which are then down-sampled times by a stack of convolution layers. The prediction targets are text tokens with a 50K sub- word vocabulary (Kudo, 2018), We also use Whisper-style training (Radford et al., 2023): all utterances are padded to 30 seconds, and the model is jointly trained to perform language identification, ASR, ST, and timestamp prediction.
There have been few studies on neural scaling laws for speech. Droppo & Elibol (2021) and Cuervo & Marxer (2024) devised neural scaling laws for self-superviser acous- tics models and speech language models, respectively. How- ever, their evaluations are limited to simple probes due to the text-less nature of these models, and cannot be easily applied to typical speech tasks. Zheng et al. (2022) and Li et al. (2021) experimented with scaling monolingual and multilingual models respectively to 1OB parameters, but the models are trained only on internal data and remain unre- leased. Neither works attempt to devise empirical scaling laws nor study the enhanced capabilities of larger models.
We conduct our experiments with the ESPNet (Watanabe et al., 2018) toolkit. Since our goal is a systematic study of large-scale speech models, we take experimental ap- proach similar to Biderman et al. (2023); we design
3

---

OWLs: Scaling Laws for Multilingual Speech Recognition and Translation Models
-10
R 0.10, 5.7e+02
Rasesin -0 6- 44
Japanese, 0075 4.200 p tenda
Chnese R=008.c= 2-236405
German, -.033
French, -000044000002 7.600-1
Spanish, 0.000.00000344 0.0440 1.76-00
Dutch C [ 1en04
-10
WERCOL
Delaruszian, O c c000020 p2 1.te:01
Ialian, R 0.85 g0.1666 2255005
Bergali
Jevanos2, -0788, a 04,, 177+05
Parslan, R = a B
87 Portuguesse, R8 0.74, a -0.33, 20233,g22660000 2.0e00
Tomm, RP 0.70 o m-335 B 3.86-044
0.34, 5.5
Turlish, 0.000 < ---0.8 1.7e100
-10
Model Size
Figure 3. The effect of model scaling on WER/CER on FLEURS. Languages are color-coded by the amount of training data. For readability, we only show the top-20 languages (by data amount) our training corpus. We find that model scaling consistently predictive of downstream WER/CER across languages. Scaling curves for other languages can be found in Figure 14 in the Appendix.
model parameters equally between both. More details about each model can be found in Appendix B.
Average
LS clel-
LS test-other
Multilingual ASR: To evaluate the multilingual perfor- mance of the OWLS models, use the 102-language FLEURS test set (Conneau et al., 2022). Figures 2 and 3 show WER for different languages as function of per- language training data size and model size respectively, and measure their correlation with WER using the co-efficient of determination, R². We find that model scaling consis- tently improves WER/CER of each language across all data levels (Figure
Tedlium
AMI
GigaSpeech
SPGISpeech
WER
VoxPopull
16
10
2). However, the amount of data used for any given language is only somewhat predictive of its WER/CER (R² ~ 0.5, Figure 2). In other words, we cannot easily fit a language-agnostic data scaling law. This of course, is expected. Some languages are naturally more difficult to model for ASR than others (i.e Spanish vs Chinese) (Taguchi & Chiang, 2024). so they will require more train- ing data. On the other hand, canguage-sppecifp model size scaling laws are highly predictive of WER/CER (R²2 = 0.95, Figure 3). Finally, we want to highlight the significant im- provement on WER in low-resource languages when scaling to larger model sizes. The average WER on the 50 lowest- resource languages (less than 35 hours of training data) in our dataset decreases from 59 to 45 when model size increases from an already large size of 1B 9B. Larger models can improve ASR performance across the board, in both low and high resource languages.
Model Parameters
Figure 4. WERs on multi-domain English ASR by model size.
experiments to prioritize training stability and controllabil- ity over squeezing out the best possible performance. We therefore use the exact same hyper-parumeters for mod- els, varying only the data or model size fit the appropriate scaling experiment. More details on training can be found in Appendix B.
4. Pre-Training Experiments
4.1. Scaling Model Size
We experiment with scaling the model parameters of the OWLS models from 0.25B to 18B parameters, roughly dou- bling the total model parameters with each iteration. This leads to a total of 7 model sizes (0.25B, 0.50B, IB, 2B, 4B, 9B, 18B). For each model size scale the depth and width of the encoder and decoder in tandem, while allocating the
Multi-domain ASR (English): We test robustness of OWLS models to different data domains by evaluating on 6 standard ASR benchmarks: AMI (Carletta, 2007), Lib- riSpeech (Panayotov al., 2015), SPGISpeech (O' Neill et al., 2021), Tedlium (Hernandez et al., 2018), VoxPopuli (Wang al., 2021b), and GigaSpeech (Chen et al., 2021).

---

OWLs: Scaling Laws for Multilingual Speech Recognition and Translation Models
Englieh
Russann
Japunwee
Chirsse
Carman
Trinnch
Span an
Catalann
Dutcr
WERVEER
Detanusskan
Torcan
lallin
Bangalt
Persian
Polis
Portugaesce
Tannl
Cantonese
Taadaah
10
16'
Thiinin Freass The Langsage
Traiiing Hours Per Language
Figure 5. The evolution of FLEURS WER/CER for the top 20 languages by data size, language and given a fixed model capacity. Left: impact on WER/CER when scaling from 11K to 180K total hours, when all data from the same distribution. Righr: impact WER/CER from adding in data from new domain/distribution (YODAS), when further scaling from 180K to 360K total hours. Plots for more languages can be found in Figure 15 in the Appendix.
training data is added for each
Figure 4 shows the results of these experiments. ASR im- proves significantly with scale across all domains, with the average WER almost halving from 12.1 to 6.3 when scaling from 0.25B to 9B parameters. The effects of scale are ap- parent even when going beyond the typical maximum ASR model size of 2B parameters, with relative reduction in WER of 11 1.3%% when scaling from 2B to 9B.
Gomon R1-1 2-11
e 2=621-3-02 3-56
n-d 10,0110000 p-230
Oollan R2- 3.70
Translation: We study the effects of parameter scaling on English to X and X to English translation. The results are shown in Figures 7 and 6 respectively. We observe that scaling the model parameters leads to significant improve- ments in BLEU scores for all languages. This observation holds true even for high- nigh-resource language pairs. For high- resource English to German, scaling from already large IB model to a 9B variant nearly doubles the BLEU score from 16.6 to 28.9 (Figure 7). Figure 7 also shows that some models are too small to functionally perform ST: the 0.25B OWLS model is unable to produce intelligible output (BLEU- < 5) for 9 of the 15 English X translation pairs. In comparison, the 9B OWLS model functions reasonably well (BLEU 15) on 12 of the 15 pairs
stors
Figure 6. BLEU scores on to English speech translation.
4.2. Scaling Data Size
We evaluate how varying the amount of data used to train OWLS model can affect downstream performance. To do so, first create smaller training splits by uniformly downsampling the 180K hour base training set by 50%, 25%, 12.5%, and 6.25%. We also experiment with using a larger amount of data by collecting an additional 180K hours from YODAS (Section 3.1). For these experiments, we fix the model size at 1B parameters. This leads a total of different models trained on 360K, 180K, 90K, 45K, 22.5K, and 1 11.55K hours of speech respectively. We use evaluation protocol similar to the one in Section 4.1, benchmarking the model on Multilingual ASR and ST.
However, there are also limitations of model scaling. Figure shows the effects of scaling on X to English ST. While 4 out of the language pairs show improvement trends similar to Figure the BLEU scores for Japanese do not in- crease significantly. Importantly, there only hour total of Japanese English ST to English data in the OWLS training corpus. We can thus conclude the following: while parum- eter scaling can significantly improve ST performance, it cannot overcome cases where there is inherently insufficient amounts of data to learn the task.
Multilingual ASR: Figure (left) shows the effect of data scaling on the WER each language from I 11.55 to 180K hours, given a fixed model capacity. While a training set generally leads to better performance for most languages, we also observe degradations in WER/CER for some, likely due interference from similar languages (e.g. Chinese in- terference for Cantonese). Figure (right) shows the impact of adding in data from a new domain/distribution (YODAS) when scaling from 180K hours to 360K hours. With addition of 180K hours of high quality data from YODAS,
5

---

OWLs: Scaling Laws for Multilingual Speech Recognition and Translation Models
Somon 65- 9-0.0
12
U.10 035
100
Chisse 21-072
15
V- C20 0058
Eakoren
Vergaken
80
C.2 3 0.233
Imbbi 0:30
Average WER
R0-00
0.35
Lanoan, D017
60
Staeraas 1
o- p ber
Jiuanow 10 0.88
18.08 (R 0.63)
0-2.2-0- 0-155
40
9:08 (R2: 0.81)
Model Paarameeerss
0.1
4.08 (R 0.81)
2.08 (R 0.82)
Figure 7. BLEU scores on English to X speech translation.
1.08 (R 0.80)
20
0.58 (R": 0.82)
0 258 (R 0.85)
Att 6 28)
Gatalann (t 0.601
10
10"
10"
10"
Esterian (R::
FLOPS
BUEU S000
(Rt
10
(R
Latvian (R 0.34)
10
Persian (P) 0.42)
Morael un (R) 3.621
Figure 9. Average multilingual WER for each model size throughout different stages training.
Siowerton (2) 0.63)
ge* 57)
Turlesh (0000
4.3. Scaling Compute
Training Language
Weish (F: 0.87)
25.0
Calalan 0.63)
Another method of evaluating the effects of scaling is by predicting the test WER as function of the FLOPS used for training. This allows models to be evaluated in the compute- equivalent setting and considers the fact that larger models will take longer to train. To model this relationship, we test OWLS models of various sizes on FLEURS with various intermediate checkpoints. Specifically, we perform infer- ence with each model after 15K, 30K, 60K, 120K, 240K, 480K, and 675K training steps. The training TFLOPS of each checkpoint is then calculated by profiling each model size for few steps and scaling the result to 15K-675K steps. We only evaluate on English and two other randomly chosen languages (Spanish and Turkish) to reduce computing costs. Figure 9 shows the evolution of average WER from the languages for each model size as training progresses. We find that for fixed parameter size, the WER of the final checkpoint can be reliably predicted as a function of the training compute (R²² = 0.82). This means that one can rea- sonably predict the final WER of the model given the WERs of initial checkpoints. As expected, smaller models are more compute efficient, being able to reach much lower WER with lower TFLOPS spent.
22.6
Freech (e 6.7
Gommar (R 0.57)
Store
20.0
Sperial: (R) 0.50)
17.5
15.0
125
10.0
Training Hours Langusge
Figure 8. BLEU scores on EN X (top) and X to EN (bottom) ST with different dataset sizes.
many languages with saturated performance when scaling from 22K to 1 180K hours (Korean, Polish, Dutch) experi- ence large improvements in WER/CER. Our findings can thus be summarized as the following: data scaling without additional diversity leads to quickly saturated performance.
Similar our findings in Figure 2, find that ST data quantity is only loosely correlated with down- stream performance (R² ~ 0.55). The top and bottom portions of Figure 8 show the change in BLEU score as the training data size increases for English to and X to English, respectively. While BLEU score is positively cor- related with larger dataset size for most translation pairs, we also observe significant degradations in English Ger- man (Figure 8). We hypothesize that this may be due to the 1B model's limited capacity as data size increases, but leave more concrete analyses to future work. Finally, we note that we exclude results from the 360K model in this analysis, since the additional 180K hours from YODAS did not contain any ST data.
Translation:
4.4. Further Scaling
We combine our findings in model and data scaling to make preliminary exploration in further scaling OWLS models. We scale 18B parameter OWLS model to 360K hours of data, which we designate as OWLS 18B v2. We compare this model with other OWLS models and SOTA ASR models
6

---

OWLs: Scaling Laws for Multilingual Speech Recognition and Translation Models
Table WER/CER OWLS models SOTA ASR models on various benchmarks: FLEURS, AISHELL (zh CN), LibriSpeech test-clean (eng), ReazonSpeech (jnn,, and Ksponspeech (kor). Canary is only trained on 4 European languages. OWLS models perform comparably, not better than, models like Whisper Large that are trained on much more data (14x more case of Whisper).
Model
FLEURS (eng)
LS clean
FLEURS (jpn)
Reazon
FLEURS (zh-CN)
AISHELL
Kspon
Canary
7.1
1.5
Whisper v3
4.1
2.0
4.9
15.1
7.7
5.1
13.4
Qwen2Audio
9.4
1.6
20.1
50.0
7.5
8.7
53.3
Sense Voice S
10.3
3.2
13.1
37.1
9.6
3.0
24.5
Sense Voice L
2.6
2.1
Seamless M
8.3
4.2
15.9
34.9
15.7
9.6
27.3
Seamless L
7.3
3.7
17.6
36.6
17.0
8.7
32.4
OWLS IB
9.7
2.3
9.2
7.8
13.8
6.2
17.5
OWLS 9B
8.5
1.9
7.7
7.3
11.6
4.8
15.8
OWLS 18B
7.7
2.0
7.2
7.5
10.6
4.8
15.2
OWLS 18B v2
6.8
2.0
6.7
7.2
10.1
4.8
15.0
larger models clearly perform better than smaller models at test-time (4.5 WER for 9B vs 8.3 WER for 0.25B). This shows the viability large-scale ASR models in production settings. Additional multilingual results can found in Appendix C.
Table 2. WER on Librispeech test-other when using greedy scarch (left) balancing test-time compute budget (right). We exclude 0.5B and IB OWLS models since there is no beam size that consumes 400-50 TFLOPS.
Params.
Beam Size
TFLOPS
WER
0.25B
1 / 10
1..7
9.6 8.3
5.2. Emergent Ability
2B
1/4
11.1
5.2 3.77
4B
1/2
42.3
4.6/4.5
L.LMs are shown exhibit drastically improved perfor- mance on certain tasks the model size increases, even if the training data remains unchanged (Wei al., 2022). In this section, study if large-scale ASR models can also exhibit these "emergent abilitiees"" We focus three abil- ities that we newly discover: orthographic understanding, code-switching, and mondegreens. Results for contextual biasing, the first known example of emergent abilities in ASR models (to our knowledge), are found Appendix G.
9B
1
47.7
4.5
(Radford et al., 2023; Puvvada et al., 2024; Barrault et al.,
2023b; Chu et al., 2024; An et al., 2024) in Table 1. OWL. 18B v2 obtains the best or second best result on the test sets, performing comparably if not better than models trained on more data, like Whisper and Qwen2Audio.
5. Test-Time Experiments
Orthographic transparency describes the relationship between the phonetics (sounds) of a language and its written form. Opaque languages (e.g. Chinese and Japanese) have complex many-to-one or one- to-many relationships from sound to symbol, making ASR particularly difficult (Taguchi & Chiang, 2024). Examples of this phenomena are shown in Table 3. We hypothesized that larger OWLS models will exhibit enhanced robustness to orthographic opacity. To measure this, we calculate the normalized CER (N-CER) by normalizing all symbols to a single orthography. This can then be compared to the unnormalized CER. model with good N-CER but poor CER has strong phonetic capabilities but poor orthographic
Orthographic Understanding:
5.1. Beam Search
One advantage of smaller models is the ability to leverage more complex decoding algorithms during inference. For larger models, using these techniques would be unfeasible within GPU memory constraints. To make the performance more fair the compute-level, we conduct analyses where all models have the same fixed test-time compute budget. Smaller models may leverage beam search with larger beam sizes, while larger ones may be constrained to only greedy decoding. Table 2 shows the WER on LibriSpeech test-other when test-time compute balanced at 400-50 TELOPS across the 0.25B, 2B, 4B, and 9B OWL.S models. We note that the 0.5B, IB, and 18B OWLS models are excluded since there is beam size that consumes similar number of TFLOPS. We first find that the WER of all models are sensitive to beam size, albeit with diminishing returns as model size increases. Even when using equivalent compute,
'In our work, we define "emergent abilities" as those exhibited by larger models and not by smaller models. Wei et al. (2022) originally used stricter definition where emergent abilities those that can not extrapolated from scaling curves. However. Schaeffer al. (2023) later showed the emergence can fact be predicted with finer-grained evaluation metrics.

---

OWLs: Scaling Laws for Multilingual Speech Recognition and Translation Models
80
Table Orthographic opacity examples of Japanese and Chi- nese. The same phone sequence can be written in different ways.
Aabic
Czech
70
Chinase
Cermar
Orthography
Example
Hndi
Romanization (zh)
shi shī shi क +
Japanese
Korean
Simp. Chinese
50
Portuguese
Trad. Chinese
3 +
Russiar
40
Spanish
Toligui
Romanization (p)
hashi
30
Hiragana
L
Katakana
20
Kanji
10
CER
N-CER
CER
N-CER
Model Size
60
15
10.9
Figure 11. CER on zero-shot English-X code-switching.
40
10
2023a). We collect an evaluation set of bilingually code- switched English for 12 languages from Yan et al. (2025) and test OWLS models of different sizes. Figure 11 shows the results on each code-switched language. We find that scaling can lead to significant reductions in code-switched CER, but the benefits are unevenly distributed. Many of the improvements lie in languages that also use the Latin alpha- bet, like Portuguese, while languages with very different orthographies (such as Chinese) see no improvement. More details about the data are in Appendix D.
28.1
31
29.7
78
76
77
7.
75
24.6
57
57
57
19.1
18.7
20
18
14.
12.4
11.5
10:5.
0.250 0.50
10
20
46
90
0.250 0.50
10
20
90
100
5009
30.9
Figure 10. Effects of model scaling on orthographic understand- ing Chinese (left) and Japanese (right). The quick saturation in N-CER shows that scaling does not have a large effect on the phonetic understanding in ASR models. However, the raw CER trend shows that large-scale models exhibit significantly stronger orthographic capabilities.
Mondegreen: Humans capable of constructing seman- tically meaningful sentences from mis-recognized speech (such as mishearing "José, can you see" from "O say can you see"). This phenomena is known as a mondegreen. We hy- pothesize that large ASR models learn more semantic map- pings than smaller ones, enhancing their ability of construct- ing mondegreens. We evaluate this technique by purpose- fully providing the model an English ASR task token along with speech from 3 non-English languages from FLEURS. The generated text is then evaluated by using the perplex- ity of a pre-trained OPT 2.7B LLM (Zhang et al., 2022b), such that a lower perplexity corresponds to a semantically plausible English sentence for humans. To ground these numbers, we also perform a qualitative analysis with 13 hu- man volunteers, who provided mean opinion score (MOS) the semantic coherence for each generation on scale from to 5 (higher is better). The results of the mondegreen evaluations are shown in Table 4. We observe that larger models obtain consistently better perplexity scores across all model sizes. Similarly, we also find that higher MOS scores trend well with model size. This suggests that larger ASR models are indeed more capable of "mis-hearing" in a semantically sound manner. While this phenomena is likely an artifact of scaling that does not directly relate to WER, we believe that such findings may lead to more research on the
understanding. Models are tested on Taiwanese Chinese Mandarin (zh-TW) and Japanese (Figure 10). The N-CER curve shows that scaling does not have a large impact on learning phoneties: small models already exhibit strong performance phonetically mapping speech to text. On the other hand, the steeper CER curve calculated from the raw model outputs indicate that larger models exhibit sig- nificantly stronger orthographic capabilities. Another key finding in this experiment was the overall robustness of larger models to zh-TW, which is a minority dialect rela- tive to Mannlln Chinese (zh-CN). Larger models are much more capable of providing fair performance across both dialects (see Table for zh-CN scores), which aligns with the findings in Section 4.1 low-resource languages.
Code-switching: In multilingual societies, it is common for more than one language to be spoken within single utterance. However, despite multilingual training, most ex- isting ASR models are incapable of accurately recognizing code-switched speech in zero-shot manner (Peng et al.,
8

---

OWLs: Scaling Laws for Multilingual Speech Recognition and Translation Models
want to recognize speech
Predicted text
Encoder
Decoder
Concat
wwx vn mnddony mmn
Hello, The NASDAQ fell Friday,
In-context example audio (k=2)
Target audio
In-context example text (k=2)
Figure The ICL inference process. The speech in-context examples and test audio are concatenated along the sequence
dimension. The text of the in-context examples also concatenated and used prompt for the decoder.
Table 4. Evaluation of mondegreen capabilities.
Table 5. Quechua CER on ICL with 0/1/2/3 examples. The overall best result bolded while the best result for each model size is underlined.
Params.
PPL
MOS
0.25B
1338
1.9
Params.
k 0
k
2
k
0.50B
728
4.1
IB
559
3.5
0,25B
36.9
35.1
33.7
34.5
2B
491
3.6
0,50B
53.3
39.2
33.8
33.9
4B
436
3.8
IB
41.8
35,0
31.6
31.8
9B
372
4.8
2B
47.3
35.1
31.9
33.2
18B
429
4.4
4B
40.4
32.4
31.2
31.8
9B
38.3
31.3
28.1
27.4
properties of neural networks that emerge scale, and how they relate the human perceptions of spoken language. More experimental details and sample inputs/outputs are in Appendix F and https: htt///aachchhhhee gi github. 10 hswwwll--oo samples, respectively.
18B
41.3
32.7
31.3
28.1
three in-context examples (Table 5). For the 4B and smaller models, performance degrades when using more than two in-context examples. Sample outputs and further details can be found in Appendix H.
5.3. In-Context Learning of OWLS
6. Conclusion and Future Work
LLMs are capable of few-shot task performance via in- context learning (ICL) (Brown et al., 2020). Large-scale ASR models like Whisper have shown potential in perform- ing ICL, albeit with very limited capabilities. In this section, evaluate if the ICL ability of OWLS models improve as the model size scales. To do so, evaluate the model on ASR for language unseen during training. We provide the model with 0 to 4 in-context examples to benchmark its ability to learn at test-time. We use Quechua as the unseen language, with data sourced from the Siminchik (Cardenas al., 2018) corpus. We perform ICL using the same k-NN approach Wang al. (2024a), where utterances with the lowest Euclidean distance (when embedded by the coder) from the target speech are selected from the training set as in-context examples. The audio from the in-context examples are concatenated with the target speech, while the concatenated text examples are fed as an input prompt (Figure 12). We find that while all model sizes are capa- ble of using in-context examples in some capacity, only the largest models (9B and 18B) take advantage of all
This paper introduces OWLS, a suite of 13 joint ASR/ST models designed help researchers understand the scal- ing behaviors of multi-modal, multi-language, multi-task models. OWLS models range from 250M to 18B parame- ters, trained on 11K to 360K hours of speech. In fact, the 18B OWLS model is the largest speech model in known literature. With OWLS, we show that the affects of scaling parameter, training data, and compute can lead to reasonable direct predictions of downstream ASR/ST performance. We also study the emergent capabilities of large-scale ASR/ST models, showing for the first time how larger speech models exhibit stronger in-context abilities and understanding of human language. In the future, we plan to (i) scale model training to even larger datasets and more diverse tasks, and (ii) investigate more scaling effects for adaptation, while also developing new benchmarks better understand the emergent capabilities of spoken language models with open and diverse research communities together.

---

OWLs: Scaling Laws for Multilingual Speech Recognition and Translation Models
Acknowledgments
M., Lam, T. K., Ma, X., Mathur, P., Matusov, E., Maurya, C., McCrae, J., Murray, K., Nakamura, S., Negri, M., Nichues, J., Niu, X., Ojha, A. K., Ortega, J., Papi, S., Polák, P., Pospfšil, A., Pecina, P., Salesky, E., Sethiya, N., Sarkar, B., Shi, J., Sikasote, C., Sperber, M., Stüker, S. Sudoh, K., Thompson, B., Waibel, A., Watanabe, S., Wilken, P., Zemánek, P., and Zevallos, R. FINDINGS OF THE IWSLT 2024 EVALUATION CAMPAIGN. In Salesky, E., Federico, M., and Carpuat, M. (eds.), Proc. IWSLT, pp. 1-11, Bangkok, Thailand (in-person and on- line), August 2024.
Parts of this work used the Bridges2 at PSC and Delta/DeltaAl NCSA computing systems through allocation CIS210014 from the Advanced Cyberinfrastructure Coordi- nation Ecosystem: Services & Support (ACCESS) program, supported by National Science Foundation grants 2138259, 2138286, 2138307, 2137603, and 2138296.
Impact Statement
This paper presents OWLS, suite of open-access, repro- ducible, large-scale joint ASR and ST models. Unlike most other ASR foundation models at this scale, all of the models in this work are trained on publicly accessible datasets and open-source codebases. To facilitate reproducibility, we will also release all intermediate checkpoints, optimizer states, and the final model checkpoint. Our goal is to provide re- searchers with additional resources and artifacts to better understand the scaling properties of large-scale speech mod- els. We also offer detailed breakdowns of computational resources and costs in the Appendix.
An, K., Chen, Q., Deng, C., Du, Z.,, Gao, C., Gao, Z.,, Gu, Y., He. T., Hu, H., Hu, K., al. Funaudiollm: Voice understanding and generation foundation models for nat- ural interaction between humans and llms. arXiv preprint 2rXi::2407.0400 2024.
Ardila, R., Branson, M., Davis, K., Kohler, M., Meyer, J., Henretty, M., Morais, R., Saunders, L.,, Tyers, F., and Weber, G. Common voice: A massively-mulllinuaal speech corpus. In LREC 2020, pp. 4218-4218888 (118-4222,2001 2020.
Babu, A., Wang, C., Tjandra, A., Lakhotia, K., Xu, Q., Goyal, N., Singh, K., von Platen, P., Saraf, Y., Pino, J., Baevski, A., Conneau, A., and Auli, M. XLS-R: Self- supervised Cross-lingual Speech Representation Learning at Scale. In Interspeech 2022, pp. 2278-2282, 2022. doi: 10.24437/lmexxpeechh0222..
Societal Consequences
There many potential societal consequences of machine learning, most of which we will not highlight here because they are common across the entire field. Instead, we will discuss the aspect of our work that is most unique: the impact on society resulting from model scaling. Training the OWLS models required many GPUs, which can consume large amounts of electricity. Although our computing costs are insignificant compared those incurred in LLM training (i.e., we use at most 48 GPUs at once), they remain large relative to most other work.
Baevski, A., Zhou, Y., Mohamed, A., and Auli, M. wav2vec 2.0: A framework for self-supervised learning of speech representations. In NeurIPS 2020, volume 33, 2020.
Bang, J.-U. et al. KsponSpeech: Korean spontaneous speech corpus for automatic speech recognition. Applied Sci- ences, 2020.
Ethical Aspects
Barrault, L., Chung, Y-AA,, Meglioli, M. C., Dale, D., Dong, N., Duppenthaler, M., Duquenne, P.-A., Ellis, B.,, Elsahar, H., Haaheim, J., et al. Seamless: Multilingual expres- sive and streaming speech translation. arxiv:2112.05187, 2023a.
Our models, like all machine learning models, are prone to bias due to uneven distributions in the training data. Al- though we show that model scaling can lead to more fair performance across different languages, it can still be prone to hallucinations and generate incorrect output.
Barrault, L., Chung, Y.-A,, Meglioli, M. C., Dale, D., Dong, N., Duquenne, P.-A., Elsahar, H., Gong, H., Heffernan, K., Hoffmman, J., et al. SeamlessM4T- massively multilingual & multimodal machine translation. arxiv:2308. 1196, 2023b.
A portion of the training data that we was accessed un- der non-commericial licenses. To follow the spirit of these datasets' access conditions, all of our models are also re- leased under non non-commercial licenses. We emphasize that the models are released for research purposes and discour- age use outside of this original use-case.
Beijing DataTang Technology Co., L. aidatatang 200zh, a free Chinese Mandarin speech corpus.
References
Biderman, S., Schoelkopf, H., Anthony, Q., Bradley, H.,
Ahmad, I. S., Anastasopoulos, A., Bojar, O., Borg, C., Carpuat, M., Cattoni, R., Cettolo, M., Chen, W., Dong, Q., Federico, M.. Haddow, B., Javorský, D., Krubiński,
O'Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., Skowron, A., Sutawika, L., and Van Der Wal, O. Pythia: suite for analyzing large
10

---

OWLs: Scaling Laws for Multilingual Speech Recognition and Translation Models
language models across training and sealing. In Proc. ICML., ICMLL'23, 2023.
570. URL https://sclanthhat/t/t emnlp-main.570
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, D.,
Chu, Y., Xu, J., Yang, Q. Wei, H., Wei, X., Guo, Z., Leng. Y., Lv, Y., He, J., Lin, J., et al. Qwen2-audio technical report. arXiv preprint arXiv:2407.10755 2024.
Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
Askell, A., Agarwal, S., Herbert- Vosss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler. D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E.,, Litwin, M., Gray. S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Lan- guage models are few-shot learners. In Proc. NeurIPS, volume 33, pp. 1877-1901, 2020.
Conneau, A., Baevski, A., Collobert, R., Mohamed, A., and Auli, M. Unsupervised Cross-Lingual Representa- tion Learning for Speech Recognition. In Interspeech 2021, pp. 2426-2430, 2021. doi: 10.21437/Interspeechh 2021-329.
Conneau, A. et al. FLBURS: Few-shot learning evaluation of universal representations of speech. In SLT 2022, 2022.
Bu, H. et al. AISHELL-1: An open-source Mandarin speech corpus and a speech recognition baseline. In O-COCOSDA, 2017.
Cuervo, S. and Marxer, R. Scaling properties of speech lan- guage models. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (ods.), Proc. EMNLP. pp. 351-361, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
Cardenas, R., Zevallos, R., Baquerizo, R., and Camacho, L. Siminchik: A speech corpus for preservation of southern Quechua. ISI-NLP, 2018.
Carletta, J. Unleashing the killer corpus: experiences in cre- ating the multi-everything AMI meeting corpus. Springer, 2007.
Dabbish, L., Stuart, C., Tsay, J., and Herbsleb, J. Leveraging transparency. IEEE software, 30(1):37-43, 2012.
Dao, T. Flashattention-22: Faster attention with better paral- lelism and work partitioning. In The Twelfth International Conference on Learning Representations, 2024.
Cattoni, R. et al. MuST-C: A multilingual corpus for end- to-end speech translation. Computer speech & language, 66, 2021.
Droppo, J. and Elibol, O. Scaling laws for acoustic models. Proc. Interspeech, pp. 2576-2580, 2021. doi: 10. 21 21437//nterspecch.200 202p1/peeech.221116444 1644.
Chen, G. et al. GigaSpeech: An evolving, multi-domaii ASR corpus with 10,000 hours of transcribed audio. In Interspeech 2021, 2021.
Fernandes, P., Ghorbani, B., Garcia, X., Freitag, M., and Firat, O. Scaling laws for multilingual neural machine translation. In Proc. ICMI,, ICML'22,, 2023.
Chen, S., Wang. C., Chen, Z., Wu, Y., Liu, S., Chen, Z., Li, J. Kanda, N., Yoshioka, T., Xiao, X., Wu, J., Zhou, L., Ren, S., Qian, Y., Qian, Y., Wu, J., Zeng, M., Yu, X., and Wei, F. WavLM: Large-scale self-supervised pre-training for full stack speech processing. JEEE JSTSP, 2022. doi: 10.1 109/SSTSP.2022.1188111
Ghorbani, B., Firat, O., Freitag, M., Bapna, A., Krikun, M., Garcia, X., Chelba, C., and Cherry, C. Scaling laws for neural machine translation. In Proc. ICLR, 2022. URL tettps::/openreee// id=hR_ id=hR_SMa8cxe SMVV.
net/ forum?
Chen, W., Shi, J., Yan, B., Berrebbi, D., Zhang, W., Peng, Y., Chang, X., Maiti, S., and Watanabe, S. Joint prediction and denoising for large-scale multilingual self-supervised learning. In ASRU 2023, 2023a.
Gordon, M. A., Duh, K., and Kaplan, Data and parame- ter scaling laws for neural machine translation. In Proc. EMNLP, pp. 5915-5922, Online and Punta Cana, Domini- can Republic, November 2021. Association for Computa- tional Linguistics. doi: 10.18653/ 478. URL https: httss://aclaatthhhocccc org 2021. emn mamlp-maing 478/.
Chen, W., Yan, B., Shi, J., Peng, Y., Maiti, S., and Watanabe, S. Improving massively multilingual ASR with auxiliary CTC objectives. In ICASSP 2023, 2023b.
Chen, W., Zhang, W., Peng, Y. Li, X., Tian, J., Shi, J.. Chang, X., Maiti, S., Livescu, K., and Watan- abe, S. Towards robust speech representation learn- ing for thousands of languages. In Al-Onaizan, Y., Bansal, M., and Chen, Y.N. (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natu- ral Language Processing, pp. 10205-10224, Miami, Florida, USA, November 2024. Association for Compu- tational Linguistics. doi: 10665//v12022.em//// main.
Graves, A., Fernández, S., Gomez, F., and Schmidhuber, J. Connectionist temporal classification: labelling unseg- mented sequence data with recurrent neural networks. In ICML 2006, pp. 369-376, 2006.
Gu, Y., Gurunath Shivakumar, P., Kolehmainen, J., Gandhe, A., Rastrow, A., and Bulyko, I. Scaling laws for discrim- inative speech recognition rescoring models. Proc. Interspeech, pp. 471-475 2023.

---

OWLs: Scaling Laws for Multilingual Speech Recognition and Translation Models
Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., et al. Scaling laws for autoregressive generative modeling. arXiv preprint ar 14701, 2020.
O'Neill, P. K., Lavrukhin, V., Majumdar, S., Noroozi, V., Zhang, Y., Kuchaiev, O., Balam, J., Dovzhenko, Y., Frey- berg, K., Shulman, M. D., Ginsburg, B., Watanabe, S., and Kucsko, G. SPGISpeech: 5,000 hours of transcribed financial audio for fully formatted end-to-end speech recognition. In Interspeech 2021, 2021.
Hernandez, D., Kaplan, J., Henighan, T., and McCan- dlish, S. Scaling laws for transfer. arXiv preprint arXiv:2102.0122 00..0.001933 2021.
Panayotov, V. et al. Librispeech: An ASR corpus based on public domain audio books. In ICASSP 2015, 2015.
Hernandez, F., Nguyen, V., Ghannay, S., Tomashenko, N., and Esteve, Y. TED-LIUM 3: Twice much data and corpus repartition for experiments on speaker adaptation. In Speech and Computer: 20th International Conference, SPECOM 2018. Springer, 2018.
Peng, P., Yan, B., Watanabe, S., and Harwath, D. Prompting the hidden talent of web-scale speech models for zero- shot task generalization. In Proc. Interspeech, 2023a.
Peng, Y., Tian, J., Yan, B., Berrebbi, D., Chang, X., Li, X., Shi, J., Arora, S., Chen, W., Sharma, R., Zhang, W. Sudo, Y., Shakeel, M., weon Jung, J., Maiti, S., and Watanabe, S. Reproducing Whisper-style training using an open-source toolkit and publicly available data. In ASRU 2023, 2023b.
Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad. H., Patwary, M. M. A., Yang, Y., and Zhou, Y. Deep learning scaling is predictable, empirically, arXiv preprint ar arXiv:1712.00409 2017.
Peng, Y., Tian, J., Chen, W. Arora, S., Yan, B., Sudo, Y., Shakeel, M., Choi, K., Shi, J., Chang, X., et al. OWSM v3. 1: Better and Faster Open Whisper-Style Speech Models based on E- Branchforrrrr. arXiv preprint arXiv:2401.1663 2024.
IARPA. The Babel Program. URL www iarpa. batrrammmbabbl... babel.
index. index.ppp/reeeeeee
Kaplan, J., McCandlish, S., Henighan, T., Brown, B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint ar arXiv:2001.08361, 2020.
Peng, Y., Muhammad, S., Sudo, Y., Chen, W., Tian, J., Lin, C.-N., and Watanabe, S. Owsm v4: Improving open whisper-style speech models via data scaling and cleaning. In Proc. Interspeech, pp. 471-475, 2025.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. ICLR 2015, 2015.
Post, M. al. Improved speech-to-text translation with the fisher and callhome Opamish-Englinn speech translation corpus. In 2013, 2013.
Kudo, T. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Associ- ation for Computational Linguistics (Volume 1: Long Pupers), pp. 66-75, Melbourne, Australia, July 2018
Pratap, V., Xu, Q.. Sriram, A., Synnaeve, G., and Collobert, R. MLS: A large-scale multilingual dataset for speech research. In Interspeech 2020, pp. 2757-2761.
Le. D., Jain, M., Keren, G., Kim, S., Shi, Y. Mahadeokar, J., Chan, J., Shangguan, Y., Fuegen, C., Kalinli, O., Saraf, Y., and Seltzer, M. L. Contextualized streaming end-to- end speech recognition with trie- deep biasing and shallow fusion. In Proc. Interspeech, pp. 1772-1776, 2021.
Pratap, V., Tjandra, A., Shi, B., Tomasello, P., Babu, A.,
Kundu, S., Elkahky, A., Ni, Z., Vyas, A., Fazel-Zarandi M., et al. Scaling speech technology to 1,000+ languages. arxiv:2305. 13516, 2023.
Puvvada, K. C., Želasko, P., Huang, H., Hrinchuk, O., Koluguri, N. R., Dhawan, K., Majumdar, S., Rastorgueva, E., Chen, Z., Lavrukhin, V., et al. Less is more: Accurate speech recognition & translation without web-scale data, arXiv preprint arXiv:2406.. 19674, 2024.
Li, B., Pang, R., Sainath, T. N., Gulati, A., Zhang, Y., Qin, J., Haghani, P., Huang, W. R., Ma, M., and Bai, J. Scaling end-to-end models for large-scale multilingual asr. In Proc. ASRU, pp. 1011-1018, 2021.
Radford, A., Kim, J. W., Xu, T., Brockman, G., Mcleavey, C., and Sutskever, I. Robust speech recognition via large- scale weak supervision. In ICML 2023, 2023.
Li, X., Metze, F., Mortensen, D. R., Black, A. W., and Watanabe, S. ASR2K: Speech Recognition for Around 2000 Languages without Audio. In Interspeech 2022, 2022. doi: 22144///meerpeech22222-00000
Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. Zero: Memory optimizations toward training trillion parameter models. SC20: International Conference for High Per- formance Computing, Networking, Storage and Analysis, pp. 1-16, 2020.
Li, X., Takamichi, S., Saeki, T., Chen, W., Shiota, S., and Watanabe, S. YODAS: Youtube-orient dataset for au- dio and speech. In ASRU 2023, 2023.
12

---

OWLs: Scaling Laws for Multilingual Speech Recognition and Translation Models
Rasley, J., Rajbhandari, S., Ruwase, O., and He, Y. Deep- speed: System optimizations enable training deep learn- ing models with over 100 billion parameters. In Proceed- ings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '20, pp- 3505-3506, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450379984.
Wang, C. et al. VoxPopuli: A Large-Scale Multilin- gual Speech Corpus for Representation Learning, Semi- Supervised Learning and Interpretation. In ACL 2021, 2021b.
Wang, S., Yang. C.-H., Wu, J., and Zhang, C. Can whis- per perform speech-based based in-context learning? In Proc. ICASSP, pp. 13421-13425, 2024a.
Rosenfeld, J. S., Rosenfeld, A., Belinkov, Y., and Shavit, N. A constructive prediction of the generalization error across scales. In International Conference on Learning Representations, 2020. URI, https: / /pppneeeeeeeeeeeeeeeeeeeeeeeeeee net/ www.rooouuuumunpppaae
Wang, S., Yang, C.-H. H., Wu, J., and Zhang, C. Bayesian example selection improves in-context learning for speech, text and visual modalities. In Proceedings of the 2024 Conference on Empirical Methods in Nat- ural Language Processing, pp. 20812-20828, Miami, Florida, USA, November 2024b. Association for Compu- tational Linguistics. doi: 10. 10.18663/11/2224emmmp 1158. URL https: emn main 1158/.
Schaeffer, R., Miranda. B., and Koyejo, S. Are emergent abilities of large language models a mirage? In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Proc. NeurIPS, volume 36, pp. 55565- 55581. Curran Associates, Inc., 2023.
hallaanthloogy.orgggoo 2024.
Watanabe, S., Hori, T., and Hershey, J. R. Language inde- pendent end-to-end architecture for joint language iden- tification and speech recognition. In 2017 IEEE Auto- matic Speech Recognition and Understanding Workshop (ASRU), pp. 265-271, 2017a. doi: 10.1 AS12 8268945.
Slizhikova, A. et al. Russian Open Speech To Text (STT/ASR) Dataset, 2020. URL https: /ggthtbt... com http//ankeers//ppn..oo
Taguchi, C. and Chiang, D. Language complexity and speech recognition accuracy: Orthographic complexity hurts, phonological complexity doesn't. arXiv preprint arXiv:2406.09200 09202 2024.
Watanabe, S., Hori, T., Kim, S.,, Hershey, J. R., and Hayashi, T. Hybrid CTC/attention architecture for end-to-end speech recognition. IEEE Journal of Selected Topics in Signal Processing, 2017b.
Tay, Y., Dehhhann, M. Abnar, S., Chung, H., Fe- dus, W., Rao, J., Narang, S., Tran, V., Yogatama, D., and Metzler, D. Scaling laws vs model ar- chitectures: How does inductive bias influence seal- ing?
Watanabe, S., Hori, T., Karita, S., Hayashi. T., Nishitoba. J., Unno, Y., Enrique Yalta Soplin, N., Heymann, J., Wiesner, M., Chen, N., Renduchintala, A., and Ochiai, T. ESPnet: End-to-end speech processing toolkit. In Interspeech 2018, 2018
Bouamor, H., Pino, J. and Bali, K. (eds.), Findings of EMNLP, pp. 12342-12364, Singa- pore, December 2023. Association for Computational Linguistics. doi: 10. 11188//1120233.nnniigg 825. URL https: / atlaaaaaaaatlggoe org/2023. findings-enn emanl 825/.
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Met- zler, D., Chi, E. H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., and Fedus, W. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022. ISSN 2835-8856.
Tian, J., Peng, Y., Chen, W., Choi, K., Livescu, K., and Watanabe, S. On the effects of heterogeneous data sources on speech-to-tex foundation models. In Interspeech 2024, pp. 1959-3963, 2024. doi: 10.21437/Interspeechh 0024-1938.
Yamagishi, et al. CSTR VCTK Corpus: English Multi- speaker Corpus for CSTR Voice Cloning Toolkit, 2019.
Yan, B., Hamed, I., Shimizu, S., Lodagala, V., Chen, W. lakovenko, O., Talafha, B., Hussein, A., Polok, A., Chang, K., et al. Cs-fleurs: massively multilingual and code- switched speech dataset. In Proc. Interspeech, pp. 471- 475, 2025.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In NeurlPS 2017, 2017.
VoxForge. VoxForge. URL http::::pwwwwww...eeppee http://wwwwwwoo..eeee/o/e/ org/.
Yang, C.-H. H., Li, B., Zhang, Y., Chen, N., Prabhavalkar, R., Sainath, T. N., and Strohman, T. From english to more languages: Parameter-efficien model reprogramming for cross-lingual speech recognition. In ICASSP 2023-2023
Wang, C. et al. CoVoST 2 and Massively Multilingual Speech Translation. In Interspeech, 2021a.
13

---

OWLs: Scaling Laws for Multilingual Speech Recognition and Translation Models
IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Yang. Z., Chen, Y., Luo, L., Yang. R., Ye, L., Cheng. G., Xu,
J., Jin, Y., Zhang, Q., Zhang, P., Xie, L.,, and Yan, Y. Open source MagicData-RAMe AMMM: rich annotated mandarin conversational (RAMC) speech dataset. In Interspeech 2022, pp. 1736-1740, 2022.
Ye, R. al. GigaST: A 10,000-hour pseudo speech transla- tion corpus. In Interspeech 2023, 2023.
Yin, Y., Mori, D., et al. ReazonSpeech: A Free and Massive Corpus for Japanese ASR, 2023.
Yu, Y., Yang, C.-H. H., Kolehmainen, J., Shivakumar, P. G., Gu, Y., Ren, S. R. R., Luo, Q., Gourav, A., Chen, I.FF,, Liu, Y.-C., et al. Low-rank adaptation of large language model rescoring for parameter-efficient speech recogni- tion. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 1-8. IEEE, 2023.
Zhai, X., Kolesnikov, A., Houlsby, N., and Beyer, L. Scaling vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12104-12113 2022.
Zhang, B. et al. WenetSpeech: A 10000+ hours multi- domain mandarin corpus for speech recognition. In ICASSP 2022, 2022a.
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. OPT: Open pre-trained transformer language models. arxiv:2005.0106 2022b.
Zhang, Y., Han, W., Qin, J., Wang, Y., Bapna, A., Chen, Z., Chen, N., Li, B., Axelrod, V., Wang, G., et al. Google USM: Scaling automatic speech recognition beyond 100 languages. arxiv:2303.01037, 2023.
Zheng, W., Xiao, A., Keren, G., Le, D., Zhang, F., Fuegen, C., Kalinli, O., Saraf, Y., and Mohamed, A. Scaling asr improves zero and few shot learning. In Proc. Interspeech, pp. 1135-5139, 2022.
14

---

OWLs: Scaling Laws for Multilingual Speech Recognition and Translation Models
Table Overview of datasets used in the 180K OWSM v3.2 dataset. The language column indicates the language used in monolingual datasets and the number of languages in multilingual datasets.
Dataset
License
Language(s)
Domain
Hours
MLS (Pratap et al.)
CC BY 4.0
8 Mandarin
Audiobook
44K
WeNetSpeech (Zhang al., 2022a)
CCBY 4.0/SA
Variety
22K
Russian Open STT (Slizhikova et al., 2020) Reazonspeech (Yin et al., 2023)
CC-BY-NC
Russian
Variety
20K
Apache 2.0
Japanese
Television
15K
Common Voice 13 (Ardila et al., 2020)
CCO-1.0
92
Read
13K
GigaSpeech (Chen al., 2021)
Apache 2.0
English
Variety
10K
GigaST (Ye al., 2023)
CC BY NC 4.0
Variety
24K
MuST-C (Cattoni et al., 2021)
BY NC ND 4.0
16
Talk
10K
CoVoST2 (Wang et al., 2021a)
BY NC 4.0
22
Read
8550
SPGI (eileil et al., 2021)
CCBY-NC-ND 4.0
English English
Finance
5000
Fisher (Post et al., 2013)
LDC
Conversation
2000
VoxPopuli (Wang et al., 2021b)
BC-N 4.0
23
Legal
1800
Googlei18n 8n (Chen et al., 2023a)
Varies
34
Variety
1328
BABEL (IARPA)
IARPA Babel License
17
Conversation
1000
FLEURS (Conneau al., 2022)
CC BY 4.0 MIT
102
News
1000
Kppenn (Bang et al., 2020)
Korean
Conversation
970
LibriSpeech (Panayotov al., 2015)
BC BY 4.0
English
Audiobook
960
MagicData (Yang et al., 2022)
CCBY-NC-ND 4.0
Mandarin
Conversation
755
TEDLIUM3 (Hernandez et al., 2018)
CCBY-NC-DD 3.0
English
Talk
500
Fisher Callhome Spanish (Post et al., 2013)
CC BY-SA 3.0
2
Conversation
241
VoxForge (VoxForge)
GPL Apache 2.0
8
Read
235
AISHELL (Bu et al., 2017)
Mandarin
Read
200
AIDATATANG (Beijing DataTang Technology Co.)
BYNNCNND 4.0
Mandarin
Read
140
AMI (Carletta, 2007)
CC BY 4.0
English
Meetings
100
VCTK (Yamagishi al., 2019)
BC 4.0
English
Read
25
A. Dataset
For the base 180K hours experiments, we use the exact same corpora as those in OWSM v3.2 (Tian et al., 2024). We emphasize that all of these corpora are publicly accessible (although not necessarily purely open-source due to some licensing restrictions). In total, this leads to 25 corpora across 151 languages. Following Tian et al. (2024), the target text data normalized by restoring punctuation and casing. In total, there are 150K hours of data for ASR and 30K hours of data for ST. Details on the license, languages, domain, and size of each corpora are shown in Table A per-language distribution of the 150K hours of ASR data shown in the third column of Table 13.
To scale to 360K hours, collect more data from YODAS (Li et al., 2023), which contains 500K hours of speech. However, since the data is crawled from YouTube, the transcripts are very noisy. We therefore obtained and used clean 180K hour subset YODAS from Peng et al. (2025), will be made publicly available in the near future. A breakdown of the amount of additional data per language is available in the last column in Table 13.
B. Training Details
All models use total effective batch size of 256 utterances and are trained for 675K steps. We use the Adam optimizer (Kingma & Ba, 2015) with piecewise scheduler (Peng al., 2024) that linearly warms up the learning rate from 0 to 5.0e-5 in the first 30K steps, 5.0e-5 to 2.0e-4 in the next 30K steps, and finally exponentially decays for the remaining training steps. For the hybrid CTC/attention (Watanabe et al., 2017b) training, we use CTC weight of 0.3. We use bfloat 16, Flash Attention (Dao, 2024), and DeepSpeed Zero Stage-2 (Rasley et al., 2020; Rajbhandari et al., 2020) to improve training efficiency.
As mentioned in Section 3.2, all OWLS models follow Transformer (Vaswani et al., 2017) encoder-decoder architecture trained using hybrid CTClattention (Graves et al., 2006; Watanabe et al., 2017b) loss. Both the encoder and decoder use
15

---

OWLs: Scaling Laws for Multilingual Speech Recognition and Translation Models
Table 7. Training details for each model size.
Params.
Data Hrs.
GPU Type
GPUs per Node
Nodes
Days Training
GPU Hours
0.25B
180K
H100
8
1,164
0.50B
180K
H100
8
1,512
IB
11K
H100
8
2,232
IB
22K
H100
8
3
2,790
IB
45K
H100
8
3
2,790
IB
90K
H100
8
3
2,790
1B
180K
H100
8
3
2,790
IB
360K
H200
4
5,120
2B
180K
H100
8
2,520
4B
180K
H100
8
3
5,148
9B
180K
H100
8
3
15
8,472
18B
180K
H100
8
6
17
19,440
sinusoidal absolute positional embeddings (Vaswani et al., 2017). The inputs to the Transformer encoder are 80-dimension log-Mel filterbanks extracted with frame shift of 10ms, which are then down-sampled 4 times by stack of convolution layers. The Transformer decoder auto-regressively predicts text tokens, which are pre-segmented with a unigram language model (Kudo, 2018) into a 50K subword vocabulary. We also use Whisper-style training (Radford et al., 2023): all utterances are padded to 30 seconds, and the model is jointly trained to perform language identification, ASR, ST, and timestamp prediction. The exact configurations for each model size are shown Table 8. We use a mix of A A100, H100, and GH200 GPUs for supervised training (Table
Table Architecture parameer details each model size.
Params.
Enc./Dec. Layers
Hidden Size
FFN Size
Attn. Heads
0,25B
768
3072
16
0.50B
1024
4096
16
IB
32
1024
4096
16
2B
16
2048
8192
64
4B
36
2048
8192
64
9B
39
2816
11264
64
18B
64
3072
12288
64
C. Beam Search
Table WER on FLEURS Spanish and Turkish when using greedy search (left) and balancing test-time compute budget (right).
Params.
Beam Size
TFLOPS
Spanish WER
Turkish WER
0.25B
1/
1.3 48.7
27.5 / 22.9
82.0 / 69.5
2B
1/4
11.1 36.2
10.6 9.4
42.3 34.5
4B
1/2
23.4 42.3
9.0 8.4
34.5 30.6
9B
1
47.7
9.7
29.2
Table 9 shows additional inference-time compute scaling experiments the Spanish and Turkish test sets of FLEURS. We use the same beam sizes as the English experiments in Table 2.
D. Code-Switching
The code-switching evaluation data collected from Yan et al. (2025). The authors create synthetic code-switching text by taking sentences from 12 non-English languages in FLEURS (Conneau al., 2022) and randomly swapping in English
16

---

OWLs: Scaling Laws for Multilingual Speech Recognition and Translation Models
translations via dictionary mapping. The swapping is done the word-level. Bilingual volunteers are then tasked to read the code-switched speech. All volunteers are native speakers in the non-English language and least fluent in English. The languages to create the code-switched text are Arabic, Czech, Chinese, German, French, Hindi, Japanese, Korean, Portuguese, Russian, Spanish, and Telugu.
E. Japanese and Taiwanese Chinese Mandarin ASR
This section expands the orthographic analyses results and compares the performance of OWLS on ReazonSpeech Japanese (Yin et al., 2023) and Common Voice Taiwanese Chinese Mandarin (Ardila et al., 2020) against Whisper Large v3 (Radford et al., 2023). The results are shown in Table 10. All OWLS models beyond 4B parameters outperform Whisper Large v3. OWLS 9B achieves the best performance on Reazonspeech with 7.3 CER, less than half of that of Whisper (15.1 CER). OWLS 18B achieves the best performance on Taiwanese Mandarin with a CER of 18.7, while Whisper has a CER of 26.9.
Table 10. ASR performance against SOTA models on Japanese (Reazonspeech) and Taiwan Chinese Mandarin (Common Voice).
Model
Japanese (ja-ip)
Taiwanese Chinese Mandarin (zh-tw)
Whisper Large v3
15.1
26.9
OWLS 0.25B
14.7
50.2
OWLS 0.5B
9.3
39.7
OWLS 1B
7.8
31.0
OWLS 2B
7.6
29.7
OWLS 4B
7.7
24.6
OWLS 9B
7.3
19.1
OWLS 18B
7.5
18.7
F. Mondegreens
As discussed in Section 5.2, mondegreens manner. For Chinese and Japanese speakers, these are known as н (köng' ër soramimi). These can either occur within a language ("José, can you see" vs "O say can you see") or across languages ("Bon Appétit" vs "Bone Apple Tea"). We focus on the cross-lingual mondegreen setting, since generating monolingual mondegreens are challenging due to the strength of modern ASR systems. To do this, we first randomly select three low-resource languages (Thai, Afrikaans, and Vietnamese) from FLEURS. We have each model perform ASR inference on these languages, but purposefully input an incorrect English language task tag".
where human mishears a phrase in somewhat semantically coherent
For the human evaluation, we have 13 volunteers rate the semantic coherence of the text corresponding to each utterance on scale from 1 to 5. Scores of indicate completely non- non-English text random strings, while scores of 5 correspond to coherent and realistic English words. We filter out all utterances with an average score across all models below 3.0, removing all utterances that are naturally unsuited for creating English mondegreens. Finally, we obtain the average human score for each individual model output, and report the score averaged across all utterances for each model in Table 4. Sample outputs are shown in Table 11.
G. Contextual Biasing
Previous studies (Peng et al., 2024) have shown that zero-shot contextual biasing is an ability emergent in larger (1B+) ASR models. In this section, we scale the evaluation to the 18B setting. We use the same Librispeech contextual biasing data as Peng et al. (2024); Le et al. (2021), where the model is prompted with a list of true target rare words and distracters. The goal of this task to reduce the biased WER (B-WER) without degrading the unbiased WER (U-WER). Similar to the results in ST (Section 4.1), we find that small models may encounter catastrophic failures contextual biasing: the 0.25B model yields WER of near 97% by frequently outputting blank predictions (Table 12). The 0.5B model also encounters performance degradations upon using contextual biasing prompts, albeit a less severe magnitude. IB+ parameter models
*We initially attempted this evaluation with high-resource non-English languages, hut found that models would ignore the incorrect task tag and always transcribe in the original language. We leave further studies of this phenomena to future work.
17

---

OWLs:
Scaling Laws for Multilingual Speech Recognition and Translation Models
Table Example mondegreen generations and their corresponding original text.
Source
Text
Original
Vir daardic rede, als wat jy op die TV sien, het die kante gesny, bo, onder en kante.
0.25B
Dore the rear of the ozvatioctiya fissic.
0.5B
For Dore the Rieda also got the optic fissure.
IB
The order did read Vatican's 's affiliate for the first time.
2B
The Daily Director also wrote the optics for his work. For the order read, also what the optieth is. The door of the red house was fatty, and the squad was very tired. For the ordinary, the oasis varies between the oasis and the oasis.
4B
9B
18B
Original
Alle burgers van die Vatikaan Stad is Rooms Katoliek.
0.25B
Alabarkers fan diva
0.5B
Alabama cares for the development of the reservation.
1B
allebergers van the valley
2B
Alabama kerrs fan the game.
4B
Alabama, Cars, Fan, Diva.
9B
All the birds catch the worm. All the workers found the vat.
18B
able better take advantage of the context words, while maintaining U-WER. In fact, only the 9B model is capable of sufficiently maintaining the U-WER while sufficiently lowering the B-WER to get overall lower WER.
H. In-Context Learning
Text-based LLMs are capable few-shot task performance via in-context learning (ICL) from text prompts at inference time. This generally done by concatenating consecutive examples together, where each example is an input and expected output pair, and feeding the concatenated text as input into decoder-only causal language model.
We perform ICL for encoder-decoder ASR models in similar manner, using the same popular formulation introduced by Wang et al. (2024a;b). The encoder is first used to extract embeddings of each speech example in the ICL training set, which are averaged across the sequence dimension and cached. This process is summarized in Figure 13. During inference time, we also extract a time-averaged embedding for the input speech and retrieve the & training samples from the cache with the smallest Euclidean distance from the embedding of the test sample. The audio of the retrieved training samples are then concatenated together, with half-second pause inserted between each sample. Finally, the speech of the input test utterance appended at the end. This will be used the encoder input. The decoder input is therefore the concatenation of the retrieved training examples, with comma inserted between each sample.
H.1. Quechua Evaluation
Quechua is a low-resource language indigenous Peru and does not appear in any of the training data that we use. To perform the Quechua ICL evaluation, use the IWSLT 2024 (Ahmad et al., 2024) version of the Siminchik corpus (Cardenas et al., 20) We filter out all utterances longer than 7 seconds and split the corpus such that speaker can only appear in the training or test set. We then further subsample the training set to 150 utterances to reduce compute costs.
18

---

OWLs: Scaling Laws for Multilingual Speech Recognition and Translation Models
Table 12. WER zero-shot contextual biasing.
Params.
test-clean U-WER
test-other U-WER
WER
B-WER
WER
B-WER
0.25B
4.05
2.76
15.08
9.62
7.33
30.82
biasing
97.73
98.41
91.84
98.88
99.47
93,49
0.50B
2.65
1.77
10.22
6.61
4.85
22.97
+ biasing
2.40
1.83
7.24
6.09
4.94
16.83
IB
2.30
1.50
9.14
5.59
4.02
20.25
+ biasing
2.04
1.54
6.31
5.19
4.19
14.50
2B
2.18
1.44
8.44
5.18
3.73
18.7
biasing
1.98
1.50
5.98
4.63
3.70
13.23
4B
2.03
1.37
7.60
4.65
3.33
16.90
+ biasing
2.02
1.68
4.89
5.13
4.47
11.32
9B
1.89
1.25
7.39
4.52
2.97
18.93
+ biasing
1.72
1.29
5.32
4.47
3.67
11.93
ICL Training audio
Target audio
MMX WwW
Encoder
Encoder
Average time
mo0omm
Select k=2 closest embeddings by L2
mooeom
mopmm
000000
Figure The ICL embedding mining process for selecting in-context examples. For a single audio input, the encoder extracts an audio embedding of the test audio and all audio samples in the ICL training set. The embeddings are averaged across the sequence dimension. The in-context examples are selected by choosing the top embeddings with the smallest distance from test audio embedding.
19

---

OWLs: Scaling Laws for Multilingual Speech Recognition and Translation Models
100
100
Rt
R2
pus R+ 0.82
Try. R 0.57
cos,
cym, 0.84
Glg: R2 0.92
swom, R 0.98
An. 0.55
0.97
HY 0.96
hun. R?
WER
Rat, RF 0.97
WER
wil, R 0.93
R
0.59
0.91
mon, R2 0.98
ekc. R2 1.89
0.34
tel, KP 0.54
lax, R? 0.98
smnh, R? 0.82
alx, R 0.93
cob,
R2-0000 0.37
luo, 0:000
Pin, -..0
0.60
ml. Q2--00 0
S179
See
100
100
kan, R 0.05
RM 0:35
hau, R2 0.57
nob, R
0.80
50
den, R 0.97
bos, -000020000000 0..09
mal R 0.97
- 0.89
WER
you, RV 0.97
Iz RC* 0.98
R2 8.54
R2 0.98
mys, R 0.89
wol, R2 0.95
snd. R 0.90
0.52
com,
tpp. je 0:08
med. x aM
0.77
uma,
shm, R 0.71
att R 0.50
Figure 14. Model scaling laws for all languages in FLEURS. For almost all languages, WER/CER strongly correlated with the power
w.I.t model parameter size.
20

---

OWLs: Scaling Laws for Multilingual Speech Recognition and Translation Models
th
Ibo
pus
80
mpm
dbb
a6m
WERIIGER
hun
WERIEER
ket
dl
uzb
kzz
bal
amch
20
mil
Training Hours Per Languege
Training Hours Por Languago
hou
1ho
nob
glc
STD
dan
boo
kam
In
hrb
WERICER
WERICER
11005
Sifh
pln
midd
np
kim
off
10
10' Training Hours Per Language
Trailing Hours Per Language
Figure Change WER/CER when adding more data per language. most languages, we ohserve same trend as Figure more data with no increase in diversity does not lead to meaningful changes in WER/CER.
21

---

OWLs: Scaling Laws for Multilingual Speech Recognition and Translation Models
Table Amount ASR training data per language in the OWSM v3.2 180K
YODAS 180K corpora for top languages
in FLEURS.
Language
ISO3 Code
OWSM v3.2 Hours
YODAS Hours
English
eng
73000
75000
Russian
rus
20183
15692
Japanese
jpn
18900
1934
Chinese
cmm
16000
176
German deu
3700
7129
French
fra
2500
8560
Spanish
spa
2000
17344
Catalan
cat
1996.7
37
Dutch
nld
1700
1193
Belarussian
bel
1319.31
0.15
Korean
kor
1000
10890
Italian
ita
700
5553
Bengali
ben
373.6
9.2
Javanese jav
372
0
Persian
fas
309
30
Polish
pol
300
465
Portuguesse
por
300
10815
Tamil tam
296
10
Cantonese yue
235
0
Turkish tur
199
2322
Thai tha
139
363
Vietnamese vie
139
4644
Pastho pus
126
0.141
Czech
ces
117
69
Welsh
cym
111.6
4.2
Kurdish
ckb
108
0
Zulu zul
107.5
0
Assamese asm
106.878
0.06
Lao
lao
105
0.007
Hungarian hun
97.3
114
Georgian kat
90.05
Uzbek
uzb
88
2.7
Lithuanian
lit
86
5
Mongolian
mon
86
0.42
Kazakh kaz
79.18
0.9
Telugu
tel
76
0.7
Amharic amh
0.33
Cebuano
ceb
Luo
luo
75
0
Arabic
74.6
89
Igbo
ibo
73.017
0
Ukranian
ukr
69
433
Croatian hrv
46.6
5.1
Urdu urd
Estonian est
42.1
Galician glg
39
Kyrgyz
kir
39
0
Finnish fin
38.5
137
Swedish
swe
35.5
76
Indonesian ind
25,3
3270
22

---

OWLs: Scaling Laws for Multilingual Speech Recognition and Translation Models
Table Amount ASR training data per language in the OWSM v3.2 180K languages in FLEURS.
YODAS 180K corpora for the bottom 50
Language
ISO3 Code OWSM v3.2 Hours
YODAS Hours
Greek
ell
24
42
Marathi
mar
23
1.1
Romanian
ron
22
0
Slovakian
slk
20.8
6.5
Bulgarian
18.27
9.1
Latvian
lav
16.5
0.704
Slovenian
slv
16.5
9.5
Gujarati
guj
16
5.32
Hindi
hin
16
261
Maltese
mlt
16
0
Kannada
kan
15.5
0.14
Xhosa
xho
15.5
Hausa
hau
15.13
Irish
gle
15
0
Maori
mri
15
0
Danish
dan
14.7
13
Kamba
kam
14
0
Lingala
lin
14
0
Malayalam
mal
13.84
2.8
Occitan
13.8
0.017
Yoruba
yor
13.5
0
Fulah
ful
12.9
0
Swahili
swh
12.7
0.9
Malaysian
mya
12
0
Sindhi
snd
12
0.009
Somali
som
12
0.6
Armenian
hye
11.8
0.24
Nepali
npi
11
0
Asturian
10.75
0
Cambodian
khm
10.3
1.86
Kabuverdianu
kea
10
0
Luganda
lug
10
0
Norwegian
nob
10
0
Nyanja
nya
10
Serbian
srp
10
2
Bosnian
bos
9.96
0
Shona
9.8
0
Hebrew
heb
9.4
0
Azerbaijani
aze
9.39
1.3
Masai
mas
9
0
Luxembourgish
8
Northern Sotho
nso
0
Tajik
tgk
8
0.032
Wolo
wol
8
0
Panjabi
pan
7.9
0.7
Filipino
fil
7.5
0
Macedonian
mkd
7
1.4
Oromo
orm
6.5
0
Umbundu umb
6.47
0
Afrikaans afr
5.5
0.041
23

---

OWLs: Scaling Laws for Multilingual Speech Recognition and Translation Models
Table 15. WER/CER for the top 50 languages in FLEURS by OWLS training data.
Languages
0.25B
0.50B
1B
2B
4B
9B
18B
English
16.8
11.8
9.7
9.5
8.5
8.5
7.7
Russian
36.4
23.8
17.5
18.5
14.7
14.8
14.5
Japanese
21.2
11.4
9.2
9.3
8.1
7.7
7.3
Chinese
26.9
17.4
13.8
13.1
12.3
11.6
10.6
German
27.0
16.6
12.7
11.7
10.2
10.0
9.5
French
36.4
23.1
17.5
16.6
14.4
13.7
13.2
Spanish
27.5
15.8
11.1
10.6
9.0
9.7
9.0
Catalan
30.5
15.2
11.4
11.1
8.8
8.6
8.8
Dutch
47.7
33.2
25.8
21.9
19.3
17.9
17.3
Belarussian
46.0
24.2
19.1
18.5
15,8
16.2
18.2
Korean
259.2
58.8
49.4
47.7
38.6
38.5
34.2
Italian
37.9
19.5
13.9
12.2
9.8
10.0
9.8
Bengali
45.5
23.1
17.8
17.2
15
13.8
13.0
Javanese
82.2
39.2
30.6
28.8
25.7
23.6
22.7
Persian
73.0
39.0
34.0
31.1
29.3
28.9
25.6
Polish
78.2
49.0
39.1
34.7
30.0
27.7
26.5
Portuguesse
50.1
29.6
20.5
20.5
15.7
14.1
21.1
Tamil
55.9
27.5
25.1
22.0
19.0
17.7
16.4
Cantonese
92.5
55.1
43.9
38.0
32.7
30.0
28.4
Turkish
82.0
51.9
44.1
42.3
34.5
29.2
26.3
Thai
59.8
41.9
32.7
23.5
20.7
18.3
17.6
Vietnamese
181.2
86.6
83.6
54.6
55.5
42.5
47.9
Pastho
114.2
108.3
81.0
79.0
76.5
63.2
64.2
Czech
81.7
50.4
39.7
36.8
31.8
29.5
36.1
Welsh
86.0
52.6
45.2
41.9
38.0
33.6
38.9
Kurdish
88.6
75.9
61.1
54.1
52.0
45.6
49.1
Zulu
98.7
72.3
64.2
61.9
57.9
51.5
51.9
Assamese
65.5
46.1
38.8
33.1
29.4
27.3
29.4
Lao
77.6
62.9
45.5
34.1
32.5
28.3
32.5
Hungarian
94.8
62.0
55.6
48.4
44.3
44.3
44.7
Georgian
130.9
117.1
106.3
87.4
74.2
54.2
45.1
Uzbek
85.3
59.6
50.9
47.3
41.0
38.4
35.2
Lithuanian
94.5
80.9
75.1
67.5
60.8
54.3
52.7
Mongolian
117.0
90.6
81.6
70.5
62.4
52.2
48.0
Kazakh
88.2
64.8
51.4
44.7
40.7
38.4
34.4
Telugu
60.3
67.0
42.8
34.6
33.5
27.9
26.1
Amharic
174.9
122.4
117.1
104.9
93.7
73.4
70.9
Cebuano
68.1
41.6
33.2
30.0
27.4
25.1
24.5
Luo
75.9
59.6
50.5
47.7
46.3
41.1
39.3
Arabic
107.5
87.0
73.6
71.3
77.5
75.6
72.1
Igbo
109.4
81.4
70.2
66.3
61.1
59.1
58.8
Ukranian
78.3
51.9
44.6
40.5
34.6
31.3
30.2
Croatian
78.7
52.3
43.3
39.3
35.1
34.2
32.2
Urdu
93.8
71.0
78.4
70.8
63.7
59.2
58.1
Estonian
87.1
55.5
45.2
40.7
35.7
31.8
30.5
Galician
52,7
37.3
28.3
26.4
22.8
21.0
20.1
Kyrgyz
79.4
54.2
44.0
38.5
34.9
32.3
31.1
Finnish
93.6
63.7
54.3
49.8
45.0
44.1
42.1
Swedish
82.0
66.1
54.2
48.9
43.1
40.7
39.1
Indonesian
73.3
39.9
31.3
29.7
24.3
22.7
22.0
24

---

OWLs: Scaling Laws for Multilingual Speech Recognition and Translation Models
Table WERCCER for the bottom 52 languages in FLEURS by OWLS training data.
Languages 0.25B
0.50B
1B
2B
4B
9B
18B
Greek
116.6
93.1
89.1
77.5
73.3
68.8
64.9
Marathi
52.0
35.6
31.6
28.4
26.7
24.0
21.9
Romanian
71.5
49.7
38.6
35.5
30.3
29.7
28.7
Slovakian
79.7
49.4
39.5
35.6
29.9
29.1
29.4
Bulgarian
85.4
67.3
59.0
55.1
48.3
45.2
43.4
Latvian
89.7
76.0
64.2
61.3
51.0
46.5
46.0
Slovenian
75.6
58.6
52.8
48.4
43.0
40.5
42.8
Gujarati
72.2
52.6
44.3
31.8
29.2
26.4
33.1
Hindi
52.9
34.5
29.2
26.5
22.6
21.2
22.7
Maltese
97.8
88.8
65.8
59.3
52.3
48.1
51.7
Kannada
68.8
57.2
35.7
25.9
25.5
21.9
23.3
Xhosa
110.5
84.7
75.3
72.5
66.4
61.2
61.2
Hausa
81.5
70.3
60.7
56.8
51.9
48.4
33.0
Irish
93.6
89.6
87.4
83.5
78.6
77.4
33.3
Maori
79.5
55.7
47.9
42.0
40.3
37.2
35.1
Danish
95.9
88.9
76.2
73.2
68.3
63.3
69.8
Kamba
94.8
82.4
74.6
71.9
69.5
65.2
64.9
Lingala
65.3
44.5
35.7
34.4
28.0
27.2
28.8
Malayalam
100.6
63.9
43.1
32.2
28.5
23.5
26.6
Occitan
75.6
63.4
57.2
53.8
48.4
47.2
47.8
Yoruba
104.8
91.3
83.5
81.1
73.2
67.6
52.1
Fulah
81.1
67.6
62.5
59.4
57.2
56.9
56.0
Swahili
50.1
27.8
22.5
22.8
21.1
18.5
17.6
Malaysian
162.8
47.0
31.3
27.3
25.6
26.0
25.6
Sindhi
126.5
90.0
75.7
68.1
63.4
56.9
54.4
Somali
96.3
87.1
79.8
77.5
72.8
71.8
70.9
Armenian
231.7
99.9
85.5
63.0
56.1
54.3
52.1
Nepali
63.3
29.7
25.3
24.8
21.5
20.0
19.8
Asturian
62.3
50.3
44.2
43.2
37.0
35.4
37.0
Cambodian
84.0
105.0
52.6
42.7
41.5
37.8
35.2
Kabuverdianu
73.1
59.3
49.4
44.9
38.1
38.2
40.6
Luganda
71.5
52.6
48.8
49.4
46.0
46.7
44.1
Norwegian
87.8
71.5
60.0
54.7
47.2
45.8
45.3
Nyanja
102.5
81.5
69.4
67.7
65.1
61.6
60.0
Serbian
85.7
71.5
71.1
54.0
46.5
47.2
47.1
Bosnian
82.0
59.3
50.0
46.1
40.8
40.2
40.7
Shona
97.7
72.6
63.3
59.8
54.5
53.5
51.7
Hebrew
508.6
132.5
100.4
87.4
88.8
81.9
76.9
Azerbaijani
100.0
77.7
67.5
62.0
57.2
55.9
53.1
Masai
71.0
51.6
41.7
37.5
34.8
31.3
30.3
Luxembourgish
94.2
86.1
75.7
71.2
65.9
61.8
60.2
Northern Sotho
95.9
77.0
61.7
58.5
51.0
49.8
49.4
Tajik
86.6
60.4
50.7
45.3
41.8
43.6
41.8
Wolo
86.7
76.8
67.7
64.6
58.1
57.3
56.7
Panjabi
172.0
62.5
45.7
42.6
38.5
37.8
35.1
Filipino
73.3
43.0
33.8
30.2
28.2
26.6
25.5
Macedonian
75.1
54.2
46.2
41.3
36.2
35.2
32.1
Oromo
94.8
89.0
102.4
80.5
79.1
77.9
71.1
Umbundu
88.5
70.2
64.4
65.3
59.4
63.5
59.1
Afrikaans
86.6
102.3
95.7
133.7
92.3
152.2
92.3
Oriya
240.4
167.8
71.4
63.0
47.9
44.8
40.1
Icelandic
108.0
102.1
91.2
85.9
77.0
74.4
72.1
25


Reviews:
[{"review_id": "x0dngA8im8", "reviewer": "Reviewer_uTgG", "summary": "This paper empirically evaluates the scaling law for speech recognition and translation in terms of training data, model size, and compute cost, using a total of 350K hours of multilingual training data.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable insights. \n\n> most of the conclusions are predictable, limiting the overall insight gained from the study\n\nWe respectfully clarify that many of our insights are not predictable. While the notion that a larger model leads to better performance is indeed obvious, showing that exact performance improvements w.r.t downstream metrics like WER can be modeled as a scaling law is a novel insight. Many previous scaling law papers model only show scaling laws w.r.t test cross-entropy loss, which is a more predictable but less actionable result for researchers.\n\n> concerns about the analysis, as the transcriptions are sourced from YouTube, and their accuracy remains uncertain...  Scaling trends observed here might not hold when using large-scale, high-quality transcribed audio data. \n\nAll of our evaluation results are performed on standard benchmarks with high-quality transcriptions, such as FLEURS or CoVoST, commonly used in evaluating many large-scale models such as [1,2]. We only use YouTube transcriptions for helping train two models in this paper (OWSL 1B and 18B 360K), which means scaling curves in figures 1-4,6,7, and 9 are not affected.\n\n## Questions \n\n1. Most of our training data is high quality [4], as it comes from common academic benchmarks. We only use noisier transcriptions for OWLS 1B 360K and OWLS 18B 360K. The purpose of Figure 2 is to show scaling patterns across all languages, and that it is unfeasible to fit a language-agnostic universal scaling law. This is a reasonable expectation, since some languages are naturally more difficult to model for ASR than others (i.e Spanish vs Chinese) [5], so they will require more training data. We acknowledge that this point was not clearly made in the paper, and will better word it in the future drafts.\n\n2. While we believe that the statement is indeed true for traditional ASR/ST models, we respectfully argue that such intuitions may not hold true for large-scale models like OWLS 18B. For example, [6] found that LLMs only require a small amount of leaked bilingual data to learn text translation. It is thus not completely out of the question that a sufficiently large ASR/ST model may be able to overcome traditional data limitations. Due to the potential openness of this question, we believe that this result is valuable evidence and proof to improve the scientific community’s understanding of large-scale models. \n\n3. A breakdown of all datasets is found in Appendix A/Table 6. The OWSM datasets are high quality [4], as they come from common academic benchmarks, such as LibriSpeech, VoxPopuli, MuST-C, and CoVoST. For YODAS, we obtain a cleaner subset from the dataset authors. They use text language identification on the transcripts (to check that the text/audio language is aligned) and CTC segmentation on the audio with a pre-trained OWSM-CTC [7] model. The worst 10% of the utterances by CTC segmentation score are then discarded. We will make this information more apparent in the main paper body in future drafts.\n\n4. We apologize for the oversight. Canary is 1B and Whisper is 1.5B. We are using Whisper Large V3, and will make it more clear in future drafts.\n\n5. We ran each model in Table 2 on LibriSpeech test-other with beam size=1, and compare the results with that of Table 2 below, showing that WER is indeed sensitive to beam size. We will include this analysis in future drafts.\n\n| Model Size | Beam Size | WER |\n|-|-|-|\n| 0.25B | 1 |  9.6\n| 0.25B | 10 |  8.3\n| 2B | 1 |  5.2\n| 2B | 4 |  4.7\n| 4B | 1 |  4.6\n| 4B | 2 |  4.5\n| 9B | 1 |  4.5\n| 9B | 2 |  4.5\n\n6. We can assert that the observed trends are consistent across multiple languages, as we perform multilingual evaluations in almost every figure/table.The only exception is beam search (Table 2), which we will rectify by including the below results on FLEURS for Spanish and Turkish. Larger models with smaller beam sizes have better WER than smaller models with larger beam sizes. The only exception is 4B and 9B, where 4B already outperforms 9B with beam size=1.\n\n| Model Size | Beam Size | Spanish | Turkish |\n|-|-|-|-|\n| 0.25B | 1 |27.5|82.0\n| 0.25B | 10 |22.9|69.5\n| 2B | 1 | 10.6|42.3\n| 2B | 4 | 9.4|34.5\n| 4B | 1 | 9.0|34.5\n| 4B | 2 | 8.4|30.6\n| 9B | 1 | 9.7|29.2\n\n[1] Robust Speech Recognition via Large-Scale Weak Supervision (Radford et al., ICML 2023).\n\n[2] Gemini: A Family of Highly Capable Multimodal Models (Gemini Team., 2024).\n\n[4] On the effects of heterogeneous data sources on speech-to-text foundation models (Tian et al., Interspeech 2024).\n\n[5] Language Complexity and Speech Recognition Accuracy: Orthographic Complexity Hurts, Phonological Complexity Doesn't. (Taguchi et al., 2024).\n\n[6] Searching for Needles in a Haystack: On the Role of Incidental Bilingualism in PaLM’s Translation Capability (Briakou et al., ACL 2023).\n\n[7] OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identification (Peng et al., ACL 2024).", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "N/A", "theoretical_claims": "This paper is primarily experiment-driven, this study does not present strong theoretical claims.", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "This paper investigates the scaling law for speech, which aligns with the scaling law observed in text data for large language models.", "essential_references_not_discussed": "The reference looks good to me.", "other_strengths_and_weaknesses": "Strength\n  This paper presents a valuable study, providing a comprehensive analysis of the effects of data scale, model size, and computational cost on speech recognition and translation. The extensive experimental evaluation is highly appreciated.\n\nWeakness\n  The paper does not introduce novel ideas or algorithms but instead presents a series of experiments. Moreover, most of the conclusions are predictable, limiting the overall insight gained from the study. Additionally, I have concerns about the analysis, as the transcriptions are sourced from YouTube, and their accuracy remains uncertain, which might affect the conclusion drawn in the paper.\n\nGiven the paper’s clear strengths and potential weaknesses, I would rate it as a weak acceptance.", "other_comments_or_suggestions": "See below", "questions_for_authors": "1. The conclusions drawn from the study may not be entirely reliable due to the noisy nature of the training data. Scaling trends observed here might not hold when using large-scale, high-quality transcribed audio data. This could also explain the discrepancy in Figure 2, whether the authors inspect the relationship between WER and the amount of training data that does not fully align with expected scaling laws.\n2. The experimental results largely align with expectations and do not provide particularly surprising insights. E.g. the statement, “while parameter scaling can significantly improve ST performance, it cannot overcome cases where there is inherently insufficient data to learn the task”, is intuitive and aligns with prior expectations.\n3. The paper does not specify the quality of transcriptions used for speech recognition and translation. The authors should describe how the transcriptions were obtained for both Yodas and OWSR datasets, as transcription quality significantly impacts scaling trends.\n4. Model Size in Table 1: In Table 1, listing the model sizes of Canary and Whisper would provide better clarity. I assume the authors are using Whisper Large v3—please confirm.\n5. The relevance of Table 2 is unclear. Given that WER is generally not highly sensitive to beam size, I would expect minimal variations. If this assumption is incorrect, the authors should provide supporting WER values to demonstrate the impact of beam size.\n6. Since this is a multilingual ASR model, observations based on a single language may not generalize across different languages. The authors should clarify whether the observed trends are consistent across multiple languages.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DomN0qAWYR", "reviewer": "Reviewer_A8Vp", "summary": "This paper investigates scaling laws for multilingual, multi-task speech-to-text models. \n\nTo achieve this, the authors introduce OWLS, a collection of ASR/ST models ranging from 0.25B to 18B parameters, with the 18B model being the largest publicly known ASR/ST model to date.\n\nThe study examines three dimensions of scaling, model size, data size, and compute budget, to understand how each factor influences performance. Additionally, the paper highlights emergent abilities that only appear when the model reaches a certain scale.\n\nThe OWLS project is fully open-sourced, with the training code, logs, and checkpoints set to be released to facilitate reproducibility and further research.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank your for your insights and comments. We organize our response by section:\n\n## Claims And Evidence:\n\n2. We understand the reviewer’s concern. While our definition was indeed focusing on the fact that scaling leads to a better model for many languages, we do acknowledge that this definition may be too narrow. We will instead rephrase the fairness claim to “Scaling to larger models can improve ASR performance across the board, in both low and high resource languages, with significant improvements in the former category.”\n\n3. We have included the accompanying audio for the mondegreen samples in Appendix Table 9 here:  https://anonymous.4open.science/w/owls-samples-88B2/ , which we hope better illustrate the phenomena. If there any particular items that can be further clarified, we will be happy to make those changes as well.\n\n4. We have included a few samples on example ICL generations https://anonymous.4open.science/r/owls-samples-C53F/README.md , which should better illustrate the improvements in ASR output as the number of in-context examples increase. While there is not a clear monotonic trend, we note that there are 3 clear clusters in ICL CER that are aligned with model size. 0.25B and 0.5B both have ~33.8 CER, 1B to 4B all have ~31.5 CER, and 9B to 18B have ~27.7 CER.\n\n## Experimental Designs Or Analyses:\n\n> For the Mondegreen section, I am not particularly sure how PPL/MOS relates to ASR performance, or in other words, how they are useful. Does this phenomenon actually help reduce WER/CER, or is it simply an artifact of large-scale training? If not, why is this ability important enough to be discussed in this paper?\n\nPPL/MOS do not have a relationship with ASR performance, since the only goal of these metrics is to measure the English coherence of the mondegreen generation. We note that this phenomenon likely does not have any impact on overall WER/CER, and is probably indeed an artifact of scaling. However, we believe that it is important to discuss and present this evaluation, because it allows researchers to better understand the properties of neural networks that emerge at scale, and how it relates to the human perceptions of spoken language. By including this evaluation, we can show that there is much we do not understand about ASR models and how they scale, opening up potential new areas of research that were previously not possible due to lack of access to such models.", "claims_and_evidence": "This paper makes several key claims:\n\n1. Scaling Laws: The most significant claim is that the performance of multilingual ASR/ST models follows a (language-specific) scaling law. This is well-supported by Figures 3, 6, and 7, which demonstrate a consistent relationship between model size, data size, and performance metrics.\n2. Benefits for Low-Resource Languages: The paper also claims that larger multilingual ASR models improve performance for low-resource languages by reducing WER. This is generally well-supported by the results. But I'm not sure how the paper demonstrates that it \"mitigates bias and improves the fairness of speech technologies.\" Without further analysis, this claim seems overstated.\n3. Emergent Abilities: The section on emergent abilities could be clearer, particularly the discussion of mondegreen phenomena.\n4. In-Context Learning (ICL): The ICL results appear inconsistent, lacking a clear trend across different model sizes and settings.", "methods_and_evaluation_criteria": "Yes, WER/CER and BLEU are both standard criteria for measuring ASR and ST performance.", "theoretical_claims": "Not applicable, as this is an empirical paper.", "experimental_designs_or_analyses": "Yes. \n\nFor the Mondegreen section, I am not particularly sure how PPL/MOS relates to ASR performance, or in other words, how they are useful. Does this phenomenon actually help reduce WER/CER, or is it simply an artifact of large-scale training? If not, why is this ability important enough to be discussed in this paper?", "relation_to_broader_scientific_literature": "This paper shows that multilingual ASR/ST models improve predictably with scale, especially for low-resource languages. It also explores emergent abilities like mondegreens and code-switching, though these findings need stronger proof. Additionally, it tests In-Context Learning (ICL) in ASR, but the results are unclear. Overall, the paper applies scaling laws to speech models and raises interesting questions about model behavior at large scales.", "essential_references_not_discussed": "Not that I am aware of.", "other_strengths_and_weaknesses": "This paper is well-written and easy to read. One minor comment is that the colors used in the figure are very friendly to color-blind people.", "other_comments_or_suggestions": "No.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U8omAu5hgl", "reviewer": "Reviewer_MsZt", "summary": "The paper introduces OWLS, a suite of multilingual speech recognition and translation models ranging from 0.25B to 18B parameters. It systematically studies scaling laws for speech tasks, demonstrating how model, data, and compute scaling impact performance. The paper claims that larger models improve low-resource language performance and exhibit emergent abilities, and it presents empirical scaling laws derived from experiments.\n\n## update after rebuttal\n\nThe authors’ response has addressed several of my initial concerns; however, it does not substantially change my overall evaluation of the work. The paper primarily focuses on scaling a standard ASR model rather than introducing novel methodological contributions or architectural innovations.\n\nWhile I acknowledge the significance of the presented experiments to the ASR community, the results neither revealed surprising findings nor provided sufficient new insights. I was particularly hoping for a more in-depth and reflective discussion on how the Audio-LLM paradigm contrasts with traditional ASR approaches—an aspect that remains unaddressed in the current version.\n\nAfter reviewing comments from other reviewers, I noted that similar concerns were shared. As such, I will maintain my original evaluation.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review and insights.\n\n> some details about data preprocessing are missing\n\nReviewer uTgG raised similar concerns. Please refer to our response to their question 3.\n\n> code-switching improvements appear inconsistent across languages.\n\nSince we are performing multilingual multi-domain training and evaluation, some per-language fluctuation is expected. However, the average code-switching CER clearly decreases with increases in model size. We will better illustrate this by including a line for the average performance, along with a table with raw scores in the appendix.\n\n> the role of domain mismatch in dataset expansion should be further explored\n\nWhile we agree that the domain mismatch should be further explored (and plan to do so in future work), we believe that further experiments are outside this paper's scope, which is focused on multilingual neural scaling rather than studying the role of data domains. The domain shift finding is a consequence of doing such large scaling analyses, rather than a hypothesis we explicitly designed an experiment to test.\n\n> The analysis of emergent abilities would be stronger with qualitative examples.\n\nTable 9 includes example mondegreen generations. We have uploaded the accompanying audio for reference in https://anonymous.4open.science/w/owls-samples-88B2/ . We also added sample generations for ICL (Table 5): https://anonymous.4open.science/r/owls-samples-C53F/README.md . We will include these in the final manuscript.\n\n> More details on hyperparameter selection and convergence criteria \n\nWe use the same hyper-parameters as OWSM v3.1, which generalized to all model sizes. We did not have explicit convergence criteria, but instead cut off training after 675K steps for fairness and budgeting. All models appear to have converged by this point in both validation and training loss. We will include convergence plots in the final draft.\n\n> does not discuss some recent multilingual speech models  (SeamlessM4T and MMS)\n\nWe reference Seamless in 2.3 and discuss how they differ from Whisper-style models. We also reference MMS at a high-level in 2.3, but will update the draft to make more direct comparisons. WER comparisons with Seamless are shown in the able below.\n\n> performance appears relatively uncompetitive compared to Audio-LLMs, such as Qwen2Audio...does not provide a direct comparison with other efficiency-focused models, such as SenseVoice\n\nBelow are comparisons with these models for 3 languages using FLEURS and non-FLEURS test sets. Chinese/Japanese use CER, English uses WER. We use the official reported scores when possible, designated by a *, otherwise we run inference ourselves. Since SenseVoice-L is not publicly available, we only include their official results.\n\n|Model| FLEURS zh | FLEURS en | FLEURS ja | AISHELL 1 zh | test-clean en | Reazon ja |\n|-|-|-|-|-|-|-|\n|OWLS 1B|13.8 | 9.7 | 9.2 | 6.2 | 2.3 | 7.8\n|OWLS 9B|11.6 | 8.5 | 7.7 | 4.8 | 1.9 | 7.3 \n|OWLS 18B|10.6 | 7.7 | 7.2 | 4.8 | 2.0 | 7.5\n|OWLS 18B v2|10.1 | 6.8 | 6.7 | 4.8 | 2 | 7.2\n|Qwen2Audio|7.5*|9.4| 20.1 | 8.7 | 1.6* | 50.0\n|SenseVoice S|9.6|10.3| 13.1 | 3.0* | 3.2 *| 37.1\n|SenseVoice L||||2.1*|2.6*\n|Seamless Medium|15.7*|8.3*|15.9*|9.6|4.2|34.9\n|Seamless Large| 17.0*|7.3*|17.6*|8.7|3.7|36.6\n\nOWLS models are competitive with Qwen2Audio:  OWLS 18B v2 outperforms QwenAudio2 on 4/6 test sets. We note that we were also unable to reproduce the results in the Qwen2Audio paper, since they did not release their prompts (we obtained a CER of 14.2 on FLEURS zh with Qwen2Audio, while official is 7.5 CER), further emphasizing the importance of the open nature of the OWLS suite. We found that Qwen2Audio was more susceptible to hallucinations compared to the other models, which is in line with previous studies on LLMs for ASR [1].\n\n## Questions\n\n1. OWLS is similar to Seamless and SenseVoice-L in terms of algorithmic efficiency, since they're all encoder-decoders.\n\n2. All benchmarks we use are in the manuscript. Code-switching and Quechua are out-of-domain.\n\n3. While it is difficult to compare the training cost to all SOTA models due to the proprietary nature of training details, the below table shows a comparison for the models where the information is available:\n\n| Model | Parameters | GPUs | Training Hours | \n|-|-|-|-|\n|Canary|1B| 128xA100 80GB |48\n|OWSM v3 |900M| 64xA100 40GB |240 \n|OWSM v3.1|1B| 64xA100 40GB |384\n|OWSM-CTC |1B| 64xA100 40GB|300 \n|OWLS |1B|24xH100 80GB |116\n|OWLS |18B| 48xH100 80GB |405 \n\nNote that training hours are not fully comparable due to differences in GPU and cluster environment. Canary also uses a pre-trained speech encoder to decrease the training time, which is not considered in the final budget. Nevertheless, we believe the OWLS training is well-optimized, as we had a limited compute budget. All of these optimizations will be open-sourced.\n\n[1] Performance evaluation of SLAM-ASR: The Good, the Bad, the Ugly, and the Way Forward. (Kumar et al., ICASSP 2025 SALMA Workshop).", "claims_and_evidence": "The claims about scaling benefits (especially for low-resource languages) are well-supported by experimental results.\nThe study effectively demonstrates the impact of scaling on WER and BLEU scores across different settings.\nWeakness:\nWhile the paper suggests that emergent abilities arise with larger models, some results (e.g., code-switching improvements) appear inconsistent across languages. More justification is needed.", "methods_and_evaluation_criteria": "The methodology is well-structured, following systematic scaling experiments.\nThe use of publicly available data enhances reproducibility, but some details about data preprocessing are missing.", "theoretical_claims": "The paper effectively extends scaling laws to speech tasks, building on prior work in text and vision. The derivation of scaling laws is reasonable, but it would benefit from a more formal theoretical justification.\nSome empirical observations (e.g., emergent abilities) could be further contextualized within existing theories.", "experimental_designs_or_analyses": "The experiments are well-controlled, and the choice of different model sizes is appropriate. The scaling trends in WER and BLEU are insightful, but the role of domain mismatch in dataset expansion (e.g., adding YODAS data) should be further explored. The analysis of emergent abilities is interesting but would be stronger with qualitative examples.", "relation_to_broader_scientific_literature": "The paper situates itself well within prior scaling laws research in NLP and speech.", "essential_references_not_discussed": "The paper does not discuss some recent multilingual speech models (e.g., SeamlessM4T and MMS from Meta).", "other_strengths_and_weaknesses": "Strengths:\n\n* The open-source approach enhances reproducibility.\n\n* The paper provides a comprehensive scaling analysis for speech models.\n\n* Strong empirical results demonstrate the model’s efficiency.\n\nWeaknesses:\n\nMy primary concern is that the performance of this work appears relatively uncompetitive compared to recent advancements in Audio-LLMs, such as Qwen2-Audio. While this paper primarily focuses on Automatic Speech Recognition (ASR) and Speech Translation (ST), Audio-LLMs support a broader range of tasks and offer wider application scenarios.\n\nI acknowledge that a notable strength of this study lies in its efficiency, achieved through hybrid CTC/Attention training. However, it remains uncertain whether this approach represents the most optimal solution by current standards. Additionally, the paper does not provide a direct comparison with other efficiency-focused models, such as SenseVoice, which raises concerns about the completeness of the evaluation.\n\nTherefore, despite the extensive experiments conducted, the paper lacks a compelling demonstration of its results, which limits its overall impact.", "other_comments_or_suggestions": "Correct the legend in Figure 1: The blue color in the legend is incorrectly assigned to Whisper, whereas the caption states that it represents OWLS. Ensure that the legend accurately reflects the intended model categorization to avoid confusion.", "questions_for_authors": "1. How does OWLS compare to existing large-scale models like SeamlessM4T and SenseVoice in terms of efficiency?\n\n2. Have you tested the generalization of OWLS on out-of-domain data beyond the current benchmarks?\n\n3. How does the training cost of OWLS compare to other state-of-the-art multilingual speech models?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bWeCzWeECb", "reviewer": "Reviewer_ogkW", "summary": "This paper investigates the effect of model size and dataset size on multilingual Automatic Speech Recognition (ASR) and Speech Translation (ST) tasks for 150 languages. The model sizes vary from 0.25B to 18B parameters. The WER vs. size curves are fitted into power law functions and the correlations are reported in terms of R^2 value. In addition, the paper test the trained models on unseen tasks or languages to show the emerging capabilities of these large multilingual models.   \nExperiments are performed on up to 360k hours of speech across 150 languages. General trends are intuitive in that larger model sizes and larger data sizes lead to lower WER and higher BLEU scores.\n\n## Update after rebuttal\nThe paper will benefit from some revision to clarify some of the points mentioned in the rebuttal comment. I would like to keep my initial score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank your for your insights and comments. We organize our response by section:\n\n## Supplementary Material\n\nWe believe that we have all the possible information for the embedding extraction process in the appendix, although we acknowledge that the description may be unclear and hard to follow through text alone. For further clarity, we will add the following figures to the manuscript to better visualize the embedding extraction method: https://anonymous.4open.science/r/owls-samples-13A3/icl_figure.md . The code to perform ICL will also be released for reproducibility.\n\n## Claims And Evidence:\n\n1. We understand the reviewer’s concern. While our definition was indeed focusing on the fact that scaling leads to a better model for many languages, we do acknowledge that this definition may be too narrow. We will instead rephrase the fairness claim to “Scaling to larger models can improve ASR performance across the board, in both low and high resource languages, with significant improvements in the former category.”\n\n2. We perform inference with the model after 15K, 30K, 60K, 120K, 240K, 480K training steps, and use these datapoints to fit a power-law function. We then extrapolate the power law to 675K steps, which is where the final checkpoint is taken from, and compare it to the true WER of the final checkpoint. In Fig 9, each datapoint corresponds to each checkpoint that we evaluated (note that some early checkpoints are not visible since we cut off the chart at 100 WER). The training FLOPS of each checkpoint is then calculated by profiling each model size for a few steps and scaling the result to 15K-675K steps. We will adjust the manuscript to make these details more clear.\n\n3. We also arrived at the conclusion that the clusters are largely distinguished by character set. However, we believe that linguistic similarity likely also plays a role. Russian, despite using Cyrillic letters, is more linguistically similar to English and the bottom cluster languages (which are all European), than the top cluster languages. We believe there may be some miscommunication for Arabic due to our presentation format. Arabic is the red line in the middle of the top cluster and certainly suffers from higher CERs. We will update the chart with different shapes for each language to make it less confusing.\n\n## Other Strengths And Weaknesses\n\n> Fig. 4, was there an experimental issue in the AMI (or Tedlium, cannot quite tell from the color) ASR runs?\n\nWe addressed below in our response to Question 3.\n\n> In-context learning is happening to some extent but the given example may not be sufficient to claim this ability.\n\nWe have included a few samples on example ICL generations https://anonymous.4open.science/r/owls-samples-C53F/README.md , which should better illustrate the improvements in ASR output as the number of in-context examples increase.  The most obvious change is from pure zero-shot (k=0) to on-shot ICL (k=1), where the model changes from outputting a completely wrong language / character set to text that looks like Quechua. With more examples, the models also start to learn more complex phone-to-orthographic mapping (\"ai\" -> \"ay\") and white space placement (\"ruraikunatukunapa -> ruraikuna tukunapa\").\n\n## Other Comments Or Suggestions\n\n1. Yes for both. Apologies for the mistake. We will fix this typo.\n2. Yes, the caption should also read PN-CER. We will fix this typo.\n\n## Questions For Authors\n\n1. We believe that the large increases in WER/CER are due to two main factors: 1) domain mismatches between the additional YouTube data and FLEURS evaluation data and 2) interference from similar languages/dialects (ie. Chinese and Cantonese). \n\n2. Addressed above in \"Claims and Evidence\" point 1.\n\n3. The fluctuation is in AMI, where the data is largely spontaneous and conversational, as the audio is sourced from business meetings. Since our training data is largely read speech and multilingual, it can be expected that there will be some small fluctuations on individual test sets despite average performance improving with scale. We will make the figure easier to read by using different shapes for the points of each dataset.\n\n4. Addressed above in \"Claims and Evidence\" point 2.\n\n5. Addressed above in \"Claims and Evidence\" point 3.", "claims_and_evidence": "Claims that are somewhat arguable:\n\n1) Claim: \"Larger models can mitigate bias and improve the fairness of speech technologies.\". Argument: This is a very generic statement that may or may not be true depending on how you define fairness. In this context, the authors might have meant language coverage and fairness due to have reasonably well ASR model for many languages. However, this is a very narrow definition. There could be multiple dimensions in which the fairness of ASR models can be analyzed and only the analyses in this paper are not sufficient to make such a big fairness claim. \n\n2) I could not quite follow how Section 4.3. tries to estimate the final checkpoint WER from the initial checkpoint WER, and also how this is reflected in Fig. 9. Hence, I could not check the validity of the following: \"one can reasonably predict the final WER of the model given the WERs of initial checkpoints.\"\n\n3) Code-switching Fig. 11. looks interesting, it seems that there are two clusters of languages, low CER ones with a clearer pattern and high CER languages with non-uniform patterns over model size. In the text, some of the difference is attributed to the character set of these languages En - PT being Latin-only versus En- ZH using different characters. However, the results seem a little confusing. For example, why En - AR or En - RU does not suffer from a similar problem?", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "NA", "experimental_designs_or_analyses": "Experiments look valid.", "relation_to_broader_scientific_literature": "The paper investigates large and extra large ASR and ST models in the multilingual scenario. Especially, the differences from 1B to 9B or 18B model that we have not empirically seen in previous studies can lead to some additional research questions for serving ASR model to low-resourced languages. \n\nOnce the source code and the models are released, this OWLS suite can be used to setup baselines for other studies easily which would be valuable.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "- Strength: OWLS can be useful for both ASR and ST community with the analysis it provides. Once the code and the setup is available, it might be helpful for researchers, too.\n\n- Weaknesses: Some arguments are over-generalized, but they do not always hold. For example, \n- Fig. 4, was there an experimental issue in the AMI (or Tedlium, cannot quite tell from the color) ASR runs? \n- The generic fairness claim (discussed above)\n- In-context learning is happening to some extent but the given example may not be sufficient to claim this ability.", "other_comments_or_suggestions": "1) Fig. 3 and Fig. 5, did you mean the Tamil language when the figure says Tami? \n\n2) What is PN-CER in Fig. 10 caption? Is it N-CER which was mentioned in the text?", "questions_for_authors": "1) Could you please explain why WER/CER is increasing with increasing data after a certain point in Fig. 5?\n\n2) Could you please rephrase the fairness claim in Section 4.1. Multilingual ASR?\n\n3) Fig. 4., could you please explain why the topmost curve is fluctuating? Also in this figure, it is hard to tell if it AMI or Tedlium or Gigaspeech (the legend font size makes it difficult).\n\n4) How does Section 4.3. estimate the final checkpoint WER from the initial checkpoint WER? How this is reflected in Fig. 9?\n\n5) Code-switching Fig. 11. looks interesting, it seems that there are two clusters of languages, low CER ones with a clearer pattern and high CER languages with non-uniform patterns over model size. In the text, some of the difference is attributed to the character set of these languages En - PT being Latin-only versus En- ZH using different characters. However, the results seem a little confusing. For example, why En - AR or En - RU does not suffer from a similar problem?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HIEGBmbiHn", "reviewer": "Reviewer_QjdC", "summary": "This paper introduces **OWLS, open-source Whisper-style models for multilingual speech recognition (ASR) and translation tasks**, and releases the trained models. The authors empirically derive scaling laws for multilingual speech processing by training OWLS models at varying scales. Experimental results demonstrate that the large OWLS models follow the scaling laws across different languages and tasks and exhibit the emergent abilities like orthographic understanding or code-switching.\n\n### Update after rebuttal\nAfter going through all the reviews from other reviewers as well as author responses, I'm still concerned with the significance of the paper's findings which are mostly expected. But I like the paper's contribution in scaling ASR models, since there have not been such works that show extensive analyses as this study. Thus, I maintain my score as weak accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank your for your insights and comments. \n\n>  I'm not quite convinced with the claim related to ICL ability. Whisper-style models like OWLS are trained primarily for ASR and translation, not instruction-following...\n\nWhile Whisper-style models are not trained for instruction-following, instruction-tuning is not a requirement for models to have the capability to perform ICL, although it will certainly improve their ICL capabilities by minimizing the distribution shift between prompts and pre-training data [1]. Early LLMs, such as GPT2 and GPT3 [2,3], were not instruction-tuned and were only trained on pure language modeling. ICL can be described as a byproduct of auto-regressive modeling, which is how ASR/ST are formulated in auto-regressive encoder-decoder frameworks such as Whisper. ICL is thus a reasonable expectation for the OWLS models because they may be able to use in-context examples (speech-text pairs) as a prior for mapping sounds to words. Examples we provide in our \"Strengths And Weaknesses\" response to Reviewer ogkW illustrate this capability. [4] also shows preliminary results for this, although we believe that their weaker results are due to the much smaller model size of the tested models. \n\n> One limitation is that the method lacks detailed justification for scaling the model specifically up to 18B parameters...\n\nWe chose the upper of 18B primarily due to budget constraints. At 18B, we are still able to perform training without sharding the model parameters across GPUs, which is why we are able to maintain such high training efficiency (training time of 17 days). If we performed model sharding, the expected training time would have been ~2 months, which was unfeasible. Additionally, we were still able to maintain the same mini batch size at 18B without performing gradient accumulation at our compute budget, as gradient accumulation would have also slowed down training significantly. We want to maintain identical batch sizes to minimize experimental variance from the hyper-parameters. \n\nIn terms of the model architecture, we equally allocate parameters between the encoder and decoder for two main reasons:\n\n- It is the approach adopted by current SOTA models such as Whisper\n- It was found to be the optimal setting in encoder-decoder scaling for NLP [5,6]\n\nFinally, we chose the Transformer architecture due to its ease of scaling. We experimented with other architectures, such as Conformer, but found it was extremely difficult to tune the hyper-parameters and get the model to converge. This is a similar finding to that of Zhang et al., when they trained Google USM [7]. Other works [8] has shown that the advantages of Conformer diminish as the model size grows. Overall, we believe that our design choices are indeed optimal when considering both effectiveness and efficiency. We will add all of this information in the final draft.\n\n> it is pretty obvious to discover the scaling laws with language-specific model size and scaling data with diversity\n\n> However, a clear weakness is the minimal methodological novelty... Moreover, most findings (i.e., better performance as the model, data, or compute scales) are obvious...\n\nWe respectfully argue that many insights of the paper are not predictable. While the notion that a larger model/data/compute leads to better performance is indeed obvious, our results showing that the exact performance improvements w.r.t downstream metrics like WER can be modeled as a scaling law is a novel insight. Many previous scaling law papers model only show scaling laws w.r.t test cross-entropy loss, which is a more predictable but less actionable result for practitioners and researchers. We also want to emphasize that the degree to which performance improves at scale is also non-obvious. For example, En to X ST BLEU increases by a very large factor of 21.8% on average when scaling from 2B (already larger than most SOTA speech models like Whisper Large (1.5B) and Seamless (1.2B)) to 18B parameters, showing that there is much to gain at scale.\n\n> Every figures contain excessively small font sizes, making them difficult to read..\n\nWe apologize for the inconvenience. We will make sure to rectify this in future editions.\n\n[1] An Explanation of In-context Learning as Implicit Bayesian Inference (Tie et al., ICLR 2022)\n\n[2] Language Models are Unsupervised Multitask Learners (Radford et al., 2019)\n\n[3] Language Models are Few-Shot Learners (Brown et al., 2020)\n\n[4] Can Whisper perform speech-based in-context learning? (Wang et al., ICASSP 2024)\n\n[5] Scaling Laws for Neural Machine Translation (Ghorbani et al., ICLR 2022)\n\n[6] Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Raffel et al., 2019)\n\n[7] Hearing the AGI: from GMM-HMM to GPT-4o. (Yu Zhang, 2024). https://youtu.be/pRUrO0x637A.\n\n[8] Revisiting Convolution-free Transformer for Speech Recognition (Hou et al., Interspeech 2024)", "claims_and_evidence": "The paper provides reasonable evidences supporting its claim regarding the difficulties in fitting universal scaling laws for language-agnostic data, which is interesting. However, I'm not quite convinced with the claim related to *ICL ability*. Whisper-style models like OWLS are trained primarily for ASR and translation, not instruction-following. These models do not inherently possess instruction-following capabilities; therefore, it is unreasonable to expect ICL behaviors from them. Also, it is pretty obvious to discover the scaling laws with language-specific model size and scaling data with diversity.", "methods_and_evaluation_criteria": "The OWLS training corpus is a simple combination of two existing datasets (OWSM and YODAS), which is not special. However, dividing the scaling analysis clearly into model size, data size, and compute budget is an effective approach. \n\nOne limitation is that the method lacks detailed justification for scaling the model specifically up to 18B parameters. While experimenting with large-scale models is valuable, the reasoning behind architectural choices, training configurations, or design principles at this scale is insufficiently elaborated. It remains unclear if the chosen design is optimal in scaling.", "theoretical_claims": "Not provided.", "experimental_designs_or_analyses": "The experimental design is well-structured, providing comprehensive analyses. Particularly noteworthy is the thoughtful experimental setup aimed at investigating emergent abilities.", "relation_to_broader_scientific_literature": "The paper relates to existing literature on ASR and AST models, and more broadly, to LLM scalability. To my knowledge, there hasn't been previous works on comprehensive scaling-law analyses at this large model scale, which is a meaningful contribution to this literature.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "A notable strength is the comprehensive scaling-law analysis, particularly as this appears to be among the first efforts to investigate large-scale models (up to 18B) specifically for ASR and AST tasks. The experimental analyses—especially the emergent abilities—are well set-up and insightful.\n\nHowever, a clear weakness is the minimal methodological novelty, given the dataset merely combines existing corpora and there is not enough grounding on the scalable model design. Moreover, most findings (i.e., better performance as the model, data, or compute scales) are obvious and not significantly groundbreaking.", "other_comments_or_suggestions": "Every figures contain excessively small font sizes, making them difficult to read. Additionally, distinguishing elements solely by color is difficult; varying shapes or increasing font sizes is strongly recommended.", "questions_for_authors": "1. The authors noted training using timestamp predictions—are these timestamps available for all datasets? If not, how was missing timestamp information handled?\n\n2. Are there any results of training the encoder other than Whisper-style encoders?\n\n3. In Sec. 5.1, the authors investigated scaling with a fixed computational budget. I'm curious if the authors have also explored model performance and scaling trends when varying beam sizes on large models without budget constraints?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces OWLS, a suite of multilingual speech recognition and translation models with various sizes (from 250M to 18B parameters). These models are trained using up to 360k hours (OWSM+YODAS) multilingual speech covering 150 languages. The authors systematically investigate the scaling behavior of multilingal speech recogntion and translation with respect to data size, model size and compute budget. In addition, the authors also investigate the emergent capabilities of large models such as orthographic understanding,
code-switching and mondegreens. Overall, the topic under investigation is important and the work is interesting. It is among the first investigations in this line of research at this scale which may have its value to the speech community. The rebuttal clears up most of the concerns and questions raised by the reviewers. There are some lingering concerns though. For example, the paper is basically experiment heavy without introducing any novel ideas or techniques.
Decision: Accept (poster)