Robust Speech Recognition via Large-Scale Weak Supervision

Alec Radford * 1 Jong Wook Kim * 1 Tao Xu 1 Greg Brockman 1 Christine McLeavey 1 Ilya Sutskever 1

Abstract
We study the capabilities of speech processing
systems trained simply to predict large amounts oftranscripts of audio on the internet. When scaled
to 680,000 hours of multilingual and multitask
supervision, the resulting models generalize well
to standard benchmarks and are often competitive
with prior fully supervised results but in a zeroshot transfer setting without the need for any finetuning. When compared to humans, the models
approach their accuracy and robustness. We arereleasing models and inference code to serve asa foundation for further work on robust speechprocessing.

1. Introduction
Progress in speech recognition has been energized by thedevelopment of unsupervised pre-training techniques exemplified by Wav2Vec 2.0 (Baevski et al., 2020). Since thesemethods learn directly from raw audio without the need forhuman labels, they can productively use large datasets of unlabeled speech and have been quickly scaled up to 1,000,000
hours of training data (Zhang et al., 2021), far more than the
1,000 or so hours typical of an academic supervised dataset.
When fine-tuned on standard benchmarks, this approach
has improved the state of the art, especially in a low-datasetting.
These pre-trained audio encoders learn high-quality representations of speech, but because they are purely unsupervised they lack an equivalently performant decoder mapping
those representations to usable outputs, necessitating a finetuning stage in order to actually perform a task such as
speech recognition1 . This unfortunately limits their usefulness and impact as fine-tuning can still be a complex
process requiring a skilled practitioner. There is an additional risk with requiring fine-tuning. Machine learning*

Equal contribution 1 OpenAI, San Francisco, CA 94110, USA.
Correspondence to: Alec Radford <alec@openai.com>, Jong
Wook Kim <jongwook@openai.com>.
1
Baevski et al. (2021) is an exciting exception - having developed a fully unsupervised speech recognition system

methods are exceedingly adept at finding patterns within a
training dataset which boost performance on held-out datafrom the same dataset. However, some of these patterns arebrittle and spurious and don’t generalize to other datasetsand distributions. In a particularly disturbing example, Radford et al. (2021) documented a 9.2% increase in object
classification accuracy when fine-tuning a computer visionmodel on the ImageNet dataset (Russakovsky et al., 2015)
without observing any improvement in average accuracy
when classifying the same objects on seven other natural
image datasets. A model that achieves “superhuman” performance when trained on a dataset can still make many
basic errors when evaluated on another, possibly precisely
because it is exploiting those dataset-specific quirks thathumans are oblivious to (Geirhos et al., 2020).
This suggests that while unsupervised pre-training has improved the quality of audio encoders dramatically, the lackof an equivalently high-quality pre-trained decoder, combined with a recommended protocol of dataset-specific finetuning, is a crucial weakness which limits their usefulness
and robustness. The goal of a speech recognition systemshould be to work reliably “out of the box” in a broad range
of environments without requiring supervised fine-tuning ofa decoder for every deployment distribution.
As demonstrated by Narayanan et al. (2018), Likhomanenko
et al. (2020), and Chan et al. (2021) speech recognition systems that are pre-trained in a supervised fashion across many
datasets/domains exhibit higher robustness and generalize
much more effectively to held-out datasets than models
trained on a single source. These works achieve this bycombining as many existing high-quality speech recognition datasets as possible. However, there is still only amoderate amount of this data easily available. SpeechStew(Chan et al., 2021) mixes together 7 pre-existing datasets
totalling 5,140 hours of supervision. While not insignificant, this is still tiny compared to the previously mentioned
1,000,000 hours of unlabeled speech data utilized in Zhang
et al. (2021).
Recognizing the limiting size of existing high-quality supervised datasets, recent efforts have created larger datasets forspeech recognition. By relaxing the requirement of goldstandard human-validated transcripts, Chen et al. (2021) and
Galvez et al. (2021) make use of sophisticated automatedRobust Speech Recognition via Large-Scale Weak Supervision2

pipelines to scale weakly supervised speech recognition
to 10,000 and 30,000 hours of noisier training data. This
trade-off between quality and quantity is often the rightcall. Although understudied so far for speech recognition,
recent work in computer vision has demonstrated that moving beyond gold-standard crowdsourced datasets such as
ImageNet (Russakovsky et al., 2015) to much larger but
weakly supervised datasets significantly improves the robustness and generalization of models (Mahajan et al., 2018;
Kolesnikov et al., 2020).

the speech recognition pipeline since it removes the needfor a separate inverse text normalization step in order toproduce naturalistic transcriptions.

Yet these new datasets are only a few times larger than thesum of existing high-quality datasets and still much smaller
than prior unsupervised work. In this work we close that
gap, scaling weakly supervised speech recognition the nextorder of magnitude to 680,000 hours of labeled audio data.
We call our approach Whisper2 . We demonstrate models
trained at this scale transfer well to existing datasets zeroshot, removing the need for any dataset-specific fine-tuning
to achieve high-quality results.
In addition to scale, our work also focuses on broadening the scope of weakly supervised pre-training beyond
English-only speech recognition to be both multilingual andmultitask. Of those 680,000 hours of audio, 117,000 hours
cover 96 other languages. The dataset also includes 125,000
hours of X→en translation data. We find that for sufficiently
large models there is no drawback and even benefits to joint
multilingual and multitask training.
Our work suggests that simple scaling of weakly supervised
pre-training has been underappreciated so far for speech
recognition. We achieve these results without the need forthe self-supervision or self-training techniques that havebeen a mainstay of recent large-scale speech recognition
work. To serve as a foundation for further research on robust
speech recognition, we release inference code and models at
the following URL: https://github.com/openai/
whisper.

2. Approach
2.1. Data Processing
Following the trend of recent work leveraging web-scale
text from the internet for training machine learning systems,
we take a minimalist approach to data pre-processing. In
contrast to a lot of work on speech recognition, we train
Whisper models to predict the raw text of transcripts without
any significant standardization, relying on the expressiveness of sequence-to-sequence models to learn to map between utterances and their transcribed form. This simplifies
2

If an acronym or basis for the name is desired, WSPSR standing for Web-scale Supervised Pretraining for Speech Recognition
can be used.

We construct the dataset from audio that is paired with transcripts on the Internet. This results in a very diverse dataset
covering a broad distribution of audio from many different
environments, recording setups, speakers, and languages.
While diversity in audio quality can help train a model to berobust, diversity in transcript quality is not similarly beneficial. Initial inspection showed a large amount of subpar
transcripts in the raw dataset. To address this, we developed
several automated filtering methods to improve transcript
quality.
Many transcripts on the internet are not actually humangenerated but the output of existing ASR systems. Recent
research has shown that training on datasets of mixed humanand machine-generated data can significantly impair the performance of translation systems (Ghorbani et al., 2021). In
order to avoid learning “transcript-ese”, we developed many
heuristics to detect and remove machine-generated transcripts from the training dataset. Many existing ASR systems output only a limited subset of written language whichremoves or normalizes away aspects that are difficult to predict from only audio signals such as complex punctuation(exclamation points, commas, and question marks), formatting whitespace such as paragraphs, or stylistic aspects suchas capitalization. An all-uppercase or all-lowercase transcript is very unlikely to be human generated. While many
ASR systems include some level of inverse text normalization, it is often simple or rule-based and still detectable fromother unhandled aspects such as never including commas.
We also use an audio language detector, which was createdby fine-tuning a prototype model trained on a prototype version of the dataset on VoxLingua107 (Valk & Alumäe, 2021)
to ensure that the spoken language matches the language ofthe transcript according to CLD2. If the two do not match,
we don’t include the (audio, transcript) pair as a speech
recognition training example in the dataset. We make an
exception if the transcript language is English and add thesepairs to the dataset as X→en speech translation training
examples instead. We use fuzzy de-duping of transcript
texts to reduce the amount of duplication and automaticallygenerated content in the training dataset.
We break audio files into 30-second segments paired withthe subset of the transcript that occurs within that time
segment. We train on all audio, including segments where
there is no speech (though with sub-sampled probability)
and use these segments as training data for voice activity
detection.
For an additional filtering pass, after training an initial model
we aggregated information about its error rate on training

Robust Speech Recognition via Large-Scale Weak Supervision3

data sources and performed manual inspection of these data
sources sorting by a combination of both high error rate anddata source size in order to identify and remove low-quality
ones efficiently. This inspection showed a large amount of
only partially transcribed or poorly aligned/misaligned transcripts as well as remaining low-quality machine-generated
captions that filtering heuristics did not detect.

ization. These components are often handled separately,
resulting in a relatively complex system around the corespeech recognition model. To reduce this complexity, we
would like to have a single model perform the entire speechprocessing pipeline, not just the core recognition part. An
important consideration here is the interface for the model.
There are many different tasks that can be performed on
the same input audio signal: transcription, translation, voiceactivity detection, alignment, and language identification
are some examples.

To avoid contamination, we perform de-duplication at a transcript level between the training dataset and the evaluation
datasets we thought were at higher risk of overlap, namely
TED-LIUM 3 (Hernandez et al., 2018).
2.2. Model
Since the focus of our work is on studying the capabilitiesof large-scale supervised pre-training for speech recognition, we use an off-the-shelf architecture to avoid confounding our findings with model improvements. We chose an
encoder-decoder Transformer (Vaswani et al., 2017) as thisarchitecture has been well validated to scale reliably. Allaudio is re-sampled to 16,000 Hz, and an 80-channel logmagnitude Mel spectrogram representation is computed on
25-millisecond windows with a stride of 10 milliseconds.
For feature normalization, we globally scale the input tobe between -1 and 1 with approximately zero mean acrossthe pre-training dataset. The encoder processes this input
representation with a small stem consisting of two convolution layers with a filter width of 3 and the GELU activation
function (Hendrycks & Gimpel, 2016) where the second
convolution layer has a stride of two. Sinusoidal position
embeddings are then added to the output of the stem after
which the encoder Transformer blocks are applied. The
transformer uses pre-activation residual blocks (Child et al.,2019), and a final layer normalization is applied to the encoder output. The decoder uses learned position embeddings
and tied input-output token representations (Press & Wolf,
2017). The encoder and decoder have the same width andnumber of transformer blocks. Figure 1 summarizes themodel architecture.
We use the same byte-level BPE text tokenizer used in GPT2 (Sennrich et al., 2015; Radford et al., 2019) for the Englishonly models and refit the vocabulary (but keep the same size)for the multilingual models to avoid excessive fragmentation on other languages since the GPT-2 BPE vocabulary is
English only.
2.3. Multitask Format
Although predicting which words were spoken in a given
audio snippet is a core part of the full speech recognition
problem and extensively studied in research, it is not the
only part. A fully featured speech recognition system caninvolve many additional components such as voice activity detection, speaker diarization, and inverse text normal-

For this kind of one-to-many mapping to work with a single
model, some form of task specification is necessary. We use
a simple format to specify all tasks and conditioning information as a sequence of input tokens to the decoder. Since
our decoder is an audio-conditional language model, we also
train it to condition on the history of text of the transcript in
the hope that it will learn to use longer-range text context
to resolve ambiguous audio. Specifically, with some probability we add the transcript text preceding the current audio
segment to the decoder’s context. We indicate the beginningof prediction with a 
 token.
First, we predict the language being spoken which is represented by a unique token for each language in our training
set (99 total). These language targets are sourced from theaforementioned VoxLingua107 model. In the case where
there is no speech in an audio segment, the model is trained
to predict a 
 token indicating this. The
next token specifies the task (either transcription or translation) with an 
 or 
token. After this, we specify whether to predict timestamps
or not by including a 
 token for thatcase. At this point, the task and desired format is fully
specified, and the output begins. For timestamp prediction, we predict time relative to the current audio segment,
quantizing all times to the nearest 20 milliseconds whichmatches the native time resolution of Whisper models, andadd additional tokens to our vocabulary for each of these.
We interleave their prediction with the caption tokens: thestart time token is predicted before each caption’s text, andthe end time token is predicted after. When a final transcript segment is only partially included in the current 30second audio chunk, we predict only its start time token
for the segment when in timestamp mode, to indicate thatthe subsequent decoding should be performed on an audio window aligned with that time, otherwise we truncate
the audio to not include the segment. Lastly, we add a
 token. We only mask out thetraining loss over the previous context text, and train themodel to predict all other tokens. Please see Figure 1 for an
overview of our format and training setup.

4Robust Speech Recognition via Large-Scale Weak Supervision

Sequence-to-sequence learning

Multitask training data (680k hours)

EN

TRANSCRIBE

0.0

The quick brown

English transcription

🗣️ “Ask not what your country can do for ⋯”📝 Ask not what your country can do for ⋯MLPcross attention

self attention

self attention⋮

🗣️ “El rápido zorro marrón salta sobre ⋯”📝 The quick brown fox jumps over ⋯Transformer
Encoder Blocks

MLP
self attention

cross attention

MLP

Any-to-English speech translation

Transformer
Decoder Blocks

MLP
cross attention

MLP🗣️ “언덕 위에 올라 내려다보면 너무나 넓고 넓은 ⋯”📝 언덕 위에 올라 내려다보면 너무나 넓고 넓은 ⋯self attention

Sinusoidal
Positional
Encoding

No speech

MLP~

cross attention
self attention

LearnedPositional
Encoding

2 × Conv1D + GELU

🔊 (background music playing)📝∅

SOT
Log-Mel Spectrogram

Multitask training format

previous
text tokens

START OFTRANSCRIPT

Custom vocabulary /promptingspecialtokens⋮⋮self attention

Non-English transcription

PREV

⋯

next-token
prediction

texttokenstimestamp
tokensLanguage
identification

X→X
Transcription

LANGUAGE
TAG

TRANSCRIBE

begin
time

NO
SPEECH

TRANSLATENOTIMESTAMPS

Voice activity
detection(VAD)

X → English
Translation

EN

TRANSCRIBE

0.0

The quick⋯

Tokens in Multitask Training Format

Time-aligned transcription

text tokens

end
time

⋯

begin
time

text tokens

end
time
EOT

text tokens

Text-only transcription(allows dataset-specific fine-tuning)

Figure 1. Overview of our approach. A sequence-to-sequence Transformer model is trained on many different speech processing tasks,
including multilingual speech recognition, speech translation, spoken language identification, and voice activity detection. All of thesetasks are jointly represented as a sequence of tokens to be predicted by the decoder, allowing for a single model to replace many different
stages of a traditional speech processing pipeline. The multitask training format uses a set of special tokens that serve as task specifiers orclassification targets, as further explained in Section 2.3.

2.4. Training Details
We train a suite of models of various sizes in order to study
the scaling properties of Whisper. Please see Table 1 for an
overview. We train with data parallelism across accelerators
using FP16 with dynamic loss scaling and activation checkpointing (Griewank & Walther, 2000; Chen et al., 2016).
Models were trained with AdamW (Loshchilov & Hutter,2017) and gradient norm clipping (Pascanu et al., 2013)
with a linear learning rate decay to zero after a warmup overthe first 2048 updates. A batch size of 256 segments wasused, and the models are trained for 220 updates which is
between two and three passes over the dataset. Due to only
training for a few epochs, over-fitting is not a large concern,and we do not use any data augmentation or regularizationand instead rely on the diversity contained within such alarge dataset to encourage generalization and robustness.
Please see Appendix F for full training hyperparameters.3
During early development and evaluation we observed that
Whisper models had a tendency to transcribe plausible but
almost always incorrect guesses for the names of speakers.
This happens because many transcripts in the pre-training
dataset include the name of the person who is speaking,encouraging the model to try to predict them, but this information is only rarely inferable from only the most recent 30
3After the original release of Whisper, we trained an additional
Large model (denoted V2) for 2.5X more epochs while adding
SpecAugment (Park et al., 2019), Stochastic Depth (Huang et al.,2016), and BPE Dropout (Provilkov et al., 2019) for regularization.
Reported results have been updated to this improved model unlessotherwise specified.

5

Robust Speech Recognition via Large-Scale Weak Supervision
Model

Layers

Width

Heads

Parameters

Tiny
Base
Small
Medium
Large

4
6
12
24
32

384
512
768
1024
1280

6
8
12
16
20

39M
74M
244M
769M
1550M

Table 1. Architecture details of the Whisper model family.

seconds of audio context. To avoid this, we fine-tune Whisper models briefly on the subset of transcripts that do notinclude speaker annotations which removes this behavior.

3. Experiments
3.1. Zero-shot Evaluation
The goal of Whisper is to develop a single robust speechprocessing system that works reliably without the need fordataset specific fine-tuning to achieve high-quality resultson specific distributions. To study this capability, we reuse a wide set of existing speech processing datasets tocheck whether Whisper is able to generalize well acrossdomains, tasks, and languages. Instead of using the standard
evaluation protocol for these datasets, which include botha train and test split, we evaluate Whisper in a zero-shotsetting without using any of the training data for each ofthese datasets so that we are measuring broad generalization.
3.2. Evaluation Metrics
Speech recognition research typically evaluates and compares systems based on the word error rate (WER) metric.
However, WER, which is based on string edit distance, penalizes all differences between the model’s output and thereference transcript including innocuous differences in transcript style. As a result, systems that output transcripts thatwould be judged as correct by humans can still have a large
WER due to minor formatting differences. While this posesa problem for all transcribers, it is particularly acute forzero-shot models like Whisper, which do not observe any
examples of specific datasets transcript formats.
This is not a novel observation; the development of evaluation metrics that better correlate with human judgement is anactive area of research, and while there are some promising
methods, none have seen widespread adoption for speech
recognition yet. We opt to address this problem with extensive standardization of text before the WER calculation
to minimize penalization of non-semantic differences. Our
text normalizer was developed through iterative manual inspection to identify common patterns where naive WER
penalized Whisper models for an innocuous difference. Appendix C includes full details. For several datasets, weobserve WER drops of up to 50 percent usually due to aquirk such as a dataset’s reference transcripts seperating
contractions from words with whitespace. We caution thisdevelopment procedure comes at a risk of overfitting to thetranscription style of Whisper models which we investigate
in Section 4.4. We are releasing the code for our text normalizer to allow for easy comparison and to help others
study the performance of speech recognition systems in
out-of-distribution settings.
3.3. English Speech Recognition
In 2015, Deep Speech 2 (Amodei et al., 2015) reported
a speech recognition system matched human-level performance when transcribing the LibriSpeech test-clean split.
As part of their analysis they concluded: “Given this result,
we suspect that there is little room for a generic speech system to further improve on clean read speech without further
domain adaptation.” Yet seven years later the SOTA WER
on LibriSpeech test-clean has dropped another 73% fromtheir 5.3% to 1.4% (Zhang et al., 2021), far below their reported human-level error rate of 5.8%. Despite this massive
and unanticipated further improvement in performance onheld-out but in-distribution data, speech recognition models trained on LibriSpeech remain far above human error
rates when used in other settings. What explains this gap
between reportedly superhuman performance in-distribution
and subhuman performance out-of-distribution?
We suspect a large part of this gap between human andmachine behavior is due to conflating different capabilities
being measured by human and machine performance on
a test set. This claim may seem confusing at first; if bothhumans and machines are taking the same test, how can it bethat different skills are being tested? The difference arises
not in the testing but in how they trained for it. Humans areoften asked to perform a task given little to no supervisionon the specific data distribution being studied. Thus humanperformance is a measure of out-of-distribution generalization. But machine learning models are usually evaluatedafter training on a large amount of supervision from theevaluation distribution, meaning that machine performanceis instead a measure of in-distribution generalization. Whileboth humans and machines are being evaluated on the sametest data, two quite different abilities are being measured
due to a difference in train data.
Whisper models, which are trained on a broad and diverse
distribution of audio and evaluated in a zero-shot setting,could potentially match human behavior much better thanexisting systems. To study whether this is the case (orwhether the difference between machine and human performance is due to yet-to-be-understood factors) we cancompare Whisper models with both human performanceand standard fine-tuned machine learning models and check
which they more closely match.

6

Robust Speech Recognition via Large-Scale Weak Supervision

Average WER on [Common Voice, CHiME-6, TED-LIUM] (%)

50
wav2vec 2.0
Large (no LM)

Whisper
Large V2

RER(%)

LibriSpeech Clean

2.7

2.7

0.0

Artie
Common Voice
Fleurs En
Tedlium
CHiME6
VoxPopuli En
CORAALAMI IHM
Switchboard
CallHomeWSJ
AMI SDM1
LibriSpeech Other

24.5
29.9
14.6
10.5
65.8
17.9
35.6
37.0
28.3
34.8
7.7
67.6
6.2

6.2
9.0
4.4
4.0
25.5
7.3
16.2
16.9
13.8
17.6
3.9
36.4
5.2

74.7
69.9
69.9
61.9
61.2
59.2
54.5
54.3
51.2
49.4
49.4
46.2
16.1

Average

29.3

12.8

55.2

Dataset40

Supervised LibriSpeech models
Zero-shot Whisper models
Zero-shot Human (Alec)
Ideal robustness (y = x)

30

20

10

0

0

1

2
3
4
5
6
WER on LibriSpeech dev-clean (%)

7

8

Figure 2. Zero-shot Whisper models close the gap to humanrobustness. Despite matching or outperforming a human on LibriSpeech dev-clean, supervised LibriSpeech models make roughly
twice as many errors as a human on other datasets demonstrating
their brittleness and lack of robustness. The estimated robustness
frontier of zero-shot Whisper models, however, includes the 95%
confidence interval for this particular human.

To quantify this difference, we examine both overall robustness, that is average performance across many distributions/datasets, and effective robustness, introduced by Taoriet al. (2020), which measures the difference in expected
performance between a reference dataset, which is usuallyin-distribution, and one or more out-of-distribution datasets.
A model with high effective robustness does better than
expected on out-of-distribution datasets as a function of its
performance on the reference dataset and approaches theideal of equal performance on all datasets. For our analysis, we use LibriSpeech as the reference dataset due to its
central role in modern speech recognition research and theavailability of many released models trained on it, whichallows for characterizing robustness behaviors. We use asuite of 12 other academic speech recognition datasets tostudy out-of-distribution behaviors. Full details about thesedatasets can be found in Appendix A.
Our main findings are summarized in Figure 2 and Table 2.
Although the best zero-shot Whisper model has a relatively
unremarkable LibriSpeech clean-test WER of 2.5, which
is roughly the performance of modern supervised baselineor the mid-2019 state of the art, zero-shot Whisper models
have very different robustness properties than supervised
LibriSpeech models and out-perform all benchmarked LibriSpeech models by large amounts on other datasets. Even

Table 2. Detailed comparison of effective robustness across various datasets. Although both models perform within 0.1% ofeach other on LibriSpeech, a zero-shot Whisper model performs
much better on other datasets than expected for its LibriSpeech
performance and makes 55.2% less errors on average. Results
reported in word error rate (WER) for both models after applying
our text normalizer.

the smallest zero-shot Whisper model, which has only 39
million parameters and a 6.7 WER on LibriSpeech test-clean
is roughly competitive with the best supervised LibriSpeechmodel when evaluated on other datasets. When compared
to a human in Figure 2, the best zero-shot Whisper models
roughly match their accuracy and robustness. For a detailed
breakdown of this large improvement in robustness, Table
2 compares the performance of the best zero-shot Whispermodel with a supervised LibriSpeech model that has theclosest performance to it on LibriSpeech test-clean. Despite
their very close performance on the reference distribution,
the zero-shot Whisper model achieves an average relative
error reduction of 55.2% when evaluated on other speech
recognition datasets.
This finding suggests emphasizing zero-shot and out-ofdistribution evaluations of models, particularly when attempting to compare to human performance, to avoid overstating the capabilities of machine learning systems due tomisleading comparisons.
3.4. Multi-lingual Speech Recognition
In order to compare to prior work on multilingual speech
recognition, we report results on two low-data benchmarks:
Multilingual LibriSpeech (MLS) (Pratap et al., 2020b) and
VoxPopuli (Wang et al., 2021) in Table 3.
Whisper performs well on Multilingual LibriSpeech, outperforming XLS-R (Babu et al., 2021), mSLAM (Bapna

7

Robust Speech Recognition via Large-Scale Weak Supervision

160

40
LO

MY

80

GU KA PA BNUZ ML KM
TG
MT

40

MR

TE

35
SR

BE

HY
KN SW IS
KK
SR
AF
FA

10
5
r2 = 0.83
2.5

0.1

1

20

ET
LB

15

TG

10

LO
UZ

5

EN

100K

1M

Figure 3. Correlation of pre-training supervision amount withdownstream speech recognition performance. The amount of
pre-training speech recognition data for a given language is very
predictive of zero-shot performance on that language in Fleurs.

Model

MLS

VoxPopuli

VP-10K + FT
XLS-R (1B)mSLAM-CTC (2B)
Maestro

10.9
9.7
-

15.3
10.6
9.1
8.1

Zero-Shot Whisper

7.3

13.6

Table 3. Multilingual speech recognition performance. Zeroshot Whisper improves performance on Multilingual LibriSpeech(MLS) but is still significantly behind both Maestro, XLS-R, andmSLAM on VoxPopuli.

et al., 2022), and Maestro (Chen et al., 2022b) in a zero-shotsetting. We caution that we do use a simple text standardizer
for this result which prevents direct comparison or claimsof SOTA performance. On VoxPopuli, however, Whisper
significantly underperforms prior work and only beats the
VP-10K+FT baseline from the original paper. We suspect
the underperformance of Whisper models on VoxPopuli
could be due to other models including this distribution asa major source for their unsupervised pre-training data andthe dataset having significantly more supervised data, which
benefits fine-tuning. While MLS has 10 hours of training
data per language, the average amount of training data per
language is roughly 10× higher for VoxPopuli.
These two benchmarks are somewhat narrow since they
only include 15 unique languages, almost all of which are in

0

HA

1

10

DA

DE
FR

SL
HY
GU
LV PA NE
LT
MR
MT
KN
AZ BE
IS

SD
AS
KK
KA PS
LN AM
MN
SO
MY

RU

ID
MS IT
EL TR AR
PL
FIL
FI
NL
HEHU
VI
FA

OC

ES

10
100
1K
10K
Hours of transcribed audio

AF

BS NB GL RO
SK HR
BG CSUK

MK

25

CY
LT

HE
AZ LV UR
HI SL
ET
TA HU
MK
AR
GL
BS
KO ZH
FIL BG RO DA
HR CS ELSK TH VIFI
NB
MS UK SV TR FR
ID CANLPL JA RU
PT DE
IT

20

SV

30

BLEU

Word Error Rate (WER)

NE

PT
CA

ES
KO

HI
ZH

ML

TH UR
TE BN

JA
CY

MI TA
SW
KM
SN

r2 = 0.24

YO

100
1K
Hours of translated audio

10K

100K

Figure 4. Correlation of pre-training supervision amount withdownstream translation performance. The amount of pretraining translation data for a given language is only moderately
predictive of Whisper’s zero-shot performance on that language in
Fleurs.

the Indo-European language family and many of which are
high-resource languages. These benchmarks only provide
limited coverage and room to study Whisper models multilingual capabilities which include training data for speech
recognition in 75 languages. To study the performance ofWhisper more broadly we also report performance on the
Fleurs dataset (Conneau et al., 2022). In particular, we wereinterested in studying the relationship between the amountof training data we have for a given language and the resulting downstream zero-shot performance for that language.
We visualize this relation in Figure 3. We find a strongsquared correlation coefficient of 0.83 between the log of
the word error rate and the log of the amount of training
data per language. Checking the regression coefficient for alinear fit to these log-log values results in an estimate that
WER halves for every 16× increase in training data. We
also observed that many of the largest outliers in terms ofworse than expected performance according to this trend arelanguages that have unique scripts and are more distantly
related to the Indo-European languages making up the majority of the training dataset such as Hebrew (HE), Telugu(TE), Chinese (ZH), and Korean (KO). These differences
could be due to a lack of transfer due to linguistic distance,
our byte level BPE tokenizer being a poor match for theselanguages, or variations in data quality.

8

X → English

High

Mid

Low

All

XMEF-X
XLS-R (2B)mSLAM-CTC (2B)
Maestro

34.2
36.1
37.8
38.2

20.2
27.7
29.6
31.3

5.9
15.1
18.5
18.4

14.7
22.1
24.8
25.2

Zero-Shot Whisper

36.2

32.6

25.2

29.1

Table 4. X→en Speech translation performance. Zero-shot
Whisper outperforms existing models on CoVoST2 in the overall,medium, and low resource settings but still moderately underperforms on high-resource languages compared to prior directlysupervised work.

Language ID

Fleursw2v-bert-51 (0.6B)
mSLAM-CTC (2B)

71.4
77.7

Zero-shot Whisper

64.5

Table 5. Language identification performance. Zero-shot Whisper’s accuracy at language identification is not competitive withprior supervised results on Fleurs. This is partially due to Whisper
being heavily penalized for having no training data for 20 of Fleurs
languages.

WER on LibriSpeech test-clean (%)

Robust Speech Recognition via Large-Scale Weak Supervision

white noise

100

pub noise

50
20
10
5
2
1

40

30

20

10

0

signal-to-noise ratio (dB)

-10 40

30

20

10

0

signal-to-noise ratio (dB)-10unispeech-sat-base-100h-libri-ft
hubert-large-ls960-ft
wav2vec2-base-100h
hubert-xlarge-ls960-ft
wav2vec2-base-960h
s2t-medium-librispeech-asr
wav2vec2-large-960h
s2t-large-librispeech-asr
wav2vec2-large-robust-ft-libri-960h
stt_en_conformer_ctc_large
wav2vec2-large-960h-lv60-self
stt_en_conformer_transducer_xlarge
asr-crdnn-rnnlm-librispeech
Whisper
asr-transformer-transformerlm-librispeech

Figure 5. WER on LibriSpeech test-clean as a function of SNR
under additive white noise (left) and pub noise (right). The
accuracy of LibriSpeech-trained models degrade faster than thebest Whisper model (⋆). NVIDIA STT models (•) perform best
under low noise but are outperformed by Whisper under high noise(SNR < 10 dB). The second-best model under low noise (▼) isfine-tuned on LibriSpeech only and degrades even more quickly.

3.5. Translation
We study the translation capabilities of Whisper models
by measuring their performance on the X→en subset of
CoVoST2 (Wang et al., 2020b). We compare with Maestro,mSLAM, and XLS-R, the highest-performing prior work.
We achieve a new state of the art of 29.1 BLEU zero-shotwithout using any of the CoVoST2 training data. We attribute this to the 68,000 hours of X→en translation data
for these languages in our pre-training dataset which, although noisy, is vastly larger than the 861 hours of training
data for X→en translation in CoVoST2. Since Whisper evaluation is zero-shot, it does particularly well on the lowest
resource grouping of CoVoST2, improving over mSLAM
by 6.7 BLEU. Conversely, the best Whisper model does not
actually improve over Maestro and mSLAM on average forthe highest resource languages.
For an additional analysis on an even wider set of languages,
we also re-purpose Fleurs, which is a speech recognition
dataset, as a translation dataset. Since the same sentences
are transcribed for every language we use the English transcripts as reference translations. In Figure 4 we visualize
the correlation between the amount of translation training
data per language and the resulting zero-shot BLEU score
on Fleurs. While there is a clear trend of improvement withincreasing training data, the squared correlation coefficientis much lower than the 0.83 observed for speech recognitionand only 0.24. We suspect this is partly caused by the noisiertraining data due to errors in audio language identification.
As an example, Welsh (CY) is an outlier with much worse
than expected performance at only 13 BLEU despite supposedly having 9,000 hours of translation data. This large
amount of Welsh translation data is surprising, ranking 4th
overall for translation data and ahead of some of the most
spoken languages in the world like French, Spanish, andRussian. Inspection shows the majority of supposedly Welsh
translation data is actually English audio with English captions where the English audio was mis-classified as Welshby the language identification system, resulting in it being
included as translation training data rather transcription dataaccording to our dataset creation rules.
3.6. Language Identification
To evaluate language identification, we use the Fleurs
dataset (Conneau et al., 2022). The zero-shot performanceof Whisper is not competitive with prior supervised work
here and underperforms the supervised SOTA by 13.6%.
However, Whisper is heavily disadvantaged for language
identification on Fleurs, since the Whisper dataset contains
no training data for 20 of the 102 languages in Fleurs, upperbounding accuracy at 80.4%. On the 82 overlapping languages the best Whisper model achieves 80.3% accuracy.

9

Robust Speech Recognition via Large-Scale Weak Supervision

3.7. Robustness to Additive Noise
We tested the noise robustness of Whisper models and 14
LibriSpeech-trained models by measuring the WER wheneither white noise or pub noise from the Audio Degradation Toolbox (Mauch & Ewert, 2013) was added to theaudio. The pub noise represents a more natural noisy environment with ambient noise and indistinct chatter typicalin a crowded restaurant or a pub. Among the 14 models,
twelve are pre-trained and/or fine-tuned on LibriSpeech, andthe other two are NVIDIA STT models trained on a mixture
dataset similar to prior work like SpeechStew that includes
LibriSpeech. The level of additive noise corresponding toa given signal-to-noise ratio (SNR) is calculated based on
the signal power of individual examples. Figure 5 shows
how the ASR performance degrades as the additive noise
becomes more intensive. There are many models that outperform our zero-shot performance under low noise (40 dB
SNR), which is unsurprising given those models are trained
primarily on LibriSpeech, but all models quickly degrade as
the noise becomes more intensive, performing worse thanthe Whisper model under additive pub noise of SNR below
10 dB. This showcases Whisper’s robustness to noise, especially under more natural distribution shifts like the pub
noise.
3.8. Long-form Transcription
Whisper models are trained on 30-second audio chunks andcannot consume longer audio inputs at once. This is not a

problem with most academic datasets comprised of shortutterances but presents challenges in real-world applications
which often require transcribing minutes- or hours-long audio. We developed a strategy to perform buffered transcription of long audio by consecutively transcribing 30-second
segments of audio and shifting the window according to thetimestamps predicted by the model. We observed that it
is crucial to have beam search and temperature schedulingbased on the repetitiveness and the log probability of themodel predictions in order to reliably transcribe long audio.
The full procedure is described in Section 4.5.
We evaluate the long-form transcription performance onseven datasets consisting of speech recordings of various
lengths and recording conditions, to cover as diverse a data
distribution as possible. These include a long-form adaptation of TED-LIUM3 (Hernandez et al., 2018) concatenated
so that each example is a full-length TED talk, a collectionof jargon-laden segments taken from The Late Show with
Stephen Colbert (Meanwhile), sets of videos/podcasts thathas been used as ASR benchmarks in online blogs (Rev16
and Kincaid46), recordings of earnings calls (Del Rio et al.,2021), and the full-length interviews from the Corpus ofRegional African American Language (CORAAL) (Gunter
et al., 2021). Full details about the long-form datasets canbe found in Appendix A.
We compare the performance with open-source models aswell as 4 commercial ASR services. The results are summarized in Figure 6, showing the distribution of word error
rates from Whisper and the 4 commercial ASR services,40
35

Word Error Rate (%)

30
25
20
15
10
5
0

TED-LIUM3

Meanwhile

WhisperCompany A

Kincaid46
Company B

Rev16
Company C

Earnings-21

Earnings-22

Company D

CORAALNVIDIA STT (CTC large)

Figure 6. Whisper is competitive with state-of-the-art commercial and open-source ASR systems in long-form transcription. The
distribution of word error rates from six ASR systems on seven long-form datasets are compared, where the input lengths range from a
few minutes to a few hours. The boxes show the quartiles of per-example WERs, and the per-dataset aggregate WERs are annotatedon each box. Our model outperforms the best open source model (NVIDIA STT) on all datasets, and in most cases, commercial ASR
systems as well.

10

Robust Speech Recognition via Large-Scale Weak Supervision

3.9. Comparison with Human Performance
Because of ambiguous or indistinct speech as well as labeling errors, there are different levels of irreducible error
in each dataset, and with WER metrics from ASR systems
alone it is difficult to make sense of how much room for
improvement exists in each dataset. To quantify how close
Whisper’s performance is to the human performance, we selected 25 recordings from the Kincaid46 dataset and used 5
services to obtain transcripts produced by professional transcribers, among which one provides computer-assisted transcription and the other four are entirely human-transcribed.
The audio selection covers various recording conditions
such as scripted and unscripted broadcast, telephone andVoIP calls, and meetings. Figure 7 shows the distributionof per-example WERs and aggregate WER across the 25
recordings, where the computer-assisted service has thelowest aggregate WER that is 1.15% point better than Whisper’s, and the pure-human performance is only a fractionof a percentage point better than Whisper’s. These results
indicate that Whisper’s English ASR performance is not
perfect but very close to human-level accuracy.

4. Analysis and Ablations
4.1. Model Scaling
A large amount of the promise in weakly supervised training approaches is their potential to use datasets much larger
than those in traditional supervised learning. However, thiscomes with the cost of using data that is possibly muchnoisier and lower quality than gold-standard supervision.
A concern with this approach is that although it may lookpromising to begin with, the performance of models trainedon this kind of data may saturate at the inherent quality levelof the dataset, which could be far below human level. A related concern is that as capacity and compute spent training
on the dataset increases, models may learn to exploit the30

Word Error Rate (%)

as well as the NVIDIA STT Conformer-CTC Large modelfrom the NeMo toolkit (Kuchaiev et al., 2019) which performed the best among the open-source models. All commercial ASR services are queried using their default English
transcription settings as of September 1st, 2022, and for
the NVIDIA STT model we used their buffered inference
implementation in the FrameBatchASR class to enablelong-form transcription. The results show that Whisper performs better than the compared models on most datasets,
especially on the Meanwhile dataset which is heavy with
uncommon words. Additionally, we note the possibility thatsome of the commercial ASR systems have been trainedon some of these publicly available datasets, and therefore
these results may not be accurately reflecting the relativerobustness of the systems.

25
20
15
10
5
0

Whisper A

B

ASR

C

D

E

F

G

H

I

human transcription
computer-assisted

Figure 7. Whisper’s performance is close to that of professional
human transcribers. This plot shows the WER distributions of
25 recordings from the Kincaid46 dataset transcribed by Whisper,
the same 4 commercial ASR systems from Figure 6 (A-D), one
computer-assisted human transcription service (E) and 4 humantranscription services (F-I). The box plot is superimposed with dots
indicating the WERs on individual recordings, and the aggregate
WER over the 25 recordings are annotated on each box.

idiosyncrasies of the dataset, and their ability to generalize
robustly to out-of-distribution data could even degrade.
To check whether this is the case, we study the zero-shot
generalization of Whisper models as a function of the modelsize. Our analysis is summarized in Figure 8. With the
exception of English speech recognition, performance continues to increase with model size across multilingual speech
recognition, speech translation, and language identification.
The diminishing returns for English speech recognition
could be due to saturation effects from approaching humanlevel performance as analysis in Section 3.9 suggests.
4.2. Dataset Scaling
At 680,000 hours of labeled audio, the Whisper dataset isone of the largest ever created in supervised speech recognition. Exactly how important is the raw dataset size toWhisper’s performance? To study this, we trained a seriesof medium-sized models on subsampled versions of thedataset which are 0.5%, 1%, 2%, 4%, and 8% of the full
dataset size and compared their performance with the same
medium-sized model trained on the whole dataset. Early
stopping based on the validation loss was used to selectmodel checkpoints for each dataset size. Evaluation was
performed on an exponential moving average estimate of
the parameters (Polyak & Juditsky, 1992) using a smoothing rate of 0.9999 to help reduce the effect of the learning
rate not fully decaying to zero for the models trained on thesubsampled datasets due to early stopping. Performanceon English and multilingual speech recognition and X→entranslation is reported in Table 6.

11

Robust Speech Recognition via Large-Scale Weak Supervision

17.5

Multilingual Speech Recognition (Fleurs)

50

Average
Large V2

12.5
10.0
7.5
5.0

X->En Translation (CoVoST2)

60
40
20

80

Average
Large V240BLEU on 21 languages

80

15.0

WER on 67 languages (%)

WER on 12 datasets (%)

100

Average
Large V2

Accuracy on 102 languages (%)

English Speech Recognition

20.0

30
20
10

Language Identification (Fleurs)
Average
Large V2

70
60
50
40

2.5
0.0
38M

73M

244M
768M 1549M
Model parameters

0
38M

73M

244M
768M 1549M
Model parameters

0
38M

73M

244M
768M 1549M
Model parameters

30
38M

73M

244M
768M 1549M
Model parameters

Figure 8. Zero-shot Whisper performance scales reliably across tasks and languages with increasing model size. Lightly shadedlines represent individual datasets or languages, showing that performance is more varied than the smooth trends in aggregate performance.
Large V2 distinguished with a dashed orange line since it includes several changes that are not present for the smaller models in thisanalysis.

Datasetsize

EnglishWER (↓)

Multilingual
WER (↓)

X→En
BLEU (↑)

3405
6811
13621
27243
54486
681070

30.5
19.6
14.4
12.3
10.9
9.9

92.4
72.7
56.6
45.0
36.4
29.2

0.2
1.7
7.9
13.9
19.2
24.8

Table 6. Performance improves with increasing dataset size.
English speech recognition performance refers to an average over
12 datasets while the Multilingual speech recognition reports performance on the overlapping subset of languages in Fleurs and
X→en translation reports average BLEU on CoVoST2. Dataset
size reported in hours.

All increases in the dataset size result in improved performance on all tasks, although we see significant variability
in improvement rates across tasks and sizes. Performance
improves rapidly on English speech recognition from 3,000
to 13,000 hours and then slows down noticeably between
13,000 and 54,000 hours. Using the full dataset, which corresponds to another 12.5× increase in size results in only afurther 1 point drop in WER. This mirrors the diminishingreturns observed with model size scaling for English speech
recognition and could similarly be explained by saturation
effects when approaching human-level performance.
Improvements in WER follow a power-law trend for multilingual speech recognition till 54,000 hours and then deviate from this trend, improving only a further 7 points
when increasing to the full dataset size. For X→en translation, performance is practically zero when training on 7,000
hours of audio or less, and then follows a roughly log-linear
improvement trend till 54,000 hours before also showing

diminishing returns when further scaling to the full dataset
size.
The general trend across tasks of diminishing returns whenmoving from 54,000 hours to our full dataset size of 680,000
hours could suggest that the current best Whisper models areunder-trained relative to dataset size and performance could
be further improved by a combination of longer training
and larger models. It could also suggest that we are nearing
the end of performance improvements from dataset size
scaling for speech recognition. Further analysis is needed tocharacterize “scaling laws” for speech recognition in orderto decided between these explanations.
4.3. Multitask and Multilingual Transfer
A potential concern with jointly training a single modelon many tasks and languages is the possibility of negativetransfer where interference between the learning of several
tasks results in performance worse than would be achievedby training on only a single task or language. To investigate
whether this is occurring, we compared the performanceof models trained on just English speech recognition with
our standard multitask and multilingual training setup andmeasured their average performance across our suite of zeroshot English speech recognition benchmarks. We adjust for
the amount of FLOPs spent training on the task of English
speech recognition as only 65% of compute is spent on thistask in a joint training setup; analysis would otherwise be
confounded by under-training on the task when compared
to a same-sized English-only model.
Our results visualized in Figure 9 show that for small models
trained with moderate amounts of compute, there is indeed
negative transfer between tasks and languages: joint models underperform English-only models trained for the sameamount of compute. However, multitask and multilingual

12Robust Speech Recognition via Large-Scale Weak Supervision

Average WER on 11 english speech recognition datasets

20

English Only
Multilingual and Multitask

18
16
14
12

10

8

10e+1910e+2010e+21FLOPs training on english speech recognition

10e+22

Figure 9. Multitask and multilingual transfer improves withscale. For small models, performance on English speech recognition degrades when trained jointly in a multitask and multilingual
setup. However, multilingual and multitask models benefit morefrom scale and eventually outperform models trained on English
data only. 95% bootstrap estimate confidence intervals are shown.

models scale better and for our largest experiments outperform their English-only counterparts demonstrating positivetransfer from other tasks. For our largest experiments, joint
models also slightly outperform English-only models evenwhen not adjusting for compute spent per task.

CORAAL
CommonVoice9.en
AMI-SDM1
CommonVoice5.1
Fleurs.en_us
AMI-IHM
Artie
LibriSpeech
TED-LIUM3
VoxPopuli.en
WSJ
CallHome
Switchboard

Open-source models
Whisper models

50
40
30
20
10
0
Relative WER reduction compared to FairSpeech's normalizer (%)
Figure 10. On most datasets, our text normalizer has similareffect on reducing WERs between Whisper models and otheropen-source models, compared to FairSpeech’s normalizer. For
each dataset, the boxplot shows the distribution of relative WER
reduction across different models in our eval suite, showing thatusing our text normalizer generally results in lower WERs than
FairSpeech’s. On a few datasets our normalizer reduces WER
significantly and more so for Whisper models, such as CallHome
and Switchboard which have many contractions in the ground truth
and WSJ which contains many numerical expressions.

ken forms of numerical and monetary expressions, such as“sixty-eight million dollars” versus “$68 million”.
4.5. Strategies for Reliable Long-form Transcription

4.4. Text Normalization
Since we developed our text normalization jointly with
Whisper to discount innocuous word errors, there is a riskthat our normalizer is overfitted to fixing Whisper’s peculiarities rather than addressing general variation in transcription.
To check this, we compared the performance of Whisperusing our normalizer versus an independently developed
one from the FairSpeech project (Koenecke et al., 2020). In
Figure 10, we visualize the differences. On most datasetsthe two normalizers perform similarly, without significant
differences in WER reduction between Whisper and compared open-source models, while on some datasets, namely
WSJ, CallHome, and Switchboard, our normalizer reduces
the WER of Whisper models’ significantly more. The differences in reduction can be traced down to different formats
used by the ground truth and how the two normalizers are penalizing them. For example, in CallHome and Switchboard,
our standardizer did not penalize differences in common
English contractions such as “you’re” versus “you are”, andin WSJ, our normalizer standardized the written and spo-

Transcribing long-form audio using Whisper relies on accurate prediction of the timestamp tokens to determine theamount to shift the model’s 30-second audio context window by, and inaccurate transcription in one window may
negatively impact transcription in the subsequent windows.
We have developed a set of heuristics that help avoid failure cases of long-form transcription, which is applied in
the results reported in sections 3.8 and 3.9. First, we use
beam search with 5 beams using the log probability as thescore function, to reduce repetition looping which happens
more frequently in greedy decoding. We start with temperature 0, i.e. always selecting the tokens with the highest probability, and increase the temperature by 0.2 up to
1.0 when either the average log probability over the generated tokens is lower than −1 or the generated text has agzip compression rate higher than 2.4. Providing the transcribed text from the preceding window as previous-text
conditioning when the applied temperature is below 0.5
further improves the performance. We found that the probability of the 
 token alone is not sufficient133.95 5.16 9.69 11.7 10.7 14.0 22.0
4.16 5.71 9.42 11.5 10.2 13.4 20.0
4.16 5.71 9.42 11.5 10.2 13.4 20.0
3.56 4.61 9.45 11.4 10.1 13.2 19.4
3.42 6.16 8.72 11.0 9.63 13.3 18.1
3.51 5.26 8.41 11.5 9.73 12.6 19.1

11.0
10.6
10.6
10.2
10.0
10.0

Rev16

Average

Earnings-22

Earnings-21

Kincaid46

billion-parameter models (Zhang et al., 2020) and using up
to 1,000,000 hours of training data (Zhang et al., 2021).
CORAAL

Greedy decoding only
+ Beam search+ Temperature fallback+ Voice activity detection+ Previous text conditioning+ Initial timestamp constraint

Meanwhile

TED-LIUM3

Robust Speech Recognition via Large-Scale Weak Supervision

Table 7. Long-form transcription performance improves incrementally as additional decoding heuristics are employed. Details oneach intervention are described in Section 4.5.

to distinguish a segment with no speech, but combining
the no-speech probability threshold of 0.6 and the averagelog-probability threshold of −1 makes the voice activity
detection of Whisper more reliable. Finally, to avoid a failure mode where the model ignores the first few words inthe input, we constrained the initial timestamp token to be
between 0.0 and 1.0 second. Table 7 shows that adding eachof the interventions above incrementally reduces the WER
overall, but not evenly across the dataset. These heuristics
serve as a workaround for the noisy predictions of the model,
and more research would be needed to further improve thereliability of long-form decoding.

5. Related Work
Scaling Speech Recognition A consistent theme acrossspeech recognition research has been documenting the benefits of scaling compute, models, and datasets. Early work applying deep learning to speech recognition found improved
performance with model depth and size and leveraged GPU
acceleration to make training these larger models tractable(Mohamed et al., 2009). Further research demonstrated thatthe benefit of deep learning approaches to speech recognition increased with dataset size, improving from being only
competitive with prior GMM-HMM systems when using
just 3 hours of TIMIT training data for phone recognition
to achieving a 30% word error rate reduction when trainedon the 2,000 hour Switchboard dataset (Seide et al., 2011).
Liao et al. (2013) is an early example of leveraging weakly
supervised learning to increase the size of a deep learning based speech recognition dataset by over 1,000 hours.
These trends continued with Deep Speech 2 (Amodei et al.,2015) being a notable system developing high-throughput
distributed training across 16 GPUs and scaling to 12,000
hours of training data while demonstrating continuing improvements at that scale. By leveraging semi-supervised
pre-training, Narayanan et al. (2018) were able to growdataset size much further and study training on 162,000
hours of labeled audio. More recent work has explored

Multitask Learning Multitask learning (Caruana, 1997)
has been studied for a long time. In speech recognition,
multi-lingual models have been explored for well over a
decade (Schultz & Kirchhoff, 2006). An inspirational andfoundational work in NLP exploring multi-task learningwith a single model is Collobert et al. (2011). Multitask
learning in the sequence-to-sequence framework (Sutskever
et al., 2014) using multiple encoders and decoders was investigated in Luong et al. (2015). The use of language codeswith a shared encoder/decoder architecture was first demonstrated for machine translation by Johnson et al. (2017),
removing the need for separate encoders and decoders. Thisapproach was simplified further into the “text-to-text” framework of McCann et al. (2018) and popularized by its successwith large transformer language models in the work of Radford et al. (2019) and Raffel et al. (2020). Toshniwal et al.
(2018) demonstrated jointly training a modern deep learning speech recognition system on several languages with asingle model, and Pratap et al. (2020a) scaled this line ofwork significantly to 50 languages with a billion-parametermodel. MUTE (Wang et al., 2020c) and mSLAM (Bapna
et al., 2022) studied joint training over both text and speechlanguage tasks, demonstrating transfer between them.

Robustness The question of how effectively models transfer and how robust they are to distribution shift and othertypes of perturbations has long been studied and is actively
being researched across many fields of machine learning.
Torralba & Efros (2011) highlighted the lack of generalization of machine learning models between datasets over a
decade ago. Many other works have shown and continually reiterated how despite high performance on IID test
sets, machine learning models can still make many mistakes
when evaluated in even slightly different settings (Lake et al.,2017; Jia & Liang, 2017; Alcorn et al., 2019; Barbu et al.,2019; Recht et al., 2019). More recently, Taori et al. (2020)studied the robustness of image classification models, andMiller et al. (2020) investigated this for question-answeringmodels. A key finding has been that multi-domain training increases robustness and generalization as discussed in
the Introduction. This finding has been replicated across
many fields in addition to speech recognition including NLP(Hendrycks et al., 2020) and computer vision (Radford et al.,2021).

6. Limitations and Future Work
From our experimental results, analyses, and ablations, wehave noted several limitations and areas for future work.

Robust Speech Recognition via Large-Scale Weak Supervision

14

Improved decoding strategies. As we have scaled Whisper, we have observed that larger models have made steadyand reliable progress on reducing perception-related errors
such as confusing similar-sounding words. Many remaining
errors, particularly in long-form transcription seem more
stubborn in nature and decidedly non-human/perceptual.
They are a combination of failure modes of seq2seq models, language models, and text-audio alignment and include
problems such as getting stuck in repeat loops, not transcribing the first or last few words of an audio segment, orcomplete hallucination where the model will output a transcript entirely unrelated to the actual audio. Although the
decoding details discussed in Section 4.5 help significantly,
we suspect fine-tuning Whisper models on a high-quality
supervised dataset and/or using reinforcement learning tomore directly optimize for decoding performance could help
further reduce these errors.

as wav2vec 2.0 change when used together with a language
model.

Increase Training Data For Lower-Resource Languages
As Figure 3 shows, Whisper’s speech recognition performance is still quite poor on many languages. The same
analysis suggests a clear route for improvement since performance on a language is very well predicted by the amountof training data for the language. Since our pre-training
dataset is currently very English-heavy due to biases ofour data collection pipeline, which sourced primarily fromEnglish-centric parts of the internet, most languages haveless than 1000 hours of training data. A targeted effort at increasing the amount of data for these rarer languages could
result in a large improvement to average speech recognition
performance even with only a small increase in our overall
training dataset size.
Studying fine-tuning In this work, we have focused on
the robustness properties of speech processing systems andas a result only studied the zero-shot transfer performanceof Whisper. While this is a crucial setting to study due to it
being representative of general reliability, for many domains
where high-quality supervised speech data does exist, it islikely that results can be improved further by fine-tuning.
An additional benefit of studying fine-tuning is that it allows
for direct comparisons with prior work since it is a much
more common evaluation setting.
Studying the impact of Language Models on Robustness
As argued in the introduction, we suspect that Whisper’s
robustness is partially due to its strong decoder, which is anaudio conditional language model. It’s currently unclear towhat degree the benefits of Whisper stem from training its
encoder, decoder, or both. This could be studied by either
ablating various design components of Whisper, such as
training a decoder-less CTC model, or by studying how theperformance of existing speech recognition encoders such

Adding Auxiliary Training Objectives Whisper departs
noticeably from most recent state-of-the-art speech recognition systems due to the lack of unsupervised pre-training
or self-teaching methods. While we have not found themnecessary to achieve good performance, it is possible thatthe results could be further improved by incorporating this.

7. Conclusion
Whisper suggests that scaling weakly supervised pretraining has been underappreciated so far in speech recognition research. We achieve our results without the need forthe self-supervision and self-training techniques that havebeen a mainstay of recent large-scale speech recognition
work and demonstrate how simply training on a large anddiverse supervised dataset and focusing on zero-shot transfer can significantly improve the robustness of a speech
recognition system.
ACKNOWLEDGMENTS
We’d like to thank the millions of people who were involvedin creating the data used by Whisper. We’d also like tothank Nick Ryder, Will Zhuk, and Andrew Carr for theconversation on the waterfall hike that inspired this project.
We are also grateful to the Acceleration and Supercomputing
teams at OpenAI for their critical work on software andhardware infrastructure this project used. We’d also like tothank Pamela Mishkin for advising the project from a policy
perspective. Finally, we are grateful to the developers ofthe many software packages used throughout this project
including, but not limited, to Numpy (Harris et al., 2020),
SciPy (Virtanen et al., 2020), ftfy (Speer, 2019), PyTorch(Paszke et al., 2019), pandas (pandas development team,2020), and scikit-learn (Pedregosa et al., 2011).

References
Alcorn, M. A., Li, Q., Gong, Z., Wang, C., Mai, L., Ku, W.S., and Nguyen, A. Strike (with) a pose: Neural networks
are easily fooled by strange poses of familiar objects. In
Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition, pp. 4845–4854, 2019.
Amodei, D., Anubhai, R., Battenberg, E., Case, C., Casper,J., Catanzaro, B., Chen, J., Chrzanowski, M., Coates,A., Diamos, G., et al. Deep speech 2: end-to-end speech
recognition in english and mandarin. arxiv. arXiv preprint
arXiv:1512.02595, 2015.
Ardila, R., Branson, M., Davis, K., Henretty, M., Kohler,M., Meyer, J., Morais, R., Saunders, L., Tyers, F. M.,

Robust Speech Recognition via Large-Scale Weak Supervision

15

and Weber, G. Common voice: A massively-multilingual
speech corpus. arXiv preprint arXiv:1912.06670, 2019.

Child, R., Gray, S., Radford, A., and Sutskever, I. Generating long sequences with sparse transformers. arXiv
preprint arXiv:1904.10509, 2019.

Babu, A., Wang, C., Tjandra, A., Lakhotia, K., Xu,
Q., Goyal, N., Singh, K., von Platen, P., Saraf, Y.,
Pino, J., et al. XLS-R: Self-supervised cross-lingual
speech representation learning at scale. arXiv preprint
arXiv:2111.09296, 2021.
Baevski, A., Zhou, H., Mohamed, A., and Auli, M. wav2vec
2.0: A framework for self-supervised learning of speech
representations. arXiv preprint arXiv:2006.11477, 2020.
Baevski, A., Hsu, W.-N., Conneau, A., and Auli, M. Unsupervised speech recognition. Advances in Neural Information Processing Systems, 34:27826–27839, 2021.
Bapna, A., Cherry, C., Zhang, Y., Jia, Y., Johnson, M.,
Cheng, Y., Khanuja, S., Riesa, J., and Conneau, A. mslam:
Massively multilingual joint pre-training for speech andtext. arXiv preprint arXiv:2202.01374, 2022.
Barbu, A., Mayo, D., Alverio, J., Luo, W., Wang, C., Gutfreund, D., Tenenbaum, J., and Katz, B. Objectnet: Alarge-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural
information processing systems, 32, 2019.
Caruana, R. Multitask learning. Machine learning, 28(1):41–75, 1997.
Chan, W., Park, D., Lee, C., Zhang, Y., Le, Q., and Norouzi,
M. SpeechStew: Simply mix all available speech recognition data to train one large neural network. arXiv preprint
arXiv:2104.02133, 2021.
Chen, G., Chai, S., Wang, G., Du, J., Zhang, W.-Q.,
Weng, C., Su, D., Povey, D., Trmal, J., Zhang, J.,et al. Gigaspeech: An evolving, multi-domain asr corpus
with 10,000 hours of transcribed audio. arXiv preprint
arXiv:2106.06909, 2021.
Chen, S., Wu, Y., Wang, C., Chen, Z., Chen, Z., Liu, S.,
Wu, J., Qian, Y., Wei, F., Li, J., et al. Unispeech-sat: Universal speech representation learning with speaker aware
pre-training. In ICASSP 2022-2022 IEEE International
Conference on Acoustics, Speech and Signal Processing(ICASSP), pp. 6152–6156. IEEE, 2022a.
Chen, T., Xu, B., Zhang, C., and Guestrin, C. Training
deep nets with sublinear memory cost. arXiv preprint
arXiv:1604.06174, 2016.
Chen, Z., Zhang, Y., Rosenberg, A., Ramabhadran, B.,
Moreno, P., Bapna, A., and Zen, H. Maestro: Matchedspeech text representations through modality matching.
arXiv preprint arXiv:2204.03409, 2022b.

Collobert, R., Weston, J., Bottou, L., Karlen, M.,
Kavukcuoglu, K., and Kuksa, P. Natural language processing (almost) from scratch. Journal of machine learning research, 12(ARTICLE):2493–2537, 2011.
Conneau, A., Ma, M., Khanuja, S., Zhang, Y., Axelrod, V.,
Dalmia, S., Riesa, J., Rivera, C., and Bapna, A. Fleurs:
Few-shot learning evaluation of universal representationsof speech. arXiv preprint arXiv:2205.12446, 2022.
Del Rio, M., Delworth, N., Westerman, R., Huang, M.,
Bhandari, N., Palakapilly, J., McNamara, Q., Dong, J.,
Zelasko, P., and Jetté, M. Earnings-21: a practical benchmark for asr in the wild. arXiv preprint arXiv:2104.11348,
2021.
Galvez, D., Diamos, G., Torres, J. M. C., Achorn, K., Gopi,
A., Kanter, D., Lam, M., Mazumder, M., and Reddi, V. J.
The people’s speech: A large-scale diverse english speech
recognition dataset for commercial usage. arXiv preprint
arXiv:2111.09344, 2021.
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., and Wichmann, F. A. Shortcut learning in deep neural networks. Nature Machine Intelligence,2(11):665–673, 2020.
Ghorbani, B., Firat, O., Freitag, M., Bapna, A., Krikun,
M., Garcia, X., Chelba, C., and Cherry, C. Scalinglaws for neural machine translation. arXiv preprint
arXiv:2109.07740, 2021.
Griewank, A. and Walther, A. Algorithm 799: revolve: animplementation of checkpointing for the reverse or adjoint mode of computational differentiation. ACM Transactions on Mathematical Software (TOMS), 26(1):19–45,
2000.
Gunter, K., Vaughn, C., and Kendall, T. Contextualizing/s/retraction: Sibilant variation and change in washington dc african american language. Language Variation
and Change, 33(3):331–357, 2021.
Harris, C. R., Millman, K. J., van der Walt, S. J., Gommers,
R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J.,
Berg, S., Smith, N. J., Kern, R., Picus, M., Hoyer, S., van
Kerkwijk, M. H., Brett, M., Haldane, A., Fernández delRı́o, J., Wiebe, M., Peterson, P., Gérard-Marchant, P.,
Sheppard, K., Reddy, T., Weckesser, W., Abbasi, H.,
Gohlke, C., and Oliphant, T. E. Array programmingwith NumPy. Nature, 585:357–362, 2020. doi: 10.1038/
s41586-020-2649-2.

Robust Speech Recognition via Large-Scale Weak Supervision

16

Hendrycks, D. and Gimpel, K. Gaussian error linear units(gelus). arXiv preprint arXiv:1606.08415, 2016.

Kuchaiev, O., Li, J., Nguyen, H., Hrinchuk, O., Leary, R.,
Ginsburg, B., Kriman, S., Beliaev, S., Lavrukhin, V.,
Cook, J., et al. Nemo: a toolkit for building ai applications
using neural modules. arXiv preprint arXiv:1909.09577,
2019.

Hendrycks, D., Liu, X., Wallace, E., Dziedzic, A., Krishnan,R., and Song, D. Pretrained transformers improve out-ofdistribution robustness. arXiv preprint arXiv:2004.06100,
2020.
Hernandez, F., Nguyen, V., Ghannay, S., Tomashenko, N. A.,
and Estève, Y. Ted-lium 3: twice as much data and corpus
repartition for experiments on speaker adaptation. In
SPECOM, 2018.
Hsu, W.-N., Bolte, B., Tsai, Y.-H. H., Lakhotia, K.,
Salakhutdinov, R., and Mohamed, A. Hubert: Selfsupervised speech representation learning by masked
prediction of hidden units. IEEE/ACM Transactions on
Audio, Speech, and Language Processing, 29:3451–3460,
2021a.
Hsu, W.-N., Sriram, A., Baevski, A., Likhomanenko, T.,
Xu, Q., Pratap, V., Kahn, J., Lee, A., Collobert, R., Synnaeve, G., et al. Robust wav2vec 2.0: Analyzing domain shift in self-supervised pre-training. arXiv preprint
arXiv:2104.01027, 2021b.
Huang, G., Sun, Y., Liu, Z., Sedra, D., and Weinberger,
K. Q. Deep networks with stochastic depth. In European
conference on computer vision, pp. 646–661. Springer,
2016.
Jia, R. and Liang, P. Adversarial examples for evaluating reading comprehension systems. arXiv preprint
arXiv:1707.07328, 2017.

Lake, B. M., Ullman, T. D., Tenenbaum, J. B., and Gershman, S. J. Building machines that learn and think likepeople. Behavioral and brain sciences, 40, 2017.
Liao, H., McDermott, E., and Senior, A. Large scale deepneural network acoustic modeling with semi-supervised
training data for youtube video transcription. In 2013
IEEE Workshop on Automatic Speech Recognition andUnderstanding, pp. 368–373. IEEE, 2013.
Likhomanenko, T., Xu, Q., Pratap, V., Tomasello, P., Kahn,
J., Avidov, G., Collobert, R., and Synnaeve, G. Rethinking evaluation in asr: Are our models robust enough?
arXiv preprint arXiv:2010.11745, 2020.
Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
Luong, M.-T., Le, Q. V., Sutskever, I., Vinyals, O., andKaiser, L. Multi-task sequence to sequence learning.
arXiv preprint arXiv:1511.06114, 2015.
Mahajan, D., Girshick, R., Ramanathan, V., He, K., Paluri,
M., Li, Y., Bharambe, A., and Van Der Maaten, L. Exploring the limits of weakly supervised pretraining. In
Proceedings of the European conference on computer
vision (ECCV), pp. 181–196, 2018.

Johnson, M., Schuster, M., Le, Q. V., Krikun, M., Wu, Y.,
Chen, Z., Thorat, N., Viégas, F., Wattenberg, M., Corrado,G., et al. Google’s multilingual neural machine translation
system: Enabling zero-shot translation. Transactions of
the Association for Computational Linguistics, 5:339–
351, 2017.

Mauch, M. and Ewert, S. The audio degradation toolbox andits application to robustness evaluation. In Proceedings ofthe 14th International Society for Music Information Retrieval Conference (ISMIR 2013), Curitiba, Brazil, 2013.
accepted.

Kendall, T. and Farrington, C. The corpus of regional
african american language. Version 2021.07. Eugene, OR:
The Online Resources for African American Language
Project. http://oraal.uoregon.edu/coraal,
2021. Accessed: 2022-09-01.

McCann, B., Keskar, N. S., Xiong, C., and Socher, R. The
natural language decathlon: Multitask learning as question answering. arXiv preprint arXiv:1806.08730, 2018.

Koenecke, A., Nam, A., Lake, E., Nudell, J., Quartey, M.,
Mengesha, Z., Toups, C., Rickford, J. R., Jurafsky, D.,
and Goel, S. Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences,
117(14):7684–7689, 2020.
Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung,J., Gelly, S., and Houlsby, N. Big transfer (bit): Generalvisual representation learning. In European conferenceon computer vision, pp. 491–507. Springer, 2020.

Meyer, J., Rauchenstein, L., Eisenberg, J. D., and Howell,
N. Artie bias corpus: An open dataset for detecting demographic bias in speech applications. In Proceedings ofthe 12th Language Resources and Evaluation Conference,
pp. 6462–6468, Marseille, France, May 2020. European
Language Resources Association. ISBN 979-10-9554634-4. URL https://aclanthology.org/2020.
lrec-1.796.
Miller, J., Krauth, K., Recht, B., and Schmidt, L. The effectof natural distribution shift on question answering models.
In ICML, 2020.

Robust Speech Recognition via Large-Scale Weak Supervision

17

Mohamed, A.-r., Dahl, G., Hinton, G., et al. Deep belief networks for phone recognition. In Nips workshop on deeplearning for speech recognition and related applications,volume 1, pp. 39, 2009.

Press, O. and Wolf, L. Using the output embedding to
improve language models. In Proceedings of the 15th
Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short
Papers, pp. 157–163, Valencia, Spain, April 2017. Association for Computational Linguistics. URL https://aclanthology.org/E17-2025.

Narayanan, A., Misra, A., Sim, K. C., Pundak, G., Tripathi,
A., Elfeky, M., Haghani, P., Strohman, T., and Bacchiani, M. Toward domain-invariant speech recognition via
large scale training. In 2018 IEEE Spoken Language
Technology Workshop (SLT), pp. 441–447. IEEE, 2018.
Panayotov, V., Chen, G., Povey, D., and Khudanpur, S.
Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on
acoustics, speech and signal processing (ICASSP), pp.
5206–5210. IEEE, 2015.
pandas development team, T. pandas-dev/pandas: Pandas, February 2020. URL https://doi.org/10.
5281/zenodo.3509134.
Park, D. S., Chan, W., Zhang, Y., Chiu, C.-C., Zoph, B.,
Cubuk, E. D., and Le, Q. V. SpecAugment: A simple data
augmentation method for automatic speech recognition.
arXiv preprint arXiv:1904.08779, 2019.
Pascanu, R., Mikolov, T., and Bengio, Y. On the difficultyof training recurrent neural networks. In International
conference on machine learning, pp. 1310–1318. PMLR,
2013.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J.,
Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga,
L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison,M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L.,
Bai, J., and Chintala, S. Pytorch: An imperative style,high-performance deep learning library. In Advancesin Neural Information Processing Systems 32, pp. 8024–
8035, 2019.
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V.,
Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P.,
Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay, E.
Scikit-learn: Machine learning in Python. Journal of
Machine Learning Research, 12:2825–2830, 2011.
Polyak, B. T. and Juditsky, A. B. Acceleration of stochastic
approximation by averaging. SIAM journal on control
and optimization, 30(4):838–855, 1992.
Pratap, V., Sriram, A., Tomasello, P., Hannun, A. Y.,
Liptchinsky, V., Synnaeve, G., and Collobert, R. Massively multilingual asr: 50 languages, 1 model, 1 billion
parameters. ArXiv, abs/2007.03001, 2020a.
Pratap, V., Xu, Q., Sriram, A., Synnaeve, G., and Collobert,
R. Mls: A large-scale multilingual dataset for speechresearch. arXiv preprint arXiv:2012.03411, 2020b.

Provilkov, I., Emelianenko, D., and Voita, E. Bpe-dropout:
Simple and effective subword regularization. arXiv
preprint arXiv:1910.13267, 2019.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., andSutskever, I. Language models are unsupervised multitask
learners. 2019.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G.,
Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark,J., Krueger, G., and Sutskever, I. Learning transferable
visual models from natural language supervision. arXiv
preprint arXiv:2103.00020, 2021.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S.,
Matena, M., Zhou, Y., Li, W., Liu, P. J., et al. Exploring
the limits of transfer learning with a unified text-to-texttransformer. J. Mach. Learn. Res., 21(140):1–67, 2020.
Ravanelli, M., Parcollet, T., Plantinga, P., Rouhe, A., Cornell, S., Lugosch, L., Subakan, C., Dawalatabad, N.,
Heba, A., Zhong, J., Chou, J.-C., Yeh, S.-L., Fu, S.-W.,
Liao, C.-F., Rastorgueva, E., Grondin, F., Aris, W., Na,
H., Gao, Y., Mori, R. D., and Bengio, Y. SpeechBrain: A
general-purpose speech toolkit, 2021. arXiv:2106.04624.
Recht, B., Roelofs, R., Schmidt, L., and Shankar, V.
Do ImageNet classifiers generalize to ImageNet? In
Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine
Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5389–5400. PMLR, 09–15 Jun 2019.
URL https://proceedings.mlr.press/v97/
recht19a.html.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S.,
Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein,M., et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211–252, 2015.
Schultz, T. and Kirchhoff, K. Multilingual speech processing. Elsevier, 2006.
Seide, F., Li, G., Chen, X., and Yu, D. Feature engineeringin context-dependent deep neural networks for conversational speech transcription. In 2011 IEEE Workshop
on Automatic Speech Recognition & Understanding, pp.
24–29. IEEE, 2011.

Robust Speech Recognition via Large-Scale Weak Supervision18

Sennrich, R., Haddow, B., and Birch, A. Neural machinetranslation of rare words with subword units. arXiv
preprint arXiv:1508.07909, 2015.

Wang, C., Wu, A., and Pino, J. Covost 2 and massively
multilingual speech-to-text translation. arXiv preprint
arXiv:2007.10310, 2020b.

Speer, R. ftfy. Zenodo, 2019. URL https://doi.org/
10.5281/zenodo.2591652. Version 5.5.

Wang, C., Riviere, M., Lee, A., Wu, A., Talnikar, C., Haziza,D., Williamson, M., Pino, J., and Dupoux, E. Voxpopuli:
A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation.
arXiv preprint arXiv:2101.00390, 2021.

Sutskever, I., Vinyals, O., and Le, Q. V. Sequence to sequence learning with neural networks. Advances in neural
information processing systems, 27, 2014.

Wang, P., Sainath, T. N., and Weiss, R. J. Multitask training
with text data for end-to-end speech recognition. arXiv
Taori, R., Dave, A., Shankar, V., Carlini, N., Recht, B.,
preprint arXiv:2010.14318, 2020c.
and Schmidt, L. Measuring robustness to natural
distribution shifts in image classification. In Larochelle,
Watanabe, S., Mandel, M., Barker, J., Vincent, E., Arora,
H., Ranzato, M., Hadsell, R., Balcan, M., and Lin,A., Chang, X., Khudanpur, S., Manohar, V., Povey, D.,
H. (eds.), Advances in Neural Information Processing
Raj, D., et al. Chime-6 challenge: Tackling multispeaker
Systems, volume 33, pp. 18583–18599. Curran Assospeech recognition for unsegmented recordings. arXiv
ciates, Inc., 2020. URL https://proceedings.
preprint arXiv:2004.09249, 2020.
neurips.cc/paper/2020/file/d8330f857a17c53d217014ee776bfd50-Paper. Xu, Q., Baevski, A., Likhomanenko, T., Tomasello, P., Conpdf.
neau, A., Collobert, R., Synnaeve, G., and Auli, M. Selftraining and pre-training are complementary for speech
Torralba, A. and Efros, A. A. Unbiased look at dataset bias.
recognition. In ICASSP 2021-2021 IEEE International
CVPR 2011, pp. 1521–1528, 2011.
Conference on Acoustics, Speech and Signal Processing(ICASSP), pp. 3030–3034. IEEE, 2021.
Toshniwal, S., Sainath, T. N., Weiss, R. J., Li, B., Moreno,
P. J., Weinstein, E., and Rao, K. Multilingual speech
Zhang, Y., Qin, J., Park, D. S., Han, W., Chiu, C.-C., Pang,
recognition with a single end-to-end model. 2018 IEEE
R., Le, Q. V., and Wu, Y. Pushing the limits of semiInternational Conference on Acoustics, Speech and Sigsupervised learning for automatic speech recognition.
nal Processing (ICASSP), pp. 4904–4908, 2018.
arXiv preprint arXiv:2010.10504, 2020.
Valk, J. and Alumäe, T. Voxlingua107: a dataset for spoken
language recognition. In 2021 IEEE Spoken Language
Technology Workshop (SLT), pp. 652–658. IEEE, 2021.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones,
L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. In Advances in neural information
processing systems, pp. 5998–6008, 2017.
Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M.,
Reddy, T., Cournapeau, D., Burovski, E., Peterson, P.,
Weckesser, W., Bright, J., van der Walt, S. J., Brett, M.,
Wilson, J., Millman, K. J., Mayorov, N., Nelson, A. R. J.,
Jones, E., Kern, R., Larson, E., Carey, C. J., Polat, İ.,Feng, Y., Moore, E. W., VanderPlas, J., Laxalde, D.,
Perktold, J., Cimrman, R., Henriksen, I., Quintero, E. A.,
Harris, C. R., Archibald, A. M., Ribeiro, A. H., Pedregosa,
F., van Mulbregt, P., and SciPy 1.0 Contributors. SciPy
1.0: Fundamental Algorithms for Scientific Computing
in Python. Nature Methods, 17:261–272, 2020. doi:
10.1038/s41592-019-0686-2.
Wang, C., Tang, Y., Ma, X., Wu, A., Okhonko, D., and Pino,
J. fairseq s2t: Fast speech-to-text modeling with fairseq.
arXiv preprint arXiv:2010.05171, 2020a.

Zhang, Y., Park, D. S., Han, W., Qin, J., Gulati, A., Shor, J.,
Jansen, A., Xu, Y., Huang, Y., Wang, S., et al. BigSSL:
Exploring the frontier of large-scale semi-supervised
learning for automatic speech recognition. arXiv preprint
arXiv:2109.13226, 2021.

Robust Speech Recognition via Large-Scale Weak Supervision

19

A. Evaluation Datasets.
A.1. Short-form English-only datasets• LibriSpeech (Panayotov et al., 2015): We used the test-clean and test-other splits from the LibriSpeech ASR corpus.
• TED-LIUM 3 (Hernandez et al., 2018): We used the test split of TED-LIUM Release 3, using the segmented manualtranscripts included in the release.
• Common Voice 5.1 (Ardila et al., 2019): We downloaded the English subset of Common Voice Corpus 5.1 from theofficial website.
• Artie bias corpus (Meyer et al., 2020): We used the Artie bias corpus. This is a subset of the Common Voice dataset.
• CallHome and Switchboard: We used the two corpora from LDC2002S09 and LDC2002T43.
• WSJ: We used LDC93S6B and LDC94S13B and followed the s5 recipe to preprocess the dataset.
• CORAAL: We used the 231 interviews from CORAAL (Kendall & Farrington, 2021) and used the preprocessing
script from the FairSpeech project.
• CHiME-6: For CHiME-6 (Watanabe et al., 2020), we downloaded the CHiME-5 dataset and followed the stage 0of the s5 track1 recipe to create the CHiME-6 dataset which fixes synchronization. We then used the binaural
recordings (* P??.wav) and the corresponding transcripts.
• AMI-IHM and AMI-SDM1: We preprocessed the AMI Corpus by following the stage 0 ad 2 of the s5b recipe.
A.2. Long-form English-only datasets• TED-LIUM 3 (Hernandez et al., 2018): We used the 11 full-length TED talks from the test split of TED-LIUM
Release 3, slicing the source audio files between the beginning of the first labeled segment and the end of the lastlabeled segment of each talk, and we used the concatenated text as the label.
• Meanwhile: This dataset consists of 64 segments from The Late Show with Stephen Colbert. The YouTube video ID
and the corresponding start and end timestamps are available as part of the code release. The labels are collected fromthe closed-caption data for each video and corrected with manual inspection.
• Rev16: We use a subset of 16 files from the 30 podcast episodes in Rev.AI’s Podcast Transcription Benchmark, after
finding that there are multiple cases where a significant portion of the audio and the labels did not match, mostly on theparts introducing the sponsors. We selected 16 episodes that do not have this error, whose “file number”s are:
3 4 9 10 11 14 17 18 20 21 23 24 26 27 29 32• Kincaid46: This dataset consists of 46 audio files and the corresponding transcripts compiled in the blog article ¡Whichautomatic transcription service is the most accurate - 2018¿ by Jason Kincaid. We used the 46 audio files and referencetranscripts from the Airtable widget in the article. For the human transcription benchmark in the paper, we use a subset
of 25 examples from this data, whose “Ref ID”s are:2 4 5 8 9 10 12 13 14 16 19 21 23 25 26 28 29 30 33 35 36 37 42 43 45• Earnings-21 (Del Rio et al., 2021) and Earnings-22: We used the files available in the speech-datasets repository, asof their 202206 version.
• CORAAL: We used the 231 full-length interviews and transcripts from (Kendall & Farrington, 2021).

Robust Speech Recognition via Large-Scale Weak Supervision20

A.3. Multilingual datasets• Multilingual LibriSpeech (Pratap et al., 2020b): We used the test splits from each language in the Multilingual
LibriSpeech (MLS) corpus.
• Fleurs (Conneau et al., 2022): We collected audio files and transcripts using the implementation available as HuggingFace datasets. To use as a translation dataset, we matched the numerical utterance IDs to find the correspondingtranscript in English.
• VoxPopuli (Wang et al., 2021): We used the get asr data.py script from the official repository to collect the ASR
data in 16 languages, including English.
• Common Voice 9 (Ardila et al., 2019): We downloaded the Common Voice Corpus 9 from the official website.
• CoVOST 2 (Wang et al., 2020b): We collected the X into English data collected using the official repository.

B. Compared Models
For comparison, we use the following models from HuggingFace, downloaded as of September 2022 using version 4.21.0 of
the transformers library:• facebook/wav2vec2-large-960h-lv60-self (Xu et al., 2021)• facebook/wav2vec2-large-robust-ft-libri-960h (Hsu et al., 2021b)• facebook/wav2vec2-base-100h (Baevski et al., 2020)• facebook/wav2vec2-base-960h (Baevski et al., 2020)• facebook/wav2vec2-large-960h (Baevski et al., 2020)• facebook/hubert-large-ls960-ft (Hsu et al., 2021a)• facebook/hubert-xlarge-ls960-ft (Hsu et al., 2021a)• facebook/s2t-medium-librispeech-asr (Wang et al., 2020a)• facebook/s2t-large-librispeech-asr (Wang et al., 2020a)• microsoft/unispeech-sat-base-100h-libri-ft (Chen et al., 2022a)• nvidia/stt en conformer ctc large (Kuchaiev et al., 2019)• nvidia/stt en conformer transducer xlarge (Kuchaiev et al., 2019)• speechbrain/asr-crdnn-rnnlm-librispeech (Ravanelli et al., 2021)• speechbrain/asr-transformer-transformerlm-librispeech (Ravanelli et al., 2021)
We note that all of the models above are entirely or partly trained on LibriSpeech.

Robust Speech Recognition via Large-Scale Weak Supervision

21

C. Text Standardization
Since Whisper may output any UTF-8 string rather than a restricted set of graphemes, the rules for text standardization need
to be more intricate and comprehensive than those defined on e.g. ASCII characters. We perform the following steps tonormalize English texts in different styles into a standardized form, which is a best-effort attempt to penalize only when aword error is caused by actually mistranscribing a word, and not by formatting or punctuation differences.
1. Remove any phrases between matching brackets ([, ]).
2. Remove any phrases between matching parentheses ((, )).
3. Remove any of the following words: hmm, mm, mhm, mmm, uh, um
4. Remove whitespace characters that comes before an apostrophe ’
5. Convert standard or informal contracted forms of English into the original form.
6. Remove commas (,) between digits
7. Remove periods (.) not followed by numbers
8. Remove symbols as well as diacritics from the text, where symbols are the characters with the Unicode category
starting with M, S, or P, except period, percent, and currency symbols that may be detected in the next step.
9. Detect any numeric expressions of numbers and currencies and replace with a form using Arabic numbers, e.g. “Tenthousand dollars” → “$10000”.
10. Convert British spellings into American spellings.
11. Remove remaining symbols that are not part of any numeric expressions.
12. Replace any successive whitespace characters with a space.
A different, language-specific set of transformations would be needed to equivalently normalize non-English text, but due toour lack of linguistic knowledge to build such normalizers for all languages, we resort to the following basic standardization
for non-English text:
1. Remove any phrases between matching brackets ([, ]).
2. Remove any phrases between matching parentheses ((, )).
3. Replace any markers, symbols, and punctuation characters with a space, i.e. when the Unicode category of eachcharacter in the NFKC-normalized string starts with M, S, or P.
4. make the text lowercase.
5. replace any successive whitespace characters with a space.
Additionally, we put a space between every letter for the languages that do not use spaces to separate words, namely Chinese,
Japanese, Thai, Lao, and Burmese, effectively measuring the character error rate instead.
We note that the above is an imperfect solution, and it will sometimes produce unintended and unexpected outputs. We do
not claim that the text format resulting from the above is more “correct” in any measure. Rather, the procedures above aredesigned to better distinguish between innocuous differences in wording and genuine mistranscriptions. Python code forthe standardization procedures above is available as part of our code and model release to facilitate future iterations andimprovements on text standardization.

22

Robust Speech Recognition via Large-Scale Weak Supervision

D. Raw Performance Tables
D.1. English Transcription
D.1.1. G REEDY DECODING

Model

LibriSpeech.test-clean

LibriSpeech.test-other

TED-LIUM3

WSJ

CallHome

Switchboard

CommonVoice5.1

Artie

CORAALCHiME6

AMI-IHMAMI-SDM1

VoxPopuli.en

Fleurs.en us

Whisper tiny.en
Whisper tiny
Whisper base.en
Whisper base
Whisper small.en
Whisper small
Whisper medium.en
Whisper medium
Whisper large
Whisper large-v2

5.6
7.6
4.2
5.0
3.1
3.4
3.1
2.9
2.7
2.7

14.6
16.9
10.2
12.4
7.4
7.6
6.3
5.9
5.6
5.2

6.0
7.0
4.9
5.5
4.0
4.3
4.1
3.8
4.0
4.0

5.0
6.7
4.6
5.1
3.3
4.0
3.3
2.9
3.1
3.9

24.1
30.0
20.9
23.0
18.2
17.5
16.2
16.4
15.8
17.6

17.8
22.8
15.2
16.8
15.7
14.5
14.1
14.0
13.1
13.8

26.3
29.6
19.0
21.6
13.1
13.5
10.6
10.3
9.5
9.0

20.0
23.9
13.4
16.9
9.7
10.3
7.6
7.2
6.7
6.2

23.9
31.0
22.6
26.0
20.2
18.1
17.5
16.6
19.4
16.2

41.3
49.6
36.4
40.2
27.6
29.3
25.3
26.4
25.6
25.5

23.7
27.6
20.5
22.0
17.5
19.0
16.4
16.6
16.4
16.9

50.3
58.1
46.7
49.9
38.0
39.6
37.2
36.0
36.9
36.4

11.7
12.7
10.0
10.0
8.1
8.3
7.4
7.4
7.3
7.3

11.6
13.7
7.6
10.1
6.0
6.6
5.0
5.4
4.6
4.4

wav2vec2-base-100h
wav2vec2-base-960h
wav2vec2-large-960h-lv60-self
wav2vec2-large-960h
wav2vec2-large-robust-ft-libri-960h
asr-crdnn-rnnlm-librispeech
asr-transformer-transformerlm-librispeech
hubert-large-ls960-ft
hubert-xlarge-ls960-ft
s2t-large-librispeech-asr
s2t-medium-librispeech-asr
stt en conformer ctc large
stt en conformer transducer xlarge
unispeech-sat-base-100h-libri-ft

6.0
3.3
1.8
2.7
2.6
3.0
2.1
2.0
1.9
3.3
3.6
2.1
1.5
5.7

13.4
8.5
3.8
6.2
5.3
9.7
5.4
4.1
3.5
8.1
8.2
4.2
2.8
13.8

17.8
12.8
7.4
10.5
9.2
17.7
11.9
8.4
8.3
14.9
15.7
4.4
4.3
17.7

13.9
8.9
4.4
7.7
6.1
10.7
7.4
5.4
5.4
9.4
9.7
2.1
1.2
13.6

46.9
40.6
29.1
34.8
23.4
59.7
38.9
29.6
29.3
54.5
58.1
11.3
12.0
46.5

40.2
32.9
22.2
28.3
19.8
56.1
33.0
22.8
22.2
40.3
42.4
8.2
7.4
40.0

47.4
36.4
19.9
29.9
20.3
43.7
30.6
20.8
19.8
38.1
39.3
7.4
4.3
45.3

40.8
30.9
15.8
24.5
16.2
33.3
23.5
16.0
14.8
30.7
31.3
4.0
1.5
38.6

47.0
39.9
29.2
35.6
29.4
83.8
44.9
32.0
31.5
50.2
52.6
13.5
19.9
44.7

79.9
68.5
56.3
65.8
58.1
81.0
79.5
60.0
58.5
79.2
79.8
30.5
36.8
74.8

48.1
40.2
30.8
37.0
31.7
57.2
44.5
33.7
33.3
53.4
60.3
15.9
20.5
47.8

81.2
71.9
57.0
67.6
61.6
85.8
75.4
59.1
58.9
79.5
85.3
39.9
48.6
77.7

28.9
21.4
13.0
17.9
15.1
30.6
17.8
14.4
14.2
21.6
22.9
6.7
6.0
29.8

23.1
17.4
10.2
14.6
11.8
32.4
17.0
10.9
10.5
18.0
19.7
8.2
6.3
22.4

Table 8. English transcription WER (%) with greedy decoding

D.1.2. B EAM SEARCH WITH TEMPERATURE FALLBACK

Model

LibriSpeech.test-clean

LibriSpeech.test-other

TED-LIUM3

WSJ

CallHome

Switchboard

CommonVoice5.1

Artie

CORAALCHiME6

AMI-IHMAMI-SDM1

VoxPopuli.en

Fleurs.en us

Whisper tiny.en
Whisper tiny
Whisper base.en
Whisper base
Whisper small.en
Whisper small
Whisper medium.en
Whisper medium
Whisper large
Whisper large-v2

5.4
6.7
4.1
4.9
3.2
3.3
3.0
2.7
2.8
2.5

12.8
15.0
9.6
11.0
6.7
7.2
5.7
5.6
5.7
4.9

5.4
6.3
4.6
5.0
4.3
4.3
4.3
4.0
4.3
3.7

4.6
5.9
4.0
4.4
3.0
3.9
2.8
2.7
3.5
2.6

21.4
24.8
18.3
20.5
17.2
17.1
14.7
15.3
16.2
16.4

16.0
18.3
14.2
15.6
13.4
13.3
12.4
13.2
14.2
13.6

23.5
26.1
17.5
19.4
12.6
12.8
10.3
9.7
8.9
8.2

18.4
20.8
13.2
15.3
9.2
9.3
7.4
6.7
6.4
5.7

21.4
25.1
18.5
20.5
17.5
16.4
15.3
14.9
15.1
14.2

42.0
48.0
35.2
40.0
29.5
30.9
27.0
27.6
25.2
24.9

22.7
25.6
21.1
21.5
17.9
19.2
17.1
17.6
17.6
17.4

54.2
57.3
49.0
50.0
42.5
43.5
39.4
43.0
37.1
39.9

10.9
11.6
9.3
9.5
8.1
8.2
7.8
7.6
7.2
7.0

10.0
12.4
7.1
8.9
5.3
6.1
4.5
4.4
4.5
4.2

Table 9. English transcription WER (%) with beam search and temperature fallback

23

Robust Speech Recognition via Large-Scale Weak Supervision

D.2. Multilingual Transcription
D.2.1. M ULTILINGUAL L IBRI S PEECH

Dutch

EnglishFrenchGermanItalianPolishPortuguese

Spanish

39.4
28.4
17.2
11.7
10.2
9.3

15.7
11.7
8.3
6.8
6.3
6.2

36.8
26.6
16.2
8.9
8.9
7.3

24.9
17.7
10.5
7.4
6.6
5.5

41.7
31.1
21.4
16.0
14.3
13.8

34.2
22.8
11.2
6.5
6.6
5.0

31.3
21.9
13.0
9.0
9.2
6.8

19.2
12.8
7.8
5.3
5.4
4.2

Model
Whisper tiny
Whisper base
Whisper small
Whisper medium
Whisper large
Whisper large-v2

Table 10. WER (%) on MLS

D.2.2. C OMMON VOICE 9
Bulgarian

Bengali

Catalan

Czech

Welsh

DanishGermanGreek

EnglishSpanishEstonian

Persian

90.9
84.4
66.4
60.3
56.0
53.8

79.3
68.1
44.8
26.7
24.1
19.9

104.1
103.7
118.6
124.7
106.0
103.4

51.0
39.9
23.8
16.4
15.3
14.1

79.7
63.1
34.1
18.8
17.1
13.5

101.8
93.8
65.4
43.6
40.3
34.2

77.2
57.5
32.1
19.3
18.3
14.4

34.5
24.5
13.0
8.5
7.7
6.4

61.9
51.5
31.7
20.0
18.3
16.0

28.8
21.9
14.5
11.2
10.1
9.4

30.3
19.6
10.3
6.9
6.4
5.6

102.1
88.1
67.2
45.6
41.4
35.1

120.3
99.0
71.9
49.9
44.8
39.4

FinnishFrenchHindi

HungarianIndonesian

ItalianJapanese

LithuanianLatvian

MalayalamMongolian

Dutch

PolishModel
Whisper tiny
Whisper base
Whisper small
Whisper medium
Whisper large
Whisper large-v2

ArabicModel
Whisper tiny
Whisper base
Whisper small
Whisper medium
Whisper large
Whisper large-v2

68.5
52.9
30.5
18.8
17.0
14.4

49.7
37.3
22.7
16.0
14.7
13.9

108.3
106.5
43.6
31.5
25.0
21.9

87.0
71.9
44.4
26.9
23.5
19.7

49.6
36.1
18.4
11.6
10.6
8.5

44.5
30.5
16.0
9.4
8.1
7.1

36.1
24.2
14.0
10.5
9.4
9.1

103.5
91.3
72.8
49.4
43.9
35.2

87.8
78.0
54.6
37.2
34.8
25.5

102.7
122.9
104.8
137.8
107.1
103.2

123.0
137.0
225.8
113.4
117.4
128.4

43.6
29.5
14.2
8.0
7.1
5.8

45.3
32.8
16.9
10.1
9.0
7.6

Model

Portuguese

Romanian

RussianSlovakSlovenian

SerbianSwedish

Tamil

Thai

TurkishUrdu

Vietnamese

ChineseWhisper tiny
Whisper base
Whisper small
Whisper medium
Whisper large
Whisper large-v2

35.2
23.7
12.5
8.1
7.1
6.3

68.2
55.9
33.2
21.5
19.8
15.8

40.6
28.8
15.0
9.3
8.2
7.1

104.0
87.2
60.4
42.0
37.9
31.9

82.0
70.3
45.5
29.8
25.1
20.6

106.1
103.0
101.3
85.6
87.4
70.5

58.2
42.4
22.1
13.7
12.4
10.6

105.7
49.5
28.7
19.6
17.6
16.1

55.9
32.1
18.1
10.5
8.8
8.0

53.6
38.6
23.7
17.7
16.6
14.5

74.7
58.6
39.1
29.9
28.1
24.2

69.3
51.6
33.3
24.4
19.9
18.2

52.4
44.9
29.4
23.2
29.1
26.8

Table 11. WER (%) on CommonVoice9

D.2.3. VOX P OPULI

Czech

GermanEnglishen accented

SpanishEstonian

FinnishFrenchCroatian

Hungarian

ItalianLithuanian

Dutch

PolishRomanian

SlovakSlovenian

Model
Whisper tiny
Whisper base
Whisper small
Whisper medium
Whisper large
Whisper large-v2

73.5
54.7
28.8
18.4
15.9
12.6

27.4
20.6
14.8
12.4
11.9
11.2

11.6
9.5
8.2
7.6
7.2
7.0

18.8
17.5
19.2
19.1
20.8
18.6

19.7
14.4
11.1
9.6
8.8
8.2

99.2
83.0
59.2
38.2
33.3
28.7

54.1
39.7
24.9
16.6
15.5
12.4

32.9
24.9
15.7
12.2
11.0
11.4

72.4
53.6
33.7
23.9
19.0
16.1

74.5
52.6
31.3
19.3
16.8
13.8

40.5
30.8
22.9
19.7
18.4
19.0

93.1
82.1
60.1
39.3
35.0
33.2

41.9
29.4
18.8
14.9
14.0
12.9

31.4
22.1
13.3
10.1
9.0
7.8

65.9
49.3
28.6
18.4
17.0
14.4

78.7
63.7
37.3
23.0
19.1
15.4

81.9
70.5
50.8
36.3
31.3
27.9

Table 12. WER (%) on VoxPopuli

24

Robust Speech Recognition via Large-Scale Weak Supervision

D.2.4. F LEURS

Afrikaans

Amharic

ArabicAssamese

Azerbaijani

Belarusian

Bulgarian

BengaliBosnianCatalanChinese

Czech

Welsh

Danish

91.2
81.5
61.1
44.9
42.6
36.7

122.9
196.8
120.2
229.3
129.3
140.3

63.4
48.8
30.6
20.4
18.1
16.0

102.0
102.0
108.0
102.3
105.6
106.2

93.1
76.4
49.1
33.1
28.7
23.4

94.0
91.3
75.1
60.4
56.6
45.4

81.0
65.1
37.3
21.4
18.4
14.6

101.6
100.6
104.4
100.6
104.9
104.1

82.1
66.7
39.4
23.9
20.7
15.7

42.8
29.0
16.2
9.6
8.0
7.3

40.5
34.1
20.8
12.1
19.6
14.7

82.8
66.0
37.6
21.3
17.4
13.3

101.3
85.3
59.3
40.8
36.6
33.0

82.0
57.6
32.8
19.5
16.8
13.8

Model
Whisper tiny
Whisper base
Whisper small
Whisper medium
Whisper large
Whisper large-v2

GermanGreek

EnglishSpanishEstonian

PersianFinnishTagalog

FrenchGalician

Gujarati

Hausa

Hebrew

Hindi

27.8
17.9
10.2
6.5
5.5
4.5

67.4
53.5
30.8
19.0
18.7
12.5

12.4
8.9
6.1
4.4
4.5
4.2

15.9
9.9
5.6
3.6
3.5
3.0

94.8
77.9
51.3
29.8
25.5
21.9

101.8
86.1
55.8
41.0
36.1
32.9

59.5
43.1
24.0
13.9
12.2
9.7

65.6
45.8
27.7
19.1
15.8
13.8

41.4
28.5
15.0
8.7
7.7
8.3

54.8
47.4
30.2
21.2
19.0
15.4

101.2
101.4
106.4
104.8
103.9
102.7

100.2
98.6
90.1
106.6
87.0
88.9

71.6
61.7
44.4
33.1
30.2
27.1

102.3
101.1
38.4
26.8
26.9
21.5

Model
Whisper tiny
Whisper base
Whisper small
Whisper medium
Whisper large
Whisper large-v2

Hungarian

ArmenianIndonesian

Icelandic

ItalianJapanese

JavaneseGeorgian

KazakhKhmer

KannadaKoreanLuxembourgish

Whisper tiny
Whisper base
Whisper small
Whisper medium
Whisper large
Whisper large-v2

Croatian

Model

79.0
59.1
33.4
19.3
16.7
13.4

83.8
65.0
38.9
24.3
21.0
17.0

118.6
126.3
86.6
60.1
53.7
44.6

51.7
33.1
16.3
10.2
8.5
7.1

113.3
95.5
72.6
49.9
43.0
38.2

29.8
17.9
9.8
5.2
4.2
4.0

37.0
22.8
12.0
7.1
6.4
5.3

107.3
89.5
88.6
67.9
87.0
nan

123.0
114.7
118.3
117.3
100.5
105.0

165.2
109.2
70.3
48.8
43.8
37.7

100.6
101.6
104.4
98.9
96.0
99.7

100.7
107.2
100.4
77.7
69.8
37.0

36.1
27.8
19.6
16.4
15.2
14.3

99.1
100.7
100.1
90.0
86.5
88.0

Model

LingalaLao

LithuanianLatvianMaori

Macedonian

MalayalamMongolian

Marathi

Malay

MalteseMyanmar

Norwegian

Nepali

Whisper tiny
Whisper base
Whisper small
Whisper medium
Whisper large
Whisper large-v2

105.4
96.7
91.3
83.2
76.8
75.6

115.1
105.1
102.2
101.4
101.6
101.5

98.5
87.3
65.6
41.1
35.2
28.1

91.6
79.8
53.2
32.0
28.3
23.1

94.5
77.5
59.5
77.8
45.7
38.5

73.3
59.9
36.9
22.0
20.6
16.5

101.5
107.4
100.9
101.1
101.4
100.7

113.7
125.7
144.2
103.7
106.2
110.5

100.3
100.3
60.2
63.2
43.7
38.3

51.2
35.1
18.9
12.2
10.2
8.7

100.8
97.6
92.2
83.2
80.5
76.6

124.8
122.6
110.1
123.0
124.5
115.7

62.0
44.0
24.2
12.9
11.4
9.5

101.8
102.4
69.5
54.4
52.2
47.1

Dutch

Occitan

Punjabi

PolishPashtoPortuguese

Romanian

RussianSindhi

SlovakSlovenian

Shona

Somali

Serbian

49.0
33.0
16.4
9.9
8.3
6.7

95.9
82.9
87.3
79.5
75.9
75.3

102.6
101.5
103.6
102.0
102.8
102.4

45.6
30.8
14.7
8.0
7.2
5.4

105.6
99.0
92.9
119.4
92.7
93.7

20.1
13.0
7.3
5.0
4.8
4.3

74.7
56.0
29.8
20.0
15.4
14.4

31.1
20.5
11.4
7.2
6.4
5.6

105.8
103.9
131.7
147.0
177.9
156.5

77.2
60.6
33.3
17.3
15.7
11.7

87.2
74.6
49.3
31.9
27.8
23.1

128.1
126.0
140.0
143.9
130.0
121.0

105.6
109.6
105.3
104.0
103.5
102.9

83.7
64.3
42.2
44.9
29.2
33.9

SwedishSwahili

Tamil

Telugu

Tajik

Thai

TurkishUkrainian

Urdu

Uzbek

Vietnamese

YorubaWhisper tiny
Whisper base
Whisper small
Whisper medium
Whisper large
Whisper large-v2

52.7
37.4
20.8
11.2
10.5
8.5

100.9
92.5
73.7
52.8
47.9
39.3

99.9
58.7
35.2
23.1
20.6
17.5

105.1
105.2
98.2
82.8
100.6
99.0

101.7
109.3
84.3
74.0
74.5
85.8

58.8
38.2
21.9
15.4
13.2
11.5

42.5
27.5
15.9
10.4
9.4
8.4

51.2
37.7
19.3
11.6
10.3
8.6

65.2
52.0
37.3
28.2
25.0
22.6

105.2
114.0
107.7
109.6
93.3
90.2

60.0
40.5
21.2
12.7
10.7
10.3

106.4
101.8
116.4
105.1
111.7
94.8

Model
Whisper tiny
Whisper base
Whisper small
Whisper medium
Whisper large
Whisper large-v2

Model

Table 13. WER (%) on Fleurs25

Robust Speech Recognition via Large-Scale Weak Supervision

D.3. Speech Translation
D.3.1. F LEURS

ArabicAssamese

Azerbaijani

Belarusian

Bulgarian

BengaliBosnianCatalanChinese

Czech

Welsh

Danish

0.1
0.3
0.2
0.9
1.1
1.9

0.1
1.0
10.6
19.9
23.8
25.5

0.4
0.4
1.2
3.5
3.9
5.4

0.1
0.8
5.8
11.7
13.1
13.7

0.8
3.3
7.1
9.8
11.0
11.7

0.4
2.7
14.8
23.9
26.2
28.5

0.4
0.7
2.7
10.6
12.0
13.2

0.4
4.1
16.8
26.0
28.0
29.7

5.2
13.1
25.1
31.9
33.7
34.2

0.6
1.9
9.3
15.1
16.8
18.4

0.6
2.7
14.2
23.6
25.6
27.8

0.6
0.7
1.3
8.4
11.2
13.0

0.7
5.0
18.1
28.6
31.6
32.7

Greek

EnglishSpanishEstonian

PersianFinnishTagalog

FrenchGalician

Gujarati

Hausa

Hebrew

Hindi

5.2
13.7
25.9
31.4
34.3
34.6

0.1
0.7
11.6
19.9
21.7
23.7

68.6
73.3
77.3
79.2
77.8
80.2

7.7
12.4
18.2
21.4
22.8
23.3

0.1
0.3
3.6
13.5
15.9
18.7

0.1
0.2
5.8
15.0
17.6
19.6

0.2
0.5
7.3
18.5
20.6
22.1

0.8
2.1
12.0
20.5
22.7
24.4

4.7
13.1
23.5
28.6
31.6
32.2

4.0
10.5
17.5
24.7
26.0
27.9

0.7
1.5
3.9
12.8
14.8
16.2

0.1
0.0
0.3
0.5
0.5
0.4

0.2
0.6
5.4
15.9
19.6
21.8

1.0
3.4
11.1
19.4
20.7
22.0

Croatian

Hungarian

ArmenianIndonesian

Icelandic

ItalianJapanese

JavaneseGeorgian

KazakhKhmer

KannadaKoreanLuxembourgish

Model
Whisper tiny
Whisper base
Whisper small
Whisper medium
Whisper large
Whisper large-v2

1.6
4.4
18.1
29.5
31.6
34.1
GermanModel
Whisper tiny
Whisper base
Whisper small
Whisper medium
Whisper large
Whisper large-v2

Amharic

Whisper tiny
Whisper base
Whisper small
Whisper medium
Whisper large
Whisper large-v2

Afrikaans

Model

0.6
3.7
14.6
23.0
25.4
27.0

0.1
0.2
4.8
15.5
18.3
21.2

0.1
0.1
0.7
10.4
13.2
16.0

0.3
2.6
16.4
24.1
27.2
29.1

0.4
0.4
1.8
6.8
6.6
9.1

5.3
11.3
17.8
21.6
23.5
23.6

0.2
1.5
9.6
14.9
17.0
18.9

0.2
0.2
1.4
5.0
5.1
6.2

0.1
0.2
0.2
1.3
2.7
2.4

0.1
0.2
0.8
4.3
6.3
5.4

0.1
0.1
0.5
3.3
5.2
6.1

0.8
0.9
2.3
8.5
9.9
11.6

0.5
3.7
12.2
19.2
20.0
21.3

0.8
1.7
5.7
13.6
15.4
16.8

Model

LingalaLao

LithuanianLatvianMaori

Macedonian

MalayalamMongolian

Marathi

Malay

MalteseMyanmar

Norwegian

Nepali

Whisper tiny
Whisper base
Whisper small
Whisper medium
Whisper large
Whisper large-v2

0.1
0.1
0.5
0.9
1.2
1.0

0.2
0.3
2.0
8.1
9.3
11.0

0.1
0.3
1.9
9.6
12.0
14.0

0.2
0.4
1.5
10.0
12.5
14.3

0.3
1.0
3.9
8.5
9.4
10.2

1.0
5.4
15.3
23.5
26.4
27.7

0.8
1.4
5.7
13.8
16.5
16.7

0.1
0.1
0.1
0.5
1.0
1.0

0.2
0.9
3.8
10.9
13.1
12.9

0.3
2.1
14.1
23.2
25.5
27.3

0.6
1.4
4.9
11.2
12.8
13.5

0.1
0.1
0.0
0.2
0.5
0.4

1.4
8.4
22.0
29.1
30.5
31.4

0.1
0.3
2.9
12.7
12.9
16.1

Dutch

Occitan

Punjabi

PolishPashtoPortuguese

Romanian

RussianSindhi

SlovakSlovenian

Shona

Somali

Serbian

2.7
7.5
15.9
21.6
22.8
24.0

1.7
4.2
9.5
15.9
16.8
20.2

0.3
1.1
4.4
12.8
14.6
15.7

0.8
5.1
14.0
19.0
21.4
22.3

0.3
0.4
0.8
2.1
3.7
3.4

12.1
22.4
31.2
35.9
37.4
38.1

1.0
4.9
18.3
26.6
29.1
31.5

3.1
12.1
19.7
24.8
26.7
27.8

0.5
0.7
2.0
5.5
5.9
5.7

0.7
4.6
14.4
22.7
25.1
26.1

0.3
1.3
6.9
14.0
16.9
17.0

0.1
0.3
0.6
1.4
1.8
1.8

0.0
0.1
0.1
0.4
0.5
0.7

0.6
5.4
19.3
27.7
30.5
32.5

SwedishSwahili

Tamil

Telugu

Tajik

Thai

TurkishUkrainian

Urdu

Uzbek

Vietnamese

YorubaWhisper tiny
Whisper base
Whisper small
Whisper medium
Whisper large
Whisper large-v2

1.8
9.1
22.9
32.1
33.1
35.3

0.1
0.1
0.1
3.1
5.3
7.2

0.2
0.4
2.1
7.0
8.5
9.2

0.3
0.4
4.0
10.8
10.9
12.5

0.2
0.2
4.4
11.4
13.0
14.5

0.2
0.7
5.8
12.8
15.2
16.1

0.2
2.4
15.7
22.9
25.7
26.6

1.2
6.9
18.7
25.8
28.0
29.4

0.4
1.5
8.8
14.9
16.3
17.2

0.0
0.2
0.5
3.8
5.8
6.0

0.1
0.9
8.5
16.6
19.5
20.4

0.2
0.5
0.5
0.9
1.2
1.4

Model
Whisper tiny
Whisper base
Whisper small
Whisper medium
Whisper large
Whisper large-v2

Model

Table 14. BLEU scores on Fleurs26

Robust Speech Recognition via Large-Scale Weak Supervision

D.3.2. C OVOST 2

ArabicCatalan

Welsh

GermanSpanishEstonian

PersianFrenchIndonesian

ItalianJapanese

LatvianMongolian

Model
Whisper tiny
Whisper base
Whisper small
Whisper medium
Whisper large
Whisper large-v2

0.2
1.2
17.7
30.6
35.5
39.7

4.9
11.0
22.3
29.2
30.3
31.8

0.4
0.5
1.0
12.1
16.1
21.5

4.0
11.7
25.3
33.2
34.3
36.3

10.5
21.3
33.0
38.4
38.0
40.1

0.2
0.3
2.4
11.4
13.4
15.0

0.1
0.1
4.9
15.5
17.5
19.3

6.1
15.4
27.3
33.6
34.4
36.4

0.3
4.9
27.6
42.3
45.4
48.1

5.1
13.0
24.0
29.5
29.1
30.9

0.3
4.9
17.3
24.6
24.2
26.1

0.1
0.5
1.4
9.7
10.5
13.9

0.1
0.1
0.2
0.2
0.3
0.1

Dutch

Portuguese

RussianSlovenian

SwedishTamil

TurkishChineseModel
Whisper tiny
Whisper base
Whisper small
Whisper medium
Whisper large
Whisper large-v2

4.3
12.4
28.1
38.1
39.3
41.2

9.5
23.2
40.6
48.7
48.6
51.6

5.7
16.1
30.9
39.4
41.6
43.3

0.4
1.4
9.2
17.7
23.9
21.6

2.0
10.5
29.9
39.5
40.3
42.9

0.1
0.4
1.7
2.9
3.7
4.2

0.2
2.8
16.8
27.0
26.7
28.3

0.4
1.4
6.8
14.0
17.1
18.0

Table 15. BLEU scores on CoVoST2

D.4. Long-form Transcription
TED-LIUM3

Meanwhile

Kincaid46

Rev16

Earnings-21

Earnings-22

CORAAL

Whisper tiny.en
Whisper tiny
Whisper base.en
Whisper base
Whisper small.en
Whisper small
Whisper medium.en
Whisper medium
Whisper large
Whisper large-v2

5.5
6.8
4.6
4.8
4.6
4.2
3.6
3.8
3.8
3.5

12.8
15.5
9.4
12.2
6.0
6.9
5.2
5.4
5.3
5.1

13.8
16.7
11.2
12.2
9.4
10.1
8.9
8.6
8.8
8.8

15.1
17.0
13.2
14.5
12.0
12.1
11.9
11.4
11.0
11.3

17.0
18.7
12.5
13.5
10.8
11.1
10.2
10.3
10.3
9.7

22.0
24.4
16.6
18.4
14.0
14.3
13.3
13.2
13.4
12.6

30.3
33.1
25.2
26.9
21.9
22.3
20.6
20.3
20.4
19.6

wav2vec2-base-100h
wav2vec2-base-960h
wav2vec2-large-960h-lv60-self
wav2vec2-large-960h
wav2vec2-large-robust-ft-libri-960h
hubert-large-ls960-ft
hubert-xlarge-ls960-ft
stt en conformer ctc large
stt en conformer transducer xlarge

17.6
12.8
7.2
10.1
8.8
8.1
8.1
4.0
5.3

27.7
19.7
11.4
16.4
15.2
12.9
12.5
9.8
10.6

39.3
32.9
21.1
27.4
22.9
22.4
22.9
13.1
17.1

35.2
29.8
21.3
26.4
23.4
23.4
23.2
14.5
19.8

45.7
37.3
21.7
30.4
23.0
23.0
23.1
12.6
16.2

57.1
46.8
28.0
40.1
31.0
30.6
31.3
17.6
19.7

55.4
49.1
36.7
43.5
36.8
37.9
38.1
25.1
38.9

Model

Table 16. Long-form English transcription WER (%)

27

Robust Speech Recognition via Large-Scale Weak Supervision

E. Training Dataset Statistics
Multilingual Speech Recognition

23446
Chinese13344
German
11100
Spanish9761
Russian9752
French8573
Portuguese
7993
Korean7054
Japanese4333
Turkish4278
Polish2585
Italian2119
Swedish
2077
Dutch
1883
Catalan1066
Finnish1014
Indonesian739
Arabic697
Ukrainian691
Vietnamese
688
Hebrew
529
Greek
473
Danish
382
Malay
379
Hungarian356
Romanian
266
Norwegian
226
Thai
192
Czech
136
Tamil
104
Urdu
91
Croatian
90
Slovak86
Bulgarian
75
Tagalog73
Welsh
67
Lithuanian65
Latvian47
Azerbaijani
41
Estonian
41
Slovenian28
Serbian
24
Persian21
Basque16
Icelandic
16
Macedonian
13
Armenian
12
Kazakh12
Hindi
11
Bosnian
8.9
Galician
5.7
Albanian
5.4
Sinhala5.4
Swahili
4.3
Telugu4.1
Afrikaans
3.8
Kannada2.4
Belarusian
1.3
Khmer
1.3
Bengali
1.1
Maltese
1.0
Haitian Creole
0.8
Punjabi
0.6
Marathi
0.6
Nepali0.6
Georgian
0.5
Malayalam
0.4
Yiddish
0.3
Uzbek
0.3
Gujarati
0.3
Tajik
0.2
Malagasy
0.1 Burmese
0.1Sundanese
0.1
Lao

10K

1K

100
10
1
Hours of audio

Dataset Components

17% Multilingual Speech Recognition(117,113 hours)

18% Translation(125,739 hours)

65% English Speech Recognition(438,218 hours)

0.1

Translation

Korean19938
Chinese11731
Japanese
8860
Welsh
8263
Russian
7687
Spanish
6693
Hindi
5438
French4481
German4309
Portuguese
3620
Arabic2286
Turkish
2241
Polish
2200
Italian
2145
Urdu
1990
Bengali
1988
Norwegian Nynorsk
1889
Dutch
1767
Vietnamese
1719
Malay
1691
Thai
1635
Latin
1614
Tamil
1484
Maori
1381
Indonesian1174
Swedish1055
Telugu
987
Greek
968
Tagalog
894
Malayalam892
Finnish
750
Khmer
672
Javanese622
Romanian
555
Hungarian554
Ukrainian509
Yoruba432
Hebrew
418
Czech
401
Slovenian395
Persian
392
Danish386
Galician
368
Hawaiian
338
Afrikaans
330
Norwegian322
Marathi
288
Swahili
282
Shona
279
Breton269
Croatian
239
Catalan236
Bosnian219
Sinhala211
Gujarati
208
Bulgarian202
Sanskrit
195
Tibetan186
Basque168
Slovak144
Serbian136
Assamese
136
Nepali
133
Belarusian
133
Punjabi
117
Armenian
116
Lithuanian
99
Kannada90
Azerbaijani
86
Yiddish85
Icelandic
84
Mongolian79
Estonian
79
Haitian Creole74
Albanian
72
Latvian68
Pashto63
Burmese59
Occitan49
Faroese46
Sindhi46
Maltese41
Georgian
40
Amharic
32
Kazakh31
Macedonian
30
Somali21
Lao20
Lingala
20
Tajik
15
Tatar
14
Luxembourgish
10
Hausa
8
Sundanese
7
Uzbek
4
Malagasy 2
Bashkir 1
Turkmen 1

1

Figure 11. Training dataset statistics

10

100
Hours of audio

1K

10K

28

Robust Speech Recognition via Large-Scale Weak Supervision

F. Hyperparameters
Hyperparameter

Value

Updates
Batch Size
Warmup Updates
Max grad norm
Optimizer
β1β2
ϵWeight Decay
Weight Init
Learning Rate Schedule
Speechless audio subsample factor
Condition on prior text rate

1048576
256
2048
1.0
AdamW
0.9
0.98
10−6
0.1
Gaussian Fan-In
Linear Decay
10×50%

Table 17. Whisper training hyperparameters.

Hyperparameter

Value

Updates
Batch SizeBPE Dropout
Stochastic Depth
SpecAugment Policy

6553601024
0.1
0.1
LibriSpeech Basic

Table 18. Hyperparameters changed for Whisper Large V2.

Model

Max Learning Rate

Tiny
Base
Small
MediumLarge
Large V2

1.5 × 10−3
1 × 10−3
5 × 10−4
2.5 × 10−4
1.75 × 10−4
2.0 × 10−4

Table 19. Whisper model learning rates.