README.md

---
language: 
- en
- zh
- de
- es
- ru
- ko
- fr
- ja
- pt
- tr
- pl
- ca
- nl
- ar
- sv
- it
- id
- hi
- fi
- vi
- iw
- uk
- el
- ms
- cs
- ro
- da
- hu
- ta
- no
- th
- ur
- hr
- bg
- lt
- la
- mi
- ml
- cy
- sk
- te
- fa
- lv
- bn
- sr
- az
- sl
- kn
- et
- mk
- br
- eu
- is
- hy
- ne
- mn
- bs
- kk
- sq
- sw
- gl
- mr
- pa
- si
- km
- sn
- yo
- so
- af
- oc
- ka
- be
- tg
- sd
- gu
- am
- yi
- lo
- uz
- fo
- ht
- ps
- tk
- nn
- mt
- sa
- lb
- my
- bo
- tl
- mg
- as
- tt
- haw
- ln
- ha
- ba
- jw
- su
tags:
- audio
- automatic-speech-recognition
pipeline_tag: automatic-speech-recognition
license: apache-2.0
---

# Whisper 

[OpenAI's Whisper](https://openai.com/blog/whisper/)

The Whisper model was proposed in [Robust Speech Recognition via Large-Scale Weak Supervision](https://cdn.openai.com/papers/whisper.pdf) by Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, Ilya Sutskever.

**Disclaimer**: The team releasing Whisper wrote an official model card, which is available [here](https://github.com/openai/whisper/blob/main/model-card.md). 
Content from **this** model card has been written by the Hugging Face team.


## Intro

The first paragraphs of the abstract read as follows : 

> We study the capabilities of speech processing systems trained simply to predict large amounts of transcripts of audio on the internet. When scaled to 680,000 hours of multilingual and multitask supervision, the resulting models generalize well to standard benchmarks and are often competitive with prior fully supervised results but in a zeroshot transfer setting without the need for any finetuning. 
> When compared to humans, the models approach their accuracy and robustness. We are releasing models and inference code to serve as a foundation for further work on robust speech processing.

The original code repository can be found [here](https://github.com/openai/whisper).

## Model description 

Whisper is an auto-regressive automatic speech recognition encoder-decoder model that was trained on 680 000 hours of 16kHz sampled multilingual audio. It was fully trained in a supervised manner, with multiple tasks : 

- English transcription 
- Any-to-English speech translation
- Non-English transcription
- No speech prediction 

To each task corresponds a sequence of tokens that are given to the decoder as *context tokens*. The beginning of a transcription always starts with `<|startoftranscript|>` which is why the `decoder_start_token` is always set to `tokenizer.encode("<|startoftranscript|>")`. The following token should be the language token, which is automatically detected in the original code. Finally, the task is define using either `<|transcribe|>` or `<|translate|>`. In addition, a `<|notimestamps|>` token is added if the task does not include timestamp prediction.


# Usage

To transcribe or translate audio files, the model has to be used along a `WhisperFeatureExtractor`.


## Transcription 
In the following example, the english only model is used. We set the `decoder_input_ids` accordingly.


### English to english 
The "<|en|>" token is used to specify that the speech is in english and should be transcribed to english 

```python
>>> from transformers import WhisperProcessor, WhisperForConditionalGeneration
>>> from datasets import load_dataset
>>> import torch

>>> # load model and processor
>>> processor = WhisperProcessor.from_pretrained("openai/whisper-large")
>>> model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-large")

>>> decoder_input_ids = processor.tokenizer.encode("<|startoftranscript|><|en|><|transcribe|><|notimestamps|>", return_tensors="pt")

>>> # load dummy dataset and read soundfiles
>>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")

>>> # tokenize
>>> input_features = processor(ds[0]["audio"]["array"], return_tensors="pt").input_features 
>>> # retrieve logits
>>> logits = model(input_features, decoder_input_ids).logits
>>> # take argmax and decode
>>> predicted_ids = torch.argmax(logits, dim=-1)
>>> transcription = processor.batch_decode(predicted_ids)
['<|en|><|transcribe|><|notimestamps|> Mr']
```

### French to French 
In order to obtain the full transcription, the `generate()` function is used. The following example demonstrates a french to french 
transcription. 

```python
>>> from transformers import WhisperProcessor, WhisperForConditionalGeneration
>>> from datasets import load_dataset
>>> import torch

>>> # load model and processor
>>> processor = WhisperProcessor.from_pretrained("openai/whisper-large")
>>> model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-large")

>>> decoder_input_ids = processor.tokenizer.encode("<|startoftranscript|><|fr|><|transcribe|><|notimestamps|>", return_tensors="pt")

>>> # load dummy dataset and read soundfiles
>>> ds = load_dataset("common_voice", "fr", split="test", streaming=True)
>>> ds = ds.cast_column("audio", datasets.Audio(sampling_rate=16_000))
>>> input_speech = next(iter(ds))["audio"]["array"]
>>> # tokenize
>>> input_features = processor(input_speech, return_tensors="pt").input_features 
>>> predicted_ids = model.generate(input_features, decoder_input_ids = decoder_input_ids, max_lenght = 460_000)
>>> transcription = processor.batch_decode(predicted_ids)
['<|startoftranscript|><|fr|><|transcribe|><|notimestamps|> Un vrai travail intéressant va enfin être mené sur ce sujet.<|endoftext|>']

>>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens = True)
[' Un vrai travail intéressant va enfin être mené sur ce sujet.']
```

## Translation 
The "<|translate|>" is used as the first decoder input token to specify the transcription task. 

### French to English

```python
>>> from transformers import WhisperProcessor, WhisperForConditionalGeneration
>>> from datasets import load_dataset
>>> import torch

>>> # load model and processor
>>> processor = WhisperProcessor.from_pretrained("openai/whisper-large")
>>> model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-large")

>>> decoder_input_ids = processor.tokenizer.encode("<|startoftranscript|><|fr|><|translate|><|notimestamps|>", return_tensors="pt")

>>> # load dummy dataset and read soundfiles
>>> ds = load_dataset("common_voice", "fr", split="test", streaming=True)
>>> ds = ds.cast_column("audio", datasets.Audio(sampling_rate=16_000))
>>> input_speech = next(iter(ds))["audio"]["array"]
>>> # tokenize
>>> input_features = processor(input_speech, return_tensors="pt").input_features 
>>> predicted_ids = model.generate(input_features, decoder_input_ids = decoder_input_ids, max_lenght = 460_000)
>>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens = True)
[' A real interesting work will be done on this subject.']
```



## Evaluation

This code snippet shows how to evaluate **openai/whisper-large** on LibriSpeech's "clean" and "other" test data.
 
```python
>>> from datasets import load_dataset
>>> from transformers import WhisperForConditionalGeneration, WhisperProcessor
>>> import soundfile as sf
>>> import torch
>>> from jiwer import wer


>>> librispeech_eval = load_dataset("librispeech_asr", "clean", split="test")

>>> model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-large").to("cuda")
>>> processor = WhisperProcessor.from_pretrained("openai/whisper-large")

>>> def map_to_pred(batch):
>>>     input_features = processor(batch["audio"]["array"], return_tensors="pt").input_features

>>>     with torch.no_grad():
>>>         logits = model(input_features.to("cuda")).logits

>>>     predicted_ids = torch.argmax(logits, dim=-1)
>>>     transcription = processor.batch_decode(predicted_ids, normalize = True)
>>>     batch['text'] = processor.tokenizer._normalize(batch['text'])
>>>     batch["transcription"] = transcription
>>>     return batch

>>> result = librispeech_eval.map(map_to_pred, batched=True, batch_size=1, remove_columns=["speech"])

>>> print("WER:", wer(result["text"], result["transcription"]))
0.030003583080317572
```



## Training Data

Out of the 680 000 hours of training data, 65% are English language audio and their corresponding transcript. 18% of the remaining data is non-English audio and their corresponding translated transcript in english, while the remaining 17% are non english audio with their non english transcript. This non-English data represents 98 different languages. 

As discussed in [the accompanying paper](https://cdn.openai.com/papers/whisper.pdf), we see that performance on transcription in a given language is directly correlated with the amount of training data we employ in that language.

## Performance and Limitations

Whisper shows strong robustness to accents, background noise, technical language, as well as zero shot translation from multiple languages into English; and that accuracy on speech recognition and translation is near the state-of-the-art level. The results are all highly correlated with the available training data in the studied language. 

A common model hallucination (a predicted text that is not present in the audio) includes notably classic ending of youtube video : "Thank you for listening. Please subscribe to the channel". 


The authors also mention that :

> The sequence-to-sequence architecture of the model makes it prone to generating repetitive texts, which can be mitigated to some degree by beam search and temperature scheduling but not perfectly. Further analysis on these limitations are provided in [the paper](https://cdn.openai.com/papers/whisper.pdf). It is likely that this behavior and hallucinations may be worse on lower-resource and/or lower-discoverability languages.



### BibTeX entry and citation info

```bibtex
@misc{zhang2022opt,
      title={Robust Speech Recognition via Large-Scale Weak Supervision.}, 
      author={Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, Ilya Sutskever},
      year={2022},
      url={https://cdn.openai.com/papers/whisper.pdf},
}
```