Training in progress, step 1000

.gitignore

@@ -0,0 +1 @@
+checkpoint-*/

README.md

@@ -0,0 +1,1244 @@
+# Whisper Fine-Tuning Event 🤗
+
+Welcome to the Whisper fine-tuning event 🎙️!
+
+For two weeks, we will endeavour to fine-tune the Whisper model to build state-of-the-art speech recognition systems in 
+the languages of our choice 🗣. We will work together as a community to achieve this, helping others and learning where 
+we can 🤗. If necessary and available, free access to A100 40 GB GPUs will kindly be provided by our cloud compute 
+partners, [Lambda](https://lambdalabs.com) 🚀.
+
+This document summarises all the relevant information required for the event 📋. Please read it thoroughly 
+and make sure to:
+- Sign-up using the [Google form](https://forms.gle/F2bpouvhDpKKisM39)
+- Join the [Hugging Face Discord server](https://hf.co/join/discord) and make sure to assign yourself **@ml-4-audio** role in #role-assignment so that you can access #events channel. 
+
+## Table of Contents
+
+- [Introduction](#introduction)
+- [Important Dates](#important-dates)
+- [Launch a Lambda Cloud GPU](#launch-a-lambda-cloud-gpu)
+- [Set Up an Environment](#set-up-an-environment)
+- [Data and Pre-Processing](#data-and-pre-processing)
+- [Fine-Tune a Whisper Model](#fine-tune-whisper)
+- [Evaluation](#evaluation)
+- [Building a Demo](#building-a-demo)
+- [Communication and Problems](#communication-and-problems)
+- [Talks](#talks)
+- [Tips and Tricks](#tips-and-tricks)
+- [Feedback](#feedback)
+
+## Introduction
+Whisper is a pre-trained model for automatic speech recognition (ASR) published in [September 2022](https://openai.com/blog/whisper/) 
+by the authors Radford et al. from OpenAI. Pre-trained on 680,000 hours of labelled data, it demonstrates a strong ability 
+to generalise to different datasets and domains. Through fine-tuning, the performance of this model can be significantly 
+boosted for a given language.
+
+In this event, we're bringing the community together to fine-tune Whisper in as many languages as possible. Our aim is 
+to achieve state-of-the-art on the languages spoken by the community. Together, we can democratise speech recognition 
+for all.
+
+We are providing training scripts, notebooks, blog posts, talks and compute (where available), so you have all the 
+resources you need to participate! You are free to chose your level of participation, from using the template script and setting 
+it to your language, right the way through to exploring advanced training methods. We encourage you to participate to 
+level that suits you best. We'll be on hand to facilitate this!
+
+Participants are allowed to fine-tune their systems on the training data of their choice, including datasets from the 
+Hugging Face Hub, web-scraped data from the internet, or private datasets. Whisper models will be evaluated 
+on the "test" split of the [Common Voice 11](https://huggingface.co/datasets/mozilla-foundation/common_voice_11_0) 
+dataset for the participant's chosen language.
+
+We believe that framing the event as a competition is fun! But at the core, the event is about
+fine-tuning Whisper in as many languages as possible as a community. We want to foster an environment where we 
+work together, help each other solve bugs, share important findings and ultimately learn something new.
+
+This README contains all the information you need for the event. It is structured such that you can read it sequentially, 
+section-by-section. **We recommend that you read the document once from start to finish before running any code.** This will 
+give you an idea of where to look for the relevant information and an idea of how the event is going to run.
+
+## Important Dates
+
+- *Introduction Talk*: 2nd December 2022
+- *Sprint start*: 5th December 2022
+- *Speaker Events*: 5th December 2022
+- *Sprint end*: 19th December 2022
+- *Results*: 23rd December 2022
+
+## Launch a Lambda Cloud GPU
+Where possible, we encourage you to fine-tune Whisper on a local GPU machine. This will mean a faster set-up and more 
+familiarity with your device. If you are running on a local GPU machine, you can skip ahead to the next section: [Set Up an Environment](#set-up-an-environment). 
+
+The training scripts can also be run as a notebook through Google Colab. We recommend you train on Google Colab if you 
+have a "Colab Pro" or "Pro+" subscription. This is to ensure that you receive a sufficiently powerful GPU on your Colab for 
+fine-tuning Whisper. If you wish to fine-tune Whisper through Google Colab, you can skip ahead to the section: [Data and Pre-Processing](#data-and-pre-processing). 
+
+If you do not have access to a local GPU or Colab Pro/Pro+, we'll endeavour to provide you with a cloud GPU instance.
+We've partnered up with Lambda to provide cloud compute for this event. They'll be providing the latest NVIDIA A100 
+40 GB GPUs, so you'll be loaded with some serious firepower! The Lambda API makes it easy to spin-up and launch 
+a GPU instance. In this section, we'll go through the steps for spinning up an instance one-by-one.
+
+<p align="center" width="100%">
+    <img width="50%" src="https://raw.githubusercontent.com/sanchit-gandhi/codesnippets/main/hf_x_lambda.jpg">
+</p>
+
+This section is split into three parts:
+
+1. [Signing-Up with Lambda](#signing-up-with-lambda)
+2. [Creating a Cloud Instance](#creating-a-cloud-instance)
+3. [Deleting a Cloud Instance](#deleting-a-cloud-instance)
+
+### Signing-Up with Lambda
+
+1. Create an account with Lambda using your email address of choice: https://cloud.lambdalabs.com/sign-up. If you already have an account, skip to step 2.
+2. Using this same email address, email `cloud@lambdal.com` with the Subject line: `Lambda cloud account for HuggingFace Whisper event - payment authentication and credit request`.
+3. Each user who emails as above will receive $110 in credits (amounting to 100 hours of 1x A100 usage).
+4. Register a valid payment method with Lambda in order to redeem the credits (see instructions below).
+
+100 hours of 1x A100 usage should enable you to complete 5-10 fine-tuning runs. To redeem these credits, you will need to 
+authorise a valid payment method with Lambda. Provided that you remain within $110 of compute spending, your card **will not** 
+be charged 💸. Registering your card with Lambda is a mandatory sign-up step that we unfortunately cannot bypass. But we 
+reiterate: you will not be charged provided you remain within $110 of compute spending!
+
+Follow steps 1-4 in the next section [Creating a Cloud Instance](#creating-a-cloud-instance) to register your
+card. If you experience issues with registering your card, contact the Lambda team on Discord (see [Communications and Problems](#communication-and-problems)).
+
+In order to maximise the free GPU hours you have available for training, we advise that you shut down GPUs when you are 
+not using them and closely monitor your GPU usage. We've detailed the steps you can follow to achieve this in [Deleting a Cloud Instance](#deleting-a-cloud-instance).
+
+### Creating a Cloud Instance
+Estimated time to complete: 5 mins
+
+*You can also follow our video tutorial to set up a cloud instance on Lambda* 👉️ [YouTube Video](https://www.youtube.com/watch?v=Ndm9CROuk5g&list=PLo2EIpI_JMQtncHQHdHq2cinRVk_VZdGW)
+
+1. Click the link: https://cloud.lambdalabs.com/instances
+2. You'll be asked to sign in to your Lambda account (if you haven't done so already).
+3. Once on the GPU instance page, click the purple button "Launch instance" in the top right.
+4. Verify a payment method if you haven't done so already. IMPORTANT: if you have followed the instructions in the previous section, you will have received $110 in GPU credits. Exceeding 100 hours of 1x A100 usage may incur charges on your credit card. Contact the Lambda team on Discord if you have issues authenticating your payment method (see [Communications and Problems](#communication-and-problems))
+5. Launching an instance:
+   1. In "Instance type", select the instance type "1x A100 (40 GB SXM4)"
+   2. In "Select region", select the region with availability closest to you.
+   3. In "Select filesystem", select "Don't attach a filesystem".
+6. You will be asked to provide your public SSH key. This will allow you to SSH into the GPU device from your local machine.
+   1. If you’ve not already created an SSH key pair, you can do so with the following command from your local device: 
+      ```bash
+      ssh-keygen
+      ```
+   2. You can find your public SSH key using the command: 
+      ```bash
+      cat ~/.ssh/id_rsa.pub
+      ```
+      (Windows: `type C:UsersUSERNAME.sshid_rsa.pub` where `USERNAME` is the name of your user)
+   4. Copy and paste the output of this command into the first text box
+   5. Give your SSH key a memorable name (e.g. `sanchits-mbp`)
+   6. Click "Add SSH Key"
+7. Select the SSH key from the drop-down menu and click "Launch instance"
+8. Read the terms of use and agree
+9. We can now see on the "GPU instances" page that our device is booting up!
+10. Once the device status changes to "✅ Running", click on the SSH login ("ssh ubuntu@..."). This will copy the SSH login to your clipboard.
+11. Now open a new command line window, paste the SSH login, and hit Enter.
+12. If asked "Are you sure you want to continue connecting?", type "yes" and press Enter.
+13. Great! You're now SSH'd into your A100 device! We're now ready to set up our Python environment!
+
+You can see your total GPU usage from the Lambda cloud interface: https://cloud.lambdalabs.com/usage
+
+Here, you can see the total charges that you have incurred since the start of the event. We advise that you check your 
+total on a daily basis to make sure that it remains below the credit allocation of $110. This ensures that you are 
+not inadvertently charged for GPU hours.
+
+If you are unable to SSH into your Lambda GPU in step 11, there is a workaround that you can try. On the [GPU instances page](https://cloud.lambdalabs.com/instances), 
+under the column "Cloud IDE", click the button "Launch". This will launch a Jupyter Lab on your GPU which will be displayed in your browser. In the 
+top left-hand corner, click "File" -> "New" -> "Terminal". This will open up a new terminal window. You can use this 
+terminal window to set up your Python environment in the next section [Set Up an Environment](#set-up-an-environment).
+
+### Deleting a Cloud Instance
+
+100 1x A100 hours should provide you with enough time for 5-10 fine-tuning runs (depending on how long you train for 
+and which size models). To maximise the GPU time you have for training, we advise that you shut down GPUs over prolonged 
+periods of time when they are not in use. Leaving a GPU running accidentally over the weekend will incur 48 hours of 
+wasted GPU hours. That's nearly half of your compute allocation! So be smart and shut down your GPU when you're not training.
+
+Creating an instance and setting it up for the first time may take up to 20 minutes. Subsequently, this process will 
+be much faster as you gain familiarity with the steps, so you shouldn't worry about having to delete a GPU and spinning one 
+up the next time you need one. You can expect to spin-up and delete 2-3 GPUs over the course of the fine-tuning event.
+
+
+We'll quickly run through the steps for deleting a Lambda GPU. You can come back to these steps after you've 
+performed your first training run and you want to shut down the GPU:
+
+1. Go to the instances page: https://cloud.lambdalabs.com/instances
+2. Click the checkbox on the left next to the GPU device you want to delete
+3. Click the button "Terminate" in the top right-hand side of your screen (under the purple button "Launch instance")
+4. Type "erase data on instance" in the text box and press "ok"
+
+Your GPU device is now deleted and will stop consuming GPU credits.
+
+## Set Up an Environment
+Estimated time to complete: 5 mins
+
+*Follow along our video tutorial detailing the set up* 👉️ [YouTube Video](https://www.youtube.com/playlist?list=PLo2EIpI_JMQtzC5feNpqQL7eToYKcOxYf)
+
+The Whisper model should be fine-tuned using **PyTorch**, **🤗 Transformers**, and, **🤗 Datasets**. In this 
+section, we'll cover how to set up an environment with the required libraries. This section assumes that you are SSH'd 
+into your GPU device. This section does not apply if you are fine-tuning the Whisper model in a Google Colab.
+
+If you are returning to this section having read through it previously and want to quickly set up an environment, you 
+can do so in one call by executing the following code cell. If this is your first time setting up an environment, we 
+recommend you read this section to understand the steps involved.
+
+```bash
+sudo add-apt-repository -y ppa:jonathonf/ffmpeg-4
+sudo apt update
+sudo apt install -y ffmpeg
+
+sudo apt-get install git-lfs
+
+python3 -m venv hf_env
+source hf_env/bin/activate
+echo "source ~/hf_env/bin/activate" >> ~/.bashrc
+
+git clone https://github.com/huggingface/community-events.git
+pip install -r community-events/whisper-fine-tuning-event/requirements.txt
+
+git config --global credential.helper store
+huggingface-cli login
+ ```
+
+### Unix Libraries
+
+First, we need to make sure we have the required NVIDIA drivers installed. We can check that we have these drivers 
+through the following command:
+
+```bash
+nvidia-smi
+```
+
+This should print a table with our NVIDIA driver version and CUDA version, and should work out of the box for Lambda GPUs!
+If you get an error running this command, refer to your device manual for installing the required NVIDIA driver.
+
+Before installing the required libraries, we'd need to install and update the Unix package `ffmpeg` to version 4:
+
+ ```bash
+sudo add-apt-repository -y ppa:jonathonf/ffmpeg-4
+sudo apt update
+sudo apt install -y ffmpeg
+ ```
+
+We'll also need the package `git-lfs` to push large model weights to the Hugging Face Hub. To check whether 
+`git-lfs` is installed, simply run:
+
+```bash
+git-lfs -v
+```
+
+The output should show something like `git-lfs/2.13.2 (GitHub; linux amd64; go 1.15.4)`. If your console states that 
+the `git-lfs` command was not found, you can install it via: 
+
+```bash
+sudo apt-get install git-lfs
+```
+
+### Python Libraries
+
+We recommend installing the required libraries in a Python virtual environment. If you're unfamiliar with Python virtual 
+environments, check out the [official user guide](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/).
+
+Let's define a variable that denotes the name of the environment we're going to create:
+
+```bash
+env_name=<your-venv-name>
+```
+
+We can create a virtual environment (venv) with this name using the following command:
+
+```bash
+python3 -m venv $env_name
+```
+
+We'll instruct our bash shell to activate the venv by default by placing the venv source command in `.bashrc`:
+
+```bash
+echo "source ~/$env_name/bin/activate" >> ~/.bashrc
+```
+
+Re-launching the bash shell will activate the venv:
+
+```bash
+bash
+```
+
+Great! We can see that our venv name is at the start of our command line - this means that we're operating from 
+within the venv. We can now go ahead and start installing the required Python packages to our venv.
+
+The [`requirements.txt`](https://github.com/huggingface/community-events/blob/main/whisper-fine-tuning-event/requirements.txt) 
+file in this directory has all the necessary Python packages we need to fine-tune Whisper, including PyTorch, Transformers 
+and Datasets. We'll install all the packages in this file through one `pip install` command.
+
+First, let's clone the `community-events` repository to our device:
+
+```bash
+git clone https://github.com/huggingface/community-events.git
+```
+
+Now we can install the packages from the `requirements.txt` file using the following command:
+
+```bash
+pip install -r community-events/whisper-fine-tuning-event/requirements.txt
+```
+
+Note: when installing packages, you might see warnings such as:
+
+```bash
+  error: invalid command 'bdist_wheel'
+  ----------------------------------------
+  ERROR: Failed building wheel for audioread
+```
+
+This is perfectly ok! It does not affect our installation.
+
+We can check that above steps installed the correct version of PyTorch to match our CUDA version. The following command should return True:
+
+```python
+python -c "import torch; print(torch.cuda.is_available())"
+```
+
+If the above command does not return True, refer to the [official instructions](https://pytorch.org/get-started/locally/) for installing PyTorch.
+
+We can now verify that `transformers` and `datasets` have been correctly installed. First, launch a Python shell:
+
+```bash
+python
+```
+
+Running the following code cell will load one sample of the [Common Voice](https://huggingface.co/datasets/common_voice) 
+dataset from the Hugging Face Hub and perform a forward pass of the "tiny" Whisper model:
+
+```python
+import torch
+from transformers import WhisperFeatureExtractor, WhisperForConditionalGeneration
+from datasets import load_dataset
+
+model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny")
+feature_extractor = WhisperFeatureExtractor.from_pretrained("openai/whisper-tiny")
+
+common_voice = load_dataset("common_voice", "en", split="validation", streaming=True)
+
+inputs = feature_extractor(next(iter(common_voice))["audio"]["array"], sampling_rate=16000, return_tensors="pt")
+input_features = inputs.input_features
+
+decoder_input_ids = torch.tensor([[1, 1]]) * model.config.decoder_start_token_id
+logits = model(input_features, decoder_input_ids=decoder_input_ids).logits
+
+print("Environment set up successful?", logits.shape[-1] == 51865)
+
+```
+
+If the final check returns True, the libraries have been installed correctly. Finally, exit the Python shell:
+
+```python
+quit()
+```
+
+The last thing we need to do is link our Hugging Face account. Run the command:
+
+```bash
+git config --global credential.helper store
+huggingface-cli login
+```
+
+And then enter an authentication token from https://huggingface.co/settings/tokens. Create a new token if you do not have 
+one already. You should make sure that this token has "write" privileges.
+
+## Data and Pre-Processing
+
+In this section, we will cover how to find suitable training data and the necessary steps to pre-process it. 
+If you are new to the 🤗 Datasets library, we highly recommend reading the comprehensive blog post: [A Complete Guide To Audio Datasets](https://huggingface.co/blog/audio-datasets). 
+This blog post will tell you everything you need to know about 🤗 Datasets and its one-line API.
+
+### Data
+
+Whisper models will be evaluated on the `"test"` split of the [Common Voice 11](https://huggingface.co/datasets/mozilla-foundation/common_voice_11_0) 
+dataset. Any data can be used to fine-tune the Whisper model **except Common Voice's `"test"` split**. This exception 
+extends to all Common Voice versions, as the test split of legacy Common Voice releases often overlaps with the 
+latest one. For instance, the test split of Common Voice 10 is largely the same as that of Common Voice 11.
+
+So, the test data:
+
+```python
+load_dataset("mozilla-foundation/common_voice_11_0", "en", split="test", use_auth_token=True)
+```
+
+More or less includes the same data as:
+
+```python
+load_dataset("mozilla-foundation/common_voice_10_0", "en", split="test", use_auth_token=True)
+```
+
+And **neither** are allowed for training purposes. However, we strongly encourage participants to make use of the other 
+Common Voice splits for training data, such as the `"train"` and `"validation"` splits:
+
+```python
+load_dataset("mozilla-foundation/common_voice_10_0", "en", split="train", use_auth_token=True)
+```
+
+For most languages, the `"train"` split of Common Voice 11 dataset offers a reasonable amount of training data. 
+For low-resource languages, it is normal procedure to combine the `"train"` and `"validation"` splits to give a larger 
+training corpus:
+
+```python
+load_dataset("mozilla-foundation/common_voice_10_0", "en", split="train+validation", use_auth_token=True)
+```
+
+This notation for combining splits (`"split_a+split_b"`) is consistent for all resources in the event. You can combine 
+splits in this same way using the fine-tuning scripts in the following section [Fine-Tune Whisper](#fine-tune-whisper).
+
+If combining the `"train"` and `"validation"` splits of the Common Voice 11 dataset still gives insufficient training 
+data for your language, you can explore using other datasets on the Hub to train your model and try 
+[Mixing Datasets](#mixing-datasets-optional) to give larger training splits.
+
+### Streaming Mode
+
+Audio datasets are very large. This causes two issues:
+1. They require a significant amount of **storage space** to download.
+2. They take a significant amount of **time** to download and process.
+
+The storage and time requirements present limitations to most speech researchers. For example, downloading the English 
+subset of the Common Voice 11 dataset (2,300 hours) requires upwards of 200GB of disk space and up to several hours 
+of download time. For these reasons, we **do not** recommend that you run the following code cell! 
+```python
+from datasets import load_dataset
+
+common_voice = load_dataset("mozilla-foundation/common_voice_11_0", "en", use_auth_token=True)
+
+# we have to wait several hours until the entire dataset is downloaded before we can access the first sample... 
+print(next(iter(common_voice["train"])))
+```
+
+However, both these issues can be solved with 🤗 Datasets. Rather than downloading the whole dataset at once, we 
+load individual samples as we cycle over the dataset, in a process called _streaming_. Since the data is loaded 
+progressively as we iterate over the dataset, we can get started with a dataset as soon as the first sample is ready. 
+This way, we don't have to wait for the entire dataset to download before we can run our code! We are also free of any 
+disk space contraints: once we're done with a sample, we discard it and load the next one to memory. This way, we only 
+have the data when we need it, and not when we don't!
+
+Streaming is enabled by passing the argument `streaming=True` to the `load_dataset` function. We can then use our 
+audio datasets in much the same way as before! For these reasons, **we highly recommend** that you try out the following 
+code cell! Just make sure you've accepted the Common Voice 11 [terms of use](https://huggingface.co/datasets/mozilla-foundation/common_voice_11_0) on the Hugging Face Hub.
+
+```python
+from datasets import load_dataset
+
+common_voice = load_dataset("mozilla-foundation/common_voice_11_0", "en", use_auth_token=True, streaming=True)
+
+# get the first sample of the dataset straight away!
+print(next(iter(common_voice["train"])))
+```
+
+The examples for this event rely heavily on streaming mode to fine-tune Whisper. With streaming mode, we can use **any 
+speech recognition dataset on the Hub with just 20GB of disk space**. As a speech recognition practitioner, this is 
+game changing! The largest speech recognition datasets are available to us regardless of our device disk space. We are 
+extremely excited to be showcasing streaming mode in this event and hope that you will enjoy using it.
+
+There is one caveat to streaming mode. When downloading a dataset to disk, the processed data is saved to our cache. If 
+we want to re-use this data, we can directly load the processed data from cache, skipping the download and processing 
+steps. Consequently, we only have to perform the downloading and processing operations once. With streaming mode, the 
+data is not downloaded to disk. Thus, neither the download nor pre-processing are cached. If we want to re-use the data, 
+the streaming steps must be repeated, with the audio files loaded and pre-processed again. Therefore, we recommend not 
+using streaming mode if your dataset is small (< 10 hours). In this case, it is faster to download and pre-process the 
+dataset in the conventional way once at the start, and then re-use it at each epoch. We provide pointers for disabling 
+streaming mode in the section [Fine-Tune Whisper](#fine-tune-whisper).
+
+If you want to read more about streaming mode, we 
+recommend you check out the aforementioned blog post: [A Complete Guide To Audio Datasets](https://huggingface.co/blog/audio-datasets).
+
+### Pre-Processing
+
+Data pre-processing is a very grey area when it comes to speech recognition. In this section, we'll try to make the 
+situation as clear as possible for you as participants.
+
+The Common Voice dataset is both cased and punctuated:
+
+```python
+print(next(iter(common_voice["train"]))["sentence"])
+```
+**Print Output:**
+```
+Why does Melissandre look like she wants to consume Jon Snow on the ride up the wall?
+```
+
+If we train the Whisper model on the raw Common Voice dataset, it will learn to predict casing and punctuation. This is 
+great when we want to use out model for actual speech recognition applications, such as transcribing meetings or 
+dictation, as the predicted transcriptions will be formatted with casing and punctuation.
+
+However, we also have the option of 'normalising' the dataset to remove any casing and punctuation. Normalising the 
+dataset makes the speech recognition task easier: the model no longer needs to distinguish between upper and lower case 
+characters, or have to predict punctuation from the audio data alone. Because of this, the word error rates are 
+naturally lower (meaning the results are better). The Whisper paper demonstrates the drastic effect that normalising 
+transcriptions can have on WER results (_c.f._ Section 4.4 of the [Whisper paper](https://cdn.openai.com/papers/whisper.pdf)). 
+But while we get lower WERs, we can't necessarily use our model in production. The lack of casing and punctuation makes 
+the predicted text from the model much harder to read. We would need additional post-processing models to restore casing and 
+punctuation in our predictions if we wanted to use it for downstream applications.
+
+There is a happy medium between the two: we can train our systems on cased and normalised transcriptions, and then 
+evaluate them on normalised text. This way, we train our systems to predict fully formatted text, but also benefit from 
+the WER improvements we get by normalising the transcriptions.
+
+The choice of whether you normalise the transcriptions is ultimately down to you. We recommend training on un-normalised 
+text and evaluating on normalised text to get the best of both worlds. Since those choices are not always obvious, feel 
+free to ask on Discord or (even better) post your question on the [forum](https://discuss.huggingface.co).
+
+| Train         | Eval          | Pros                                                           | Cons                                     |
+|---------------|---------------|----------------------------------------------------------------|------------------------------------------|
+| Un-normalised | Un-normalised | * Predict casing + punctuation<br>* One logic for train / eval | * WERs are higher                        |
+| Un-normalised | Normalised    | * Predict casing + punctuation<br>* WERs are lower             | * Different logic for train / eval       |
+| Normalised    | Normalised    | * One logic for train / eval<br>* WERs are lower               | * No casing / punctuation in predictions |
+
+With the provided training scripts, it is trivial to toggle between removing or retaining punctuation and casing, 
+requiring at most three lines of code change. Switching between the different modes is explained in more detail in the 
+following section [Fine-Tune Whisper](#fine-tune-whisper).
+
+If you want to find out more about pre- and post-processing for speech recognition, we refer you in the direction of 
+the paper: [ESB: A Benchmark For Multi-Domain End-to-End Speech Recognition](https://arxiv.org/abs/2210.13352).
+
+The following two subsections are optional. They cover how you can mix datasets to form larger training splits and how 
+you can use custom data to fine-tune your model. If the Common Voice 11 dataset has sufficient data in your language to 
+fine-tune your model, you can skip to the next section [Fine-Tune Whisper](#fine-tune-whisper).
+
+### Mixing Datasets (optional)
+
+If the Common Voice 11 dataset contains insufficient training data to fine-tune Whisper in your language, you can explore mixing 
+different datasets to create a larger combined training set. Incorporating supplementary training data is almost always beneficial for training. 
+The Whisper paper demonstrates the significant effect that increasing the amount of training data can have on downstream 
+performance (_c.f._ Section 4.2 of the [paper](https://cdn.openai.com/papers/whisper.pdf)). There are a number of datasets 
+that are available on the Hugging Face Hub that can be downloaded via the 🤗 Datasets library in much the same way as 
+Common Voice 11.
+
+We recommend selecting from the following four datasets on the Hugging Face Hub for multilingual speech recognition:
+
+| Dataset                                                                                       | Languages | Casing | Punctuation |
+|-----------------------------------------------------------------------------------------------|-----------|--------|-------------|
+| [Common Voice 11](https://huggingface.co/datasets/mozilla-foundation/common_voice_11_0)       | 100+      | ✅      | ✅           |
+| [VoxPopuli](https://huggingface.co/datasets/facebook/voxpopuli)                               | 15        | ❌      | ✅           |
+| [Multilingual LibriSpeech](https://huggingface.co/datasets/facebook/multilingual_librispeech) | 6         | ❌      | ❌           |
+| [FLEURS](https://huggingface.co/datasets/google/fleurs)                                       | 100+      | ✅      | ✅           |
+
+
+<!---
+<details>
+<summary>
+
+#### Common Voice 11
+
+</summary>
+
+[Common Voice 11](https://huggingface.co/datasets/mozilla-foundation/common_voice_11_0) is a crowd-sourced 
+open-licensed speech dataset where speakers record text from Wikipedia in various languages. Since anyone can contribute 
+recordings, there is significant variation in both audio quality and speakers. The audio conditions are challenging, with 
+recording artefacts, accented speech, hesitations, and the presence of foreign words. The transcriptions are both cased 
+and punctuated. As of version 11, there are over 100 languages available, both low and high-resource.
+</details>
+<details>
+<summary>
+
+#### VoxPopuli
+
+</summary>
+
+[VoxPopuli](https://huggingface.co/datasets/facebook/voxpopuli) is a large-scale multilingual speech corpus consisting 
+of data sourced from 2009-2020 European Parliament event recordings. Consequently, it occupies the unique domain of 
+oratory, political speech, largely sourced from non-native speakers. It contains labelled audio-transcription data for 
+15 European languages. The transcriptions are punctuated but not cased.
+</details>
+<details>
+<summary>
+
+#### Multilingual LibriSpeech
+
+</summary>
+
+[Multilingual LibriSpeech](https://huggingface.co/datasets/facebook/multilingual_librispeech) is the multilingual 
+equivalent of the [LibriSpeech ASR](https://huggingface.co/datasets/librispeech_asr) corpus. It comprises a large corpus 
+of read audiobooks taken from the [LibriVox](https://librivox.org/) project, making it a suitable dataset for academic 
+research. It contains data split into eight high-resource languages - English, German, Dutch, Spanish, French, Italian, 
+Portuguese and Polish. The transcriptions are neither punctuated nor cased.
+</details>
+<details>
+<summary>
+
+#### FLEURS
+
+</summary>
+
+[FLEURS](https://huggingface.co/datasets/google/fleurs) (Few-shot Learning Evaluation of Universal Representations of 
+Speech) is a dataset for evaluating speech recognition systems in 102 languages, including many that are classified as 
+'low-resource'. The data is derived from the [FLoRes-101](https://arxiv.org/abs/2106.03193) dataset, a machine 
+translation corpus with 3001 sentence translations from English to 101 other languages. Native speakers are recorded 
+narrating the sentence transcriptions in their native language. The recorded audio data is paired with the sentence 
+transcriptions to yield a multilingual speech recognition over all 101 languages. The training sets contain 
+approximately 10 hours of supervised audio-transcription data per language. Transcriptions come in two formats: un-normalised 
+(`"raw_transcription"`) and normalised (`"transcription"`).
+</details>
+
+The previously mentioned blog post provides a more in-depth explanation of the main English speech recognition, 
+multilingual speech recognition and speech translation datasets on the Hub: [A Complete Guide To Audio Datasets](https://huggingface.co/blog/audio-datasets#a-tour-of-audio-datasets-on-the-hub)  
+
+You can also explore all speech recognition datasets on the Hub to find one suited for your language and needs: [ASR datasets on the Hub](https://huggingface.co/datasets?task_categories=task_categories:automatic-speech-recognition&sort=downloads).
+--->
+
+You can try training on these datasets individually, or mix them to form larger train sets.
+
+When mixing datasets, you should ensure the transcription format is consistent across datasets. For example, if you mix 
+Common Voice 11 (cased + punctuated) with VoxPopuli (un-cased + punctuated), you will need to lower-case **all the text** 
+for both training and evaluation, such that the transcriptions are consistent across training samples (un-cased + punctuated). 
+
+Likewise, if mixing Common Voice 11 (cased + punctuated) with Multilingual LibriSpeech (un-cased + un-punctuated), you 
+should make sure to remove all casing and punctuation in **all the text** for both training and evaluation, such that 
+all transcriptions are un-cased and un-punctuated for all training samples.
+
+Having a mismatch in formatting for different training samples can reduce the final performance of your fine-tuned Whisper 
+model.
+
+If you want to combine multiple datasets for training, you can refer to the code-snippet provided for interleaving 
+datasets with streaming mode: [interleave_streaming_datasets.ipynb](https://github.com/huggingface/community-events/blob/main/whisper-fine-tuning-event/interleave_streaming_datasets.ipynb).
+
+### Custom Data (optional)
+
+In addition to publicly available data on the Hugging Face Hub, participants can also make use of their own audio data 
+for training. When using your own audio data, please make sure that you **are allowed to use the audio data**. For 
+instance, if the audio data is taken from media platforms, such as YouTube, please verify that the media platform and 
+the owner of the data have given their approval to use the audio data in the context of machine learning research. If 
+you are not sure whether the data you want to use has the appropriate licensing, please contact the Hugging Face team 
+on Discord.
+
+<!--- TODO: VB - tutorial for adding own data via audio folder --->
+
+## Fine-Tune Whisper
+
+Throughout the event, participants are encouraged to leverage the official pre-trained [Whisper checkpoints](https://huggingface.co/models?pipeline_tag=automatic-speech-recognition&sort=downloads&search=whisper).
+The Whisper checkpoints come in five configurations of varying model sizes.
+The smallest four are trained on either English-only or multilingual data.
+The largest checkpoint is multilingual only. The checkpoints are summarised in the following table with links to the 
+models on the Hugging Face Hub:
+
+| Size   | Layers | Width | Heads | Parameters | English-only                                         | Multilingual                                      |
+|--------|--------|-------|-------|------------|------------------------------------------------------|---------------------------------------------------|
+| tiny   | 4      | 384   | 6     | 39 M       | [✓](https://huggingface.co/openai/whisper-tiny.en)   | [✓](https://huggingface.co/openai/whisper-tiny)  |
+| base   | 6      | 512   | 8     | 74 M       | [✓](https://huggingface.co/openai/whisper-base.en)   | [✓](https://huggingface.co/openai/whisper-base)   |
+| small  | 12     | 768   | 12    | 244 M      | [✓](https://huggingface.co/openai/whisper-small.en)  | [✓](https://huggingface.co/openai/whisper-small)  |
+| medium | 24     | 1024  | 16    | 769 M      | [✓](https://huggingface.co/openai/whisper-medium.en) | [✓](https://huggingface.co/openai/whisper-medium) |
+| large  | 32     | 1280  | 20    | 1550 M     | x                                                    | [✓](https://huggingface.co/openai/whisper-large)  |
+
+The English-only checkpoints should be used for English speech recognition. For all other languages, one should use the 
+multilingual checkpoints.
+
+We recommend using the tiny model for rapid prototyping. **We advise that the small or medium checkpoints are used for 
+fine-tuning**. These checkpoints achieve comparable performance to the large checkpoint, but can be trained much faster 
+(and hence for much longer!).
+
+A complete guide to Whisper fine-tuning can be found in the blog post: [Fine-Tune Whisper with 🤗 Transformers](https://huggingface.co/blog/fine-tune-whisper).
+While it is not necessary to have read this blog post before fine-tuning Whisper, it is strongly advised to gain 
+familiarity with the fine-tuning code.
+
+There are three ways in which you can execute the fine-tuning code:
+1. [Python Script](#python-script)
+2. [Jupyter Notebook](#jupyter-notebook)
+3. [Google Colab](#google-colab)
+
+1 and 2 are applicable when running on a local GPU or cloud GPU instance (such as on Lambda). 3 applies if you have 
+a Google Colab Pro/Pro+ subscription and want to run training in a Google Colab. The proceeding instructions for running 
+each of these methods are quite lengthy. Feel free to read through each of them to get a better idea for which one you 
+want to use for training. Once you've read through, we advise you pick one method and stick to it!
+
+For the walk-through, we'll assume that we're fine-tuning the Whisper model on Spanish ("es") on the Common Voice 11 
+dataset. We'll point out where you'll need to change variables to run the script for your language of choice.
+
+Before jumping into any training, make sure you've accepted the Common Voice 11 [terms of use](https://huggingface.co/datasets/mozilla-foundation/common_voice_11_0) 
+on the Hugging Face Hub.
+
+### Python Script
+*Checkout the video tutorial detailing how to fine-tune your whisper model via the CLI* 👉️ [YouTube Video](https://www.youtube.com/playlist?list=PLo2EIpI_JMQuKpnFm1ntcLKP6gq0l0f1Q)
+
+1. **Create a model repository**
+
+The steps for running training with a Python script assume that you are SSH'd into your GPU device and have set up 
+your environment according to the previous section [Set Up an Environment](#set-up-an-environment).
+
+First, we need to create a model repository on the Hugging Face Hub. This repository will contain all the required files 
+to reproduce the training run, alongside model weights, training logs and a README.md card. You can either create a model 
+repository directly on the Hugging Face Hub using the link: https://huggingface.co/new Or, via the CLI. Here, we'll show 
+how to use the CLI. 
+
+Let's pick a name for our fine-tuned Whisper model: *whisper-small-es*. We can run the following command to create a 
+repository under this name.
+
+```bash
+huggingface-cli repo create whisper-small-es
+```
+(change "es" to your language code)
+
+We can now see the model on the Hub, *e.g.* under https://huggingface.co/sanchit-gandhi/whisper-small-es
+
+Let's clone the repository so that we can place our training script and model weights inside:
+
+```bash
+git lfs install
+git clone https://huggingface.co/sanchit-gandhi/whisper-small-es
+```
+
+(be sure to change the repo address to `https://huggingface.co/<your-user-name>/<your-repo-name>`)
+
+We can then enter the repository using the `cd` command:
+
+```bash
+cd whisper-small-es
+```
+
+2. **Add training script and `run` command**
+
+We encourage participants to add all the relevant files for training directly to the model repository. This way, 
+training runs are fully reproducible.
+
+We provide a Python training script for fine-tuning Whisper with 🤗 Datasets' streaming mode: [`run_speech_recognition_seq2seq_streaming.py`](https://github.com/huggingface/community-events/blob/main/whisper-fine-tuning-event/run_speech_recognition_streaming.py)
+This script can be copied to your model repository with the following command:
+
+```bash
+cp ~/community-events/whisper-fine-tuning-event/run_speech_recognition_seq2seq_streaming.py .
+```
+
+This will download a copy of the training script to your model repository.
+
+We can then define the model, training and data arguments for fine-tuning:
+
+```bash
+echo 'python run_speech_recognition_seq2seq_streaming.py \
+	--model_name_or_path="openai/whisper-small" \
+	--dataset_name="mozilla-foundation/common_voice_11_0" \
+	--dataset_config_name="es" \
+	--language="spanish" \
+	--train_split_name="train+validation" \
+	--eval_split_name="test" \
+	--model_index_name="Whisper Small Spanish" \
+	--max_steps="5000" \
+	--output_dir="./" \
+	--per_device_train_batch_size="64" \
+	--per_device_eval_batch_size="32" \
+	--logging_steps="25" \
+	--learning_rate="1e-5" \
+	--warmup_steps="500" \
+	--evaluation_strategy="steps" \
+	--eval_steps="1000" \
+	--save_strategy="steps" \
+	--save_steps="1000" \
+	--generation_max_length="225" \
+	--length_column_name="input_length" \
+	--max_duration_in_seconds="30" \
+	--text_column_name="sentence" \
+	--freeze_feature_encoder="False" \
+	--report_to="tensorboard" \
+	--metric_for_best_model="wer" \
+	--greater_is_better="False" \
+	--load_best_model_at_end \
+	--gradient_checkpointing \
+	--fp16 \
+	--overwrite_output_dir \
+	--do_train \
+	--do_eval \
+	--predict_with_generate \
+	--do_normalize_eval \
+	--streaming \
+	--use_auth_token \
+	--push_to_hub' >> run.sh
+```
+
+Make sure to change the `--dataset_config_name` and `--language` to the correct values for your language! See also how 
+we combine the train and validation splits as `--train_split_name="train+validation"`. This is recommended for low-resource 
+languages (it probably isn't strictly necessary for Spanish, where the `"train"` split for Common Voice 11 contains 
+ample training data). We also assign a `"model_index_name"` - a pretty name that will go on the model card. If you are 
+training on a very small dataset (< 10 hours), it is advisable to disable streaming mode: `--streaming="False"`.
+
+We provide the train/eval batch sizes for the "small" checkpoint fine-tuned on a 1x A100 device. Depending on your device and checkpoint, 
+you might need to lower these values. Refer to the subsection [Recommended Training Configurations](#recommended-training-configurations) 
+for suggested batch-sizes for other devices and checkpoints.
+
+3. **Launch training 🚀**
+
+We recommend running training through a `tmux` session. This means that training won't be interrupted when you close 
+your SSH connection. To start a `tmux` session named `mysession`: 
+
+```bash
+tmux new -s mysession
+```
+(if `tmux` is not installed, you can install it through: `sudo apt-get install tmux`)
+
+Once in the `tmux` session, we can launch training:
+
+```bash
+bash run.sh
+```
+
+Training should take approximately 8 hours, with a final cross-entropy loss of **1e-4** and word error rate of **32.6%**.
+
+Since we're in a `tmux` session, we're free to close our SSH window without stopping training!
+
+If you close your SSH connection and want to rejoin the `tmux` window, you can SSH into your GPU and then connect to 
+your session with the following command:
+
+```bash
+tmux a -t mysession
+```
+
+It will be like you never left! 
+
+`tmux` guide: https://www.hamvocke.com/blog/a-quick-and-easy-guide-to-tmux/
+
+### Jupyter Notebook
+*We've detailed these steps in a video tutorial to help you get up to speed faster* 👉️ [YouTube Video](https://www.youtube.com/playlist?list=PLo2EIpI_JMQs9z-N4v8L_Jb4KF6kAkylX)
+
+1. **SSH port forwarding**
+
+The steps for running training with a Python script assume that you have set up your environment according to the 
+previous section [Set Up an Environment](#set-up-an-environment) and are **not** SSH'd into your GPU device. If you are 
+SSH'd into your GPU device, you can close this SSH window and start from your local machine.
+
+The command to SSH into our GPU looked something as follows:
+
+```bash
+ssh ubuntu@104.171.202.236
+```
+
+When running a Jupyter Notebook, we need to "forward" the SSH port from the remote port to the local one. This amounts 
+to adding `-L 8888:localhost:8888` to the end of our SSH command. We can SSH into our remote machine using this modified 
+SSH command:
+
+```bash
+ssh ubuntu@104.171.202.236 -L 8888:localhost:8888
+```
+
+Be sure to change the `ssh ubuntu@...` part to your corresponding SSH command, it's simply the `-L 8888:localhost:8888` 
+part added onto the end that is new. If you want to find out more about SSH port forwarding, we recommend you read the guide: 
+[SSH/OpenSSH/PortForwarding](https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding).
+
+2. **Create a model repository (copied from previous subsection [Python Script](#python-script))**
+
+First, we need to create a model repository on the Hugging Face Hub. This repository will contain all the required files 
+to reproduce the training run, alongside model weights, training logs and a README.md card. 
+
+You can either create a model repository directly on the Hugging Face Hub using the link: https://huggingface.co/new
+Or, via the CLI. Here, we'll show how to use the CLI. 
+
+Let's pick a name for our fine-tuned Whisper model: *whisper-small-es*. We can run the following command to create a 
+repository under this name.
+
+```bash
+huggingface-cli repo create whisper-small-es
+```
+(change "es" to your language code)
+
+We can now see the model on the Hub, *e.g.* under https://huggingface.co/sanchit-gandhi/whisper-small-es
+
+Let's clone the repository so that we can place our training script and model weights inside:
+
+```bash
+git lfs install
+git clone https://huggingface.co/sanchit-gandhi/whisper-small-es
+```
+
+(be sure to change the repo address to `https://huggingface.co/<your-user-name>/<your-repo-name>`)
+
+We can then enter the repository using the `cd` command:
+
+```bash
+cd whisper-small-es
+```
+
+3. **Add notebook**
+
+We encourage participants to add all the training notebook directly to the model repository. This way, 
+training runs are fully reproducible.
+
+We provide an iPython notebook for fine-tuning Whisper with 🤗 Datasets' streaming mode: [`fine-tune-whisper-streaming.ipynb`](https://github.com/huggingface/community-events/blob/main/whisper-fine-tuning-event/fine-tune-whisper-streaming.ipynb)
+This notebook can be copied to your model repository with the following command:
+
+```bash
+cp ~/community-events/whisper-fine-tuning-event/fine-tune-whisper-streaming.ipynb .
+```
+
+If you are fine-tuning Whisper on a very small dataset (< 10 hours), it is advised that you use the non-streaming notebook 
+[`fine-tune-whisper-non-streaming.ipynb`](https://github.com/huggingface/community-events/blob/main/whisper-fine-tuning-event/fine-tune-whisper-non-streaming.ipynb) 
+(see section [Streaming Mode](#streaming-mode)). This notebook can be copied to your model repository with the following 
+command:
+
+```bash
+cp ~/community-events/whisper-fine-tuning-event/fine-tune-whisper-non-streaming.ipynb .
+```
+
+4. **Launch Jupyter**
+
+First, we need to make sure `jupyterlab` is installed:
+
+```bash
+pip install jupyterlab
+```
+
+We can then link `jupyter lab` to our venv:
+```bash
+python -m ipykernel install --user --name=<your-venv-name>
+```
+
+We recommend running training through a `tmux` session. This means that training won't be interrupted when you close 
+your SSH connection. To start a `tmux` session named `mysession`:
+
+```bash
+tmux new -s mysession
+```
+(if `tmux` is not installed, you can install it through: `sudo apt-get install tmux`)
+
+Once in the `tmux` session, we can launch `jupyter lab`:
+
+```bash
+jupyter lab --port 8888
+```
+
+5. **Open Jupyter in browser**
+
+Now, this is the hardest step of running training from a Jupyter Notebook! Open a second terminal window on your local 
+machine and SSH into your GPU again. This time, it doesn't matter whether we include the `-L 8888:localhost:8888` part, 
+the important thing is that you re-enter your GPU device in a new SSH window.
+
+Once SSH'd into your GPU, view all running `jupyter lab` sessions:
+
+```bash
+jupyter lab list
+```
+
+Copy the URL for the lab corresponding to port 8888 your clipboard, it will take the form `http://localhost:8888/?token=...`. 
+On your local desktop, open a web browser window (Safari, Firefox, Chrome, etc.). Paste the URL into the browser web 
+address bar and press Enter.
+
+Voilà! We're now running a Jupyter Notebook on our GPU machine through the web browser on our local device!
+
+6. **Open fine-tuning notebook**
+
+We can use the file explorer on the left to go to our model repository and open the Jupyter notebook `fine_tune_whisper_streaming.ipynb`. 
+In the top right of the notebook, you'll see a small window that says "Python 3". Clicking on this window will open a 
+dropdown menu, from which we can select a Python kernel. Select your venv from this dropdown menu. This will ensure that 
+you run the notebook in the venv we previously set up.
+
+You can now run this notebook from start to finish and fine-tune the Whisper model as you desire 🤗 The notebook 
+contains pointers for where you need to change variables for your language.
+
+Since we're operating within a `tmux` session, we're free to close our SSH connection and browser window when we desire. 
+Training won't be interrupted by closing this window. However, the notebook will cease to update, so you should make 
+sure that training is working before closing the notebook. You can monitor training progress through your model repo 
+on the Hugging Face Hub under the "Training Metrics" tab.
+
+### Google Colab
+The Google Colab for fine-tuning Whisper is entirely self-contained. No need to set up an environment or sping up a GPU.
+You can access it through the following link:
+
+<a target="_blank" href="https://colab.research.google.com/github/huggingface/community-events/blob/main/whisper-fine-tuning-event/fine_tune_whisper_streaming_colab.ipynb">
+    <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
+</a>
+
+### Recommended Training Configurations
+
+In this section, we provide guidance for appropriate training and evaluation batch sizes depending on your GPU device. 
+Since the Whisper model expects log-Mel input features of a fixed dimension, the GPU memory required by the models is 
+the same for audio samples of any length. Thus, these recommendations should stand for all 16/40GB GPU devices. However, 
+if you experience out-of-memory errors, we recommend reducing the `per_device_train_batch_size` by factors of 2 and 
+increasing the `gradient_accumulation_steps` to compensate.
+
+If you want to explore methods for reducing the memory of the Whisper model, check out the section [Tips and Tricks](#tips-and-tricks).
+
+#### V100 / 16 GB GPU
+
+| Model  | Train Batch Size | Gradient Acc Steps | Eval Batch size |
+|--------|------------------|--------------------|-----------------|
+| small  | 16               | 2                  | 8               |
+| medium | 2                | 16                 | 1               |
+
+It is advised to run the "small" checkpoint if training on a V100 device. Running the medium checkpoint will take 
+upwards of 12 hours for 5k training steps. We reckon you're better off training the "small" checkpoint for longer!
+
+#### A100 / 40GB GPU
+
+| Model  | Train Batch Size | Gradient Acc Steps | Eval Batch size |
+|--------|------------------|--------------------|-----------------|
+| small  | 64               | 1                  | 32              |
+| medium | 32               | 1                  | 16              |
+
+### Punctuation, Casing and Normalisation
+
+When using the Python training script, removing casing for the training data is enabled by passing the flag `--do_lower_case`. 
+Removing punctuation in the training data is achieved by passing the flag `--do_remove_punctuation`. Both of these flags 
+default to False, and we **do not** recommend setting either of them to True. This will ensure your fine-tuned model 
+learns to predict casing and punctuation. Normalisation is only applied during evaluation by setting the flag 
+`--do_normalize_eval` (which defaults to True and recommend setting). Normalisation is performed according to the 
+'official' Whisper normaliser. This normaliser applies the following basic standardisation for non-English text:
+1. Remove any phrases between matching brackets ([, ]).
+2. Remove any phrases between matching parentheses ((, )).
+3. Replace any markers, symbols, and punctuation characters with a space, i.e. when the Unicode category of each character in the NFKC-normalized string starts with M, S, or P.
+4. Make the text lowercase.
+5. Replace any successive whitespace characters with a space.
+
+Similarly, in the notebooks, removing casing in the training data is enabled by setting the variable `do_lower_case = True`, 
+and punctuation by `do_remove_punctuation = True`. We do not recommend setting either of these to True to ensure that 
+your model learns to predict casing and punctuation. Thus, they are set to False by default. Normalisation is only 
+applied during evaluation by setting the variable `do_normalize_eval=True` (which we do recommend setting). 
+
+## Evaluation
+
+We'll be running a live leaderboard throughout the event to track the best performing models across all languages. The leaderboard will track your models performance across *all* the speech recognition models available on the hub for your chosen language and dataset.
+
+You can find the leaderboard [here](https://huggingface.co/spaces/autoevaluate/leaderboards?dataset=common_voice_11_0&only_verified=0&task=automatic-speech-recognition&config=th&split=train%2Bvalidation&metric=wer) 📈.
+
+Each participant should evaluate their fine-tuned Whisper checkpoint on the `"test"` split of the Common Voice 11 
+dataset for their respective language. For languages that are not part of the Common Voice 11 dataset, please contact 
+the organisers on Discord so that we can work together to find suitable evaluation data.
+
+We recommend running evaluation during training by setting your eval dataset to the `"test"` split of Common Voice 11.
+We'll also provide you with a standalone evaluation script so that you can test your model after training on Common Voice 
+or other datasets of your choice.
+
+In addition to running evaluation while training, you can noe use your Whisper checkpoints to run evaluation on *any* speech recognition dataset on the hub. The [run_eval_whisper_streaming.py](https://github.com/huggingface/community-events/blob/main/whisper-fine-tuning-event/run_eval_whisper_streaming.py) script loads your whisper checkpoints, runs batch inference on your specified dataset and returns the WER.
+
+You can use the script as follows:
+```bash
+python run_eval_whisper_streaming.py --model_id="openai/whisper-tiny" --dataset="google/fleurs" --config="ar_eg" --device=0 --language="ar"
+```
+
+The evaluation script can be customised with the following parameters:
+1. `model_id` - Whisper model identifier e.g. `openai/whisper-tiny`
+2. `dataset` - Dataset name to evaluate the `model_id` on. Default value: `mozilla-foundation/common_voice_11_0`
+3. `config` - Config of the dataset. e.g. `'en'` for the English split of Common Voice
+4. `split` - Split of the dataset. Default value: `test`
+5. `batch_size` - Number of samples to go through each streamed batch for inference. Default value: `16`
+6. `max_eval_samples` - Max number of samples to be evaluated from the dataset. Put a lower number e.g. `64` for testing this script. **Only use this for testing the script**
+7. `streaming` - Whether you'd like to download the entire dataset or stream it during the evaluation. Default value: `True`
+8. `language` - Language you want the `model_id` to transcribe the audio in.
+9. `device` - The device to run the pipeline on. e.g. `0` for running on GPU 0. Default value: -1 for CPU.
+
+## Building a Demo
+
+Finally, on to the fun part! Time to sit back and watch the model transcribe audio. We've created a [template Gradio demo](https://huggingface.co/spaces/whisper-event/whisper-demo) 
+that you can use to showcase your fine-tuned Whisper model 📢
+
+Click the link to duplicate the template demo to your account: https://huggingface.co/spaces/whisper-event/whisper-demo?duplicate=true
+
+We recommend giving your space a similar name to your fine-tuned model (e.g. `whisper-demo-es`) and setting the visibility 
+to "Public". 
+
+Once you've duplicated the Space to your account, click "Files and versions" -> "app.py" -> "edit". Change the model 
+identifier to your fine-tuned model (line 9). Scroll to the bottom of the page and click "Commit changes to `main`". 
+The demo will reboot, this time using your fine-tuned model. You can share this demo with your friends and family so 
+that they can use the model that you've trained!
+
+*Checkout our video tutorial to get a better understanding 👉️ [YouTube Video](https://www.youtube.com/watch?v=VQYuvl6-9VE)*
+
+## Communication and Problems
+
+If you encounter any problems or have any questions, you should use one of the following platforms
+depending on your type of problem. Hugging Face is an "open-source-first" organisation, meaning 
+that we'll try to solve all problems in the most public and transparent way possible so that everybody
+in the community benefits.
+
+The following paragraph summarises the platform to use for each kind of problem:
+
+- Problem/question/bug with the 🤗 Datasets library that you think is a general problem that also impacts other people, please open an [Issue on Datasets](https://github.com/huggingface/datasets/issues/new?assignees=&labels=bug&template=bug-report.md&title=) and ping @sanchit-gandhi and @vaibhavs10.
+- Problem/question/bug with the 🤗 Transformers library that you think is a general problem that also impacts other people, please open an [Issue on Transformers](https://github.com/huggingface/transformers/issues/new?assignees=&labels=&template=bug-report.md&title=) and ping @sanchit-gandhi and @vaibhavs10.
+- Problem/question with a modified, customised training script that is less likely to impact other people, please post your problem/question [on the forum](https://discuss.huggingface.co/) and ping @sanchit-gandhi and @vaibhavs10.
+- Problem/question regarding access or set up of a Lambda GPU, please ask in the Discord channel **#lambdalabs-infra-support**.
+- Other questions regarding the event, rules of the event, or if you are unsure where to post your question, please ask in the Discord channel **#events**.
+
+## Talks
+
+We are very excited to be hosting talks from Open AI, Meta AI and Hugging Face to help you get a better understanding of the Whisper model, the VoxPopuli dataset and details about the fine-tuning event itself!
+
+| **Speaker**                  | **Topic**                                                  | **Time**                      | **Video**                                                                                                                 |
+|------------------------------|------------------------------------------------------------|-------------------------------|---------------------------------------------------------------------------------------------------------------------------|
+| Sanchit Gandhi, Hugging Face | Introduction to Whisper Fine-Tuning Event                  | 15:00 UTC, 2nd December, 2022 | [![Youtube](https://www.youtube.com/s/desktop/f506bd45/img/favicon_32.png)](https://www.youtube.com/watch?v=1cVBLOMlv3w)  |
+| Jong Wook Kim, OpenAI        | [Whisper Model](https://cdn.openai.com/papers/whisper.pdf) | 16:30 UTC, 5th December, 2022 | [![Youtube](https://www.youtube.com/s/desktop/f506bd45/img/favicon_32.png)](https://www.youtube.com/watch?v=fZMiD8sDzzg ) |
+| Changhan Wang, MetaAI        | [VoxPopuli Dataset](https://arxiv.org/abs/2101.00390)      | 17:30 UTC, 5th December, 2022 | [![Youtube](https://www.youtube.com/s/desktop/f506bd45/img/favicon_32.png)](https://www.youtube.com/watch?v=fZMiD8sDzzg ) |
+
+## Tips and Tricks
+
+We include three memory saving tricks that you can explore to run the fine-tuning scripts with larger batch-sizes and 
+potentially larger checkpoints.
+
+### Adam 8bit
+The [Adam optimiser](https://arxiv.org/abs/1412.6980a) requires two params (betas) for every model parameter. So the memory requirement of the optimiser is 
+**two times** that of the model. You can switch to using an 8bit version of the Adam optimiser from [`bitsandbytes`](https://github.com/TimDettmers/bitsandbytes#bitsandbytes). 
+This will cast the optimiser parameters into 8bit precision, saving you a lot of memory and potentially allowing you to run bigger batch sizes.
+To use Adam 8bit, you first need to pip install `bitsandbytes`:
+
+```bash
+pip install bitsandbytes
+```
+
+Then, set `optim="adamw_bnb_8bit"`, either in your `run.sh` file if running from a Python script, or when you 
+instantiate the Seq2SeqTrainingArguments from a Jupyter Notebook or Google Colab:
+
+```python
+from transformers import Seq2SeqTrainingArguments
+
+training_args = Seq2SeqTrainingArguments(
+    output_dir="./",
+    per_device_train_batch_size=64,
+    gradient_accumulation_steps=1,  # increase by 2x for every 2x decrease in batch size
+    learning_rate=1e-5,
+    warmup_steps=500,
+    max_steps=5000,
+    gradient_checkpointing=True,
+    fp16=True,
+    evaluation_strategy="steps",
+    per_device_eval_batch_size=8,
+    predict_with_generate=True,
+    generation_max_length=225,
+    save_steps=1000,
+    eval_steps=1000,
+    logging_steps=25,
+    report_to=["tensorboard"],
+    load_best_model_at_end=True,
+    metric_for_best_model="wer",
+    greater_is_better=False,
+    push_to_hub=True,
+    optim="adamw_bnb_8bit"
+)
+```
+
+### Adafactor
+
+Rather than using Adam, you can use a different optimiser all together. Adam requires two optimiser params per one model 
+param, but [Adafactor](https://arxiv.org/abs/1804.04235) uses only one. To enable Adafactor, set `optim="adafactor"` in the 
+`Seq2SeqTrainingArguments`. You can expect to double your training batch size when using Adafactor compared to Adam. 
+
+A word of caution: Adafactor is untested for fine-tuning Whisper, so we are unsure sure how 
+Adafactor performance compares to Adam! Typically, using Adafactor results in **slower convergence** than using Adam or 
+Adam 8bit. For this reason, we recommend Adafactor as an **experimental feature** only.
+
+### DeepSpeed
+
+DeepSpeed is a framework for training larger deep learning models with limited GPU resources by optimising GPU utilisation. 
+We provide implementation details for DeepSpeed ZeRo Stage 2, which partitions the optimiser states (ZeRO stage 1) and gradients 
+(ZeRO stage 2). With DeepSpeed, it is more than possible to train the medium Whisper checkpoint on a V100, or the large 
+checkpoint on an A100. For more details, we refer you to the blog post by the original authors: [DeepSpeed ZeRO](https://www.microsoft.com/en-us/research/blog/zero-deepspeed-new-system-optimizations-enable-training-models-with-over-100-billion-parameters/).
+
+Using DeepSpeed with 🤗 Transformers is straightforward. First, we need to install the packages 🤗 Accelerate and DeepSpeed: 
+
+```bash
+pip install -U accelerate deepspeed
+```
+
+The DeepSpeed configuration file specifies precisely what form of optimiser/gradient offloading we are going to perform.
+The key to getting a huge improvement on a single GPU with DeepSpeed is to have at least the provided DeepSpeed configuration 
+in the configuration file [`ds_config.json`](https://github.com/huggingface/community-events/blob/main/whisper-fine-tuning-event/ds_config.json). 
+
+You can copy the DeepSpeed configuration file to your model repository as follows:
+
+```bash
+cp ~/community-events/whisper-fine-tuning-event/ds_config.json .
+```
+
+### Python Script
+
+Using DeepSpeed with the Python training script requires two changes to the `run.sh` file. Firstly, we launch the script using `deepspeed` 
+instead of Python. Secondly, we pass the DeepSpeed config `ds_config.json` as a training argument. The remainder of the `run.sh` 
+file takes the same format as using the native Trainer configuration:
+
+```bash
+deepspeed run_speech_recognition_seq2seq_streaming.py \
+	--deepspeed="ds_config.json" \
+	--model_name_or_path="openai/whisper-small" \
+	--dataset_name="mozilla-foundation/common_voice_11_0" \
+	--dataset_config_name="es" \
+	--language="spanish" \
+	--train_split_name="train+validation" \
+	--eval_split_name="test" \
+	--model_index_name="Whisper Small Spanish" \
+	--max_steps="5000" \
+	--output_dir="./" \
+	--per_device_train_batch_size="64" \
+	--per_device_eval_batch_size="32" \
+	--logging_steps="25" \
+	--learning_rate="1e-5" \
+	--warmup_steps="500" \
+	--evaluation_strategy="steps" \
+	--eval_steps="1000" \
+	--save_strategy="steps" \
+	--save_steps="1000" \
+	--generation_max_length="225" \
+	--length_column_name="input_length" \
+	--max_duration_in_seconds="30" \
+	--text_column_name="sentence" \
+	--freeze_feature_encoder="False" \
+	--report_to="tensorboard" \
+	--metric_for_best_model="wer" \
+	--greater_is_better="False" \
+	--load_best_model_at_end \
+	--gradient_checkpointing \
+	--fp16 \
+	--overwrite_output_dir \
+	--do_train \
+	--do_eval \
+	--predict_with_generate \
+	--do_normalize_eval \
+	--streaming \
+	--use_auth_token \
+	--push_to_hub
+```
+
+### Jupyter Notebook
+
+Using DeepSpeed with the template Jupyter Notebooks requires two changes. Firstly, we add the following code cell at the 
+start of the notebook to configure the DeepSpeed environment:
+
+```python
+# DeepSpeed requires a distributed environment even when only one process is used.
+# This emulates a launcher in the notebook
+import os
+
+os.environ["MASTER_ADDR"] = "localhost"
+os.environ["MASTER_PORT"] = "9994"  # modify if RuntimeError: Address already in use
+os.environ["RANK"] = "0"
+os.environ["LOCAL_RANK"] = "0"
+os.environ["WORLD_SIZE"] = "1"
+```
+
+Secondly, we pass the DeepSpeed config file to the training args:
+
+```python
+training_args = Seq2SeqTrainingArguments(..., deepspeed="ds_config.json")
+```
+
+### Recommended Batch Sizes with DeepSpeed
+
+Using DeepSpeed, it is possible to fit larger batch sizes and even larger checkpoints on your device, be it a V100 or 
+A100. We provide recommended batch sizes for the three checkpoint sizes of interest for 16GB GPUs and 40GB GPUs. As before,
+these batch sizes are only indicative: you should tune the batch size depending on your device, checkpoint and language.
+
+#### V100 / 16 GB GPU
+
+| Model  | Train Batch Size | Gradient Acc Steps | Eval Batch size | Speed   |
+|--------|------------------|--------------------|-----------------|---------|
+| small  | 32               | 1                  | 16              | 1.3s/it |
+| medium | 16               | 1 or 2             | 8               | 2.0s/it |
+| large  | 8                | 2 or 4             | 4               | 3.8s/it |
+
+#### A100 / 40GB GPU
+
+| Model  | Train Batch Size | Gradient Acc Steps | Eval Batch size | Speed   |
+|--------|------------------|--------------------|-----------------|---------|
+| small  | 64               | 1                  | 32              | 2.3s/it |
+| medium | 64               | 1                  | 32              | 5.8s/it |
+| large  | 32               | 1 or 2             | 16              | 5.9s/it |
+
+
+## Scripts & Colabs
+
+1. [Whirlwind tour of Whispering with 🤗Transformers](https://colab.research.google.com/drive/1l290cRv4RdvuLNlSeo9WexByHaNWs3s3?usp=sharing)
+2. [8bit inference for Whisper large model (6.5 gig VRAM) 🤯](https://colab.research.google.com/drive/1EMOwwfm1V1fHxH7eT1LLg7yBjhTooB6j?usp=sharing)
+
+<!--- TODO: VB - Move these colabs to a GitHub repo --->
+
+## Feedback
+
+We would love to get your feedback on the event! If you have a spare ten minutes, we'd appreciate you filling out the 
+feedback form at: https://forms.gle/7hvrTE8NaSdQwwU68

added_tokens.json

@@ -0,0 +1,109 @@
+{
+  "<|af|>": 50327,
+  "<|am|>": 50334,
+  "<|ar|>": 50272,
+  "<|as|>": 50350,
+  "<|az|>": 50304,
+  "<|ba|>": 50355,
+  "<|be|>": 50330,
+  "<|bg|>": 50292,
+  "<|bn|>": 50302,
+  "<|bo|>": 50347,
+  "<|br|>": 50309,
+  "<|bs|>": 50315,
+  "<|ca|>": 50270,
+  "<|cs|>": 50283,
+  "<|cy|>": 50297,
+  "<|da|>": 50285,
+  "<|de|>": 50261,
+  "<|el|>": 50281,
+  "<|endoftext|>": 50257,
+  "<|en|>": 50259,
+  "<|es|>": 50262,
+  "<|et|>": 50307,
+  "<|eu|>": 50310,
+  "<|fa|>": 50300,
+  "<|fi|>": 50277,
+  "<|fo|>": 50338,
+  "<|fr|>": 50265,
+  "<|gl|>": 50319,
+  "<|gu|>": 50333,
+  "<|haw|>": 50352,
+  "<|ha|>": 50354,
+  "<|hi|>": 50276,
+  "<|hr|>": 50291,
+  "<|ht|>": 50339,
+  "<|hu|>": 50286,
+  "<|hy|>": 50312,
+  "<|id|>": 50275,
+  "<|is|>": 50311,
+  "<|it|>": 50274,
+  "<|iw|>": 50279,
+  "<|ja|>": 50266,
+  "<|jw|>": 50356,
+  "<|ka|>": 50329,
+  "<|kk|>": 50316,
+  "<|km|>": 50323,
+  "<|kn|>": 50306,
+  "<|ko|>": 50264,
+  "<|la|>": 50294,
+  "<|lb|>": 50345,
+  "<|ln|>": 50353,
+  "<|lo|>": 50336,
+  "<|lt|>": 50293,
+  "<|lv|>": 50301,
+  "<|mg|>": 50349,
+  "<|mi|>": 50295,
+  "<|mk|>": 50308,
+  "<|ml|>": 50296,
+  "<|mn|>": 50314,
+  "<|mr|>": 50320,
+  "<|ms|>": 50282,
+  "<|mt|>": 50343,
+  "<|my|>": 50346,
+  "<|ne|>": 50313,
+  "<|nl|>": 50271,
+  "<|nn|>": 50342,
+  "<|nocaptions|>": 50362,
+  "<|notimestamps|>": 50363,
+  "<|no|>": 50288,
+  "<|oc|>": 50328,
+  "<|pa|>": 50321,
+  "<|pl|>": 50269,
+  "<|ps|>": 50340,
+  "<|pt|>": 50267,
+  "<|ro|>": 50284,
+  "<|ru|>": 50263,
+  "<|sa|>": 50344,
+  "<|sd|>": 50332,
+  "<|si|>": 50322,
+  "<|sk|>": 50298,
+  "<|sl|>": 50305,
+  "<|sn|>": 50324,
+  "<|so|>": 50326,
+  "<|sq|>": 50317,
+  "<|sr|>": 50303,
+  "<|startoflm|>": 50360,
+  "<|startofprev|>": 50361,
+  "<|startoftranscript|>": 50258,
+  "<|su|>": 50357,
+  "<|sv|>": 50273,
+  "<|sw|>": 50318,
+  "<|ta|>": 50287,
+  "<|te|>": 50299,
+  "<|tg|>": 50331,
+  "<|th|>": 50289,
+  "<|tk|>": 50341,
+  "<|tl|>": 50348,
+  "<|transcribe|>": 50359,
+  "<|translate|>": 50358,
+  "<|tr|>": 50268,
+  "<|tt|>": 50351,
+  "<|uk|>": 50280,
+  "<|ur|>": 50290,
+  "<|uz|>": 50337,
+  "<|vi|>": 50278,
+  "<|yi|>": 50335,
+  "<|yo|>": 50325,
+  "<|zh|>": 50260
+}

config.json

@@ -0,0 +1,41 @@
+{
+  "_name_or_path": "openai/whisper-tiny",
+  "activation_dropout": 0.0,
+  "activation_function": "gelu",
+  "architectures": [
+    "WhisperForConditionalGeneration"
+  ],
+  "attention_dropout": 0.0,
+  "begin_suppress_tokens": [
+    220,
+    50257
+  ],
+  "bos_token_id": 50257,
+  "d_model": 384,
+  "decoder_attention_heads": 6,
+  "decoder_ffn_dim": 1536,
+  "decoder_layerdrop": 0.0,
+  "decoder_layers": 4,
+  "decoder_start_token_id": 50258,
+  "dropout": 0.0,
+  "encoder_attention_heads": 6,
+  "encoder_ffn_dim": 1536,
+  "encoder_layerdrop": 0.0,
+  "encoder_layers": 4,
+  "eos_token_id": 50257,
+  "forced_decoder_ids": null,
+  "init_std": 0.02,
+  "is_encoder_decoder": true,
+  "max_length": 448,
+  "max_source_positions": 1500,
+  "max_target_positions": 448,
+  "model_type": "whisper",
+  "num_hidden_layers": 4,
+  "num_mel_bins": 80,
+  "pad_token_id": 50257,
+  "scale_embedding": false,
+  "torch_dtype": "float32",
+  "transformers_version": "4.26.0.dev0",
+  "use_cache": true,
+  "vocab_size": 51865
+}

ds_config.json

@@ -0,0 +1,50 @@
+{
+    "fp16": {
+        "enabled": "auto",
+        "loss_scale": 0,
+        "loss_scale_window": 1000,
+        "initial_scale_power": 16,
+        "hysteresis": 2,
+        "min_loss_scale": 1
+    },
+
+    "optimizer": {
+        "type": "AdamW",
+        "params": {
+            "lr": "auto",
+            "betas": "auto",
+            "eps": "auto",
+            "weight_decay": "auto"
+        }
+    },
+
+   "scheduler": {
+         "type": "WarmupDecayLR",
+         "params": {
+             "last_batch_iteration": -1,
+             "total_num_steps": "auto",
+             "warmup_min_lr": "auto",
+             "warmup_max_lr": "auto",
+             "warmup_num_steps": "auto"
+         }
+     },
+
+    "zero_optimization": {
+        "stage": 2,
+        "offload_optimizer": {
+            "device": "cpu",
+            "pin_memory": true
+        },
+        "allgather_partitions": true,
+        "allgather_bucket_size": 2e8,
+        "overlap_comm": true,
+        "reduce_scatter": true,
+        "reduce_bucket_size": 2e8,
+        "contiguous_gradients": true
+    },
+
+    "gradient_accumulation_steps": "auto",
+    "gradient_clipping": "auto",
+    "train_batch_size": "auto",
+    "train_micro_batch_size_per_gpu": "auto"
+}

fine-tune-whisper-non-streaming.ipynb

@@ -0,0 +1,1207 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "75b58048-7d14-4fc6-8085-1fc08c81b4a6",
+   "metadata": {
+    "id": "75b58048-7d14-4fc6-8085-1fc08c81b4a6"
+   },
+   "source": [
+    "# Fine-Tune Whisper For Multilingual ASR with 🤗 Transformers"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "fbfa8ad5-4cdc-4512-9058-836cbbf65e1a",
+   "metadata": {
+    "id": "fbfa8ad5-4cdc-4512-9058-836cbbf65e1a"
+   },
+   "source": [
+    "In this Colab, we present a step-by-step guide on how to fine-tune Whisper \n",
+    "for any multilingual ASR dataset using Hugging Face 🤗 Transformers. This is a \n",
+    "more \"hands-on\" version of the accompanying [blog post](https://huggingface.co/blog/fine-tune-whisper). \n",
+    "For a more in-depth explanation of Whisper, the Common Voice dataset and the theory behind fine-tuning, the reader is advised to refer to the blog post."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "afe0d503-ae4e-4aa7-9af4-dbcba52db41e",
+   "metadata": {
+    "id": "afe0d503-ae4e-4aa7-9af4-dbcba52db41e"
+   },
+   "source": [
+    "## Introduction"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9ae91ed4-9c3e-4ade-938e-f4c2dcfbfdc0",
+   "metadata": {
+    "id": "9ae91ed4-9c3e-4ade-938e-f4c2dcfbfdc0"
+   },
+   "source": [
+    "Whisper is a pre-trained model for automatic speech recognition (ASR) \n",
+    "published in [September 2022](https://openai.com/blog/whisper/) by the authors \n",
+    "Alec Radford et al. from OpenAI. Unlike many of its predecessors, such as \n",
+    "[Wav2Vec 2.0](https://arxiv.org/abs/2006.11477), which are pre-trained \n",
+    "on un-labelled audio data, Whisper is pre-trained on a vast quantity of \n",
+    "**labelled** audio-transcription data, 680,000 hours to be precise. \n",
+    "This is an order of magnitude more data than the un-labelled audio data used \n",
+    "to train Wav2Vec 2.0 (60,000 hours). What is more, 117,000 hours of this \n",
+    "pre-training data is multilingual ASR data. This results in checkpoints \n",
+    "that can be applied to over 96 languages, many of which are considered \n",
+    "_low-resource_.\n",
+    "\n",
+    "When scaled to 680,000 hours of labelled pre-training data, Whisper models \n",
+    "demonstrate a strong ability to generalise to many datasets and domains.\n",
+    "The pre-trained checkpoints achieve competitive results to state-of-the-art \n",
+    "ASR systems, with near 3% word error rate (WER) on the test-clean subset of \n",
+    "LibriSpeech ASR and a new state-of-the-art on TED-LIUM with 4.7% WER (_c.f._ \n",
+    "Table 8 of the [Whisper paper](https://cdn.openai.com/papers/whisper.pdf)).\n",
+    "The extensive multilingual ASR knowledge acquired by Whisper during pre-training \n",
+    "can be leveraged for other low-resource languages; through fine-tuning, the \n",
+    "pre-trained checkpoints can be adapted for specific datasets and languages \n",
+    "to further improve upon these results. We'll show just how Whisper can be fine-tuned \n",
+    "for low-resource languages in this Colab."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e59b91d6-be24-4b5e-bb38-4977ea143a72",
+   "metadata": {
+    "id": "e59b91d6-be24-4b5e-bb38-4977ea143a72"
+   },
+   "source": [
+    "<figure>\n",
+    "<img src=\"https://raw.githubusercontent.com/sanchit-gandhi/notebooks/main/whisper_architecture.svg\" alt=\"Trulli\" style=\"width:100%\">\n",
+    "<figcaption align = \"center\"><b>Figure 1:</b> Whisper model. The architecture \n",
+    "follows the standard Transformer-based encoder-decoder model. A \n",
+    "log-Mel spectrogram is input to the encoder. The last encoder \n",
+    "hidden states are input to the decoder via cross-attention mechanisms. The \n",
+    "decoder autoregressively predicts text tokens, jointly conditional on the \n",
+    "encoder hidden states and previously predicted tokens. Figure source: \n",
+    "<a href=\"https://openai.com/blog/whisper/\">OpenAI Whisper Blog</a>.</figcaption>\n",
+    "</figure>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "21b6316e-8a55-4549-a154-66d3da2ab74a",
+   "metadata": {
+    "id": "21b6316e-8a55-4549-a154-66d3da2ab74a"
+   },
+   "source": [
+    "The Whisper checkpoints come in five configurations of varying model sizes.\n",
+    "The smallest four are trained on either English-only or multilingual data.\n",
+    "The largest checkpoint is multilingual only. All nine of the pre-trained checkpoints \n",
+    "are available on the [Hugging Face Hub](https://huggingface.co/models?search=openai/whisper). The \n",
+    "checkpoints are summarised in the following table with links to the models on the Hub:\n",
+    "\n",
+    "| Size   | Layers | Width | Heads | Parameters | English-only                                         | Multilingual                                      |\n",
+    "|--------|--------|-------|-------|------------|------------------------------------------------------|---------------------------------------------------|\n",
+    "| tiny   | 4      | 384   | 6     | 39 M       | [✓](https://huggingface.co/openai/whisper-tiny.en)   | [✓](https://huggingface.co/openai/whisper-tiny.)  |\n",
+    "| base   | 6      | 512   | 8     | 74 M       | [✓](https://huggingface.co/openai/whisper-base.en)   | [✓](https://huggingface.co/openai/whisper-base)   |\n",
+    "| small  | 12     | 768   | 12    | 244 M      | [✓](https://huggingface.co/openai/whisper-small.en)  | [✓](https://huggingface.co/openai/whisper-small)  |\n",
+    "| medium | 24     | 1024  | 16    | 769 M      | [✓](https://huggingface.co/openai/whisper-medium.en) | [✓](https://huggingface.co/openai/whisper-medium) |\n",
+    "| large  | 32     | 1280  | 20    | 1550 M     | x                                                    | [✓](https://huggingface.co/openai/whisper-large)  |\n",
+    "\n",
+    "For demonstration purposes, we'll fine-tune the multilingual version of the \n",
+    "[`\"small\"`](https://huggingface.co/openai/whisper-small) checkpoint with 244M params (~= 1GB). \n",
+    "As for our data, we'll train and evaluate our system on a low-resource language \n",
+    "taken from the [Common Voice](https://huggingface.co/datasets/mozilla-foundation/common_voice_11_0)\n",
+    "dataset. We'll show that with as little as 8 hours of fine-tuning data, we can achieve \n",
+    "strong performance in this language."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3a680dfc-cbba-4f6c-8a1f-e1a5ff3f123a",
+   "metadata": {
+    "id": "3a680dfc-cbba-4f6c-8a1f-e1a5ff3f123a"
+   },
+   "source": [
+    "------------------------------------------------------------------------\n",
+    "\n",
+    "\\\\({}^1\\\\) The name Whisper follows from the acronym “WSPSR”, which stands for “Web-scale Supervised Pre-training for Speech Recognition”."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b219c9dd-39b6-4a95-b2a1-3f547a1e7bc0",
+   "metadata": {
+    "id": "b219c9dd-39b6-4a95-b2a1-3f547a1e7bc0"
+   },
+   "source": [
+    "## Load Dataset"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "674429c5-0ab4-4adf-975b-621bb69eca38",
+   "metadata": {
+    "id": "674429c5-0ab4-4adf-975b-621bb69eca38"
+   },
+   "source": [
+    "Using 🤗 Datasets, downloading and preparing data is extremely simple. \n",
+    "We can download and prepare the Common Voice splits in just one line of code. \n",
+    "\n",
+    "First, ensure you have accepted the terms of use on the Hugging Face Hub: [mozilla-foundation/common_voice_11_0](https://huggingface.co/datasets/mozilla-foundation/common_voice_11_0). Once you have accepted the terms, you will have full access to the dataset and be able to download the data locally.\n",
+    "\n",
+    "Since Hindi is very low-resource, we'll combine the `train` and `validation` \n",
+    "splits to give approximately 8 hours of training data. We'll use the 4 hours \n",
+    "of `test` data as our held-out test set:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a2787582-554f-44ce-9f38-4180a5ed6b44",
+   "metadata": {
+    "id": "a2787582-554f-44ce-9f38-4180a5ed6b44"
+   },
+   "outputs": [],
+   "source": [
+    "from datasets import load_dataset, DatasetDict\n",
+    "\n",
+    "common_voice = DatasetDict()\n",
+    "\n",
+    "common_voice[\"train\"] = load_dataset(\"mozilla-foundation/common_voice_11_0\", \"hi\", split=\"train+validation\", use_auth_token=True)\n",
+    "common_voice[\"test\"] = load_dataset(\"mozilla-foundation/common_voice_11_0\", \"hi\", split=\"test\", use_auth_token=True)\n",
+    "\n",
+    "print(common_voice)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d5c7c3d6-7197-41e7-a088-49b753c1681f",
+   "metadata": {
+    "id": "d5c7c3d6-7197-41e7-a088-49b753c1681f"
+   },
+   "source": [
+    "Most ASR datasets only provide input audio samples (`audio`) and the \n",
+    "corresponding transcribed text (`sentence`). Common Voice contains additional \n",
+    "metadata information, such as `accent` and `locale`, which we can disregard for ASR.\n",
+    "Keeping the notebook as general as possible, we only consider the input audio and\n",
+    "transcribed text for fine-tuning, discarding the additional metadata information:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "20ba635d-518c-47ac-97ee-3cad25f1e0ce",
+   "metadata": {
+    "id": "20ba635d-518c-47ac-97ee-3cad25f1e0ce"
+   },
+   "outputs": [],
+   "source": [
+    "common_voice = common_voice.remove_columns([\"accent\", \"age\", \"client_id\", \"down_votes\", \"gender\", \"locale\", \"path\", \"segment\", \"up_votes\"])\n",
+    "\n",
+    "print(common_voice)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2d63b2d2-f68a-4d74-b7f1-5127f6d16605",
+   "metadata": {
+    "id": "2d63b2d2-f68a-4d74-b7f1-5127f6d16605"
+   },
+   "source": [
+    "## Prepare Feature Extractor, Tokenizer and Data"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "601c3099-1026-439e-93e2-5635b3ba5a73",
+   "metadata": {
+    "id": "601c3099-1026-439e-93e2-5635b3ba5a73"
+   },
+   "source": [
+    "The ASR pipeline can be de-composed into three stages: \n",
+    "1) A feature extractor which pre-processes the raw audio-inputs\n",
+    "2) The model which performs the sequence-to-sequence mapping \n",
+    "3) A tokenizer which post-processes the model outputs to text format\n",
+    "\n",
+    "In 🤗 Transformers, the Whisper model has an associated feature extractor and tokenizer, \n",
+    "called [WhisperFeatureExtractor](https://huggingface.co/docs/transformers/main/model_doc/whisper#transformers.WhisperFeatureExtractor)\n",
+    "and [WhisperTokenizer](https://huggingface.co/docs/transformers/main/model_doc/whisper#transformers.WhisperTokenizer) \n",
+    "respectively.\n",
+    "\n",
+    "We'll go through details for setting-up the feature extractor and tokenizer one-by-one!"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "560332eb-3558-41a1-b500-e83a9f695f84",
+   "metadata": {
+    "id": "560332eb-3558-41a1-b500-e83a9f695f84"
+   },
+   "source": [
+    "### Load WhisperFeatureExtractor"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "32ec8068-0bd7-412d-b662-0edb9d1e7365",
+   "metadata": {
+    "id": "32ec8068-0bd7-412d-b662-0edb9d1e7365"
+   },
+   "source": [
+    "The Whisper feature extractor performs two operations:\n",
+    "1. Pads / truncates the audio inputs to 30s: any audio inputs shorter than 30s are padded to 30s with silence (zeros), and those longer that 30s are truncated to 30s\n",
+    "2. Converts the audio inputs to _log-Mel spectrogram_ input features, a visual representation of the audio and the form of the input expected by the Whisper model"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "589d9ec1-d12b-4b64-93f7-04c63997da19",
+   "metadata": {
+    "id": "589d9ec1-d12b-4b64-93f7-04c63997da19"
+   },
+   "source": [
+    "<figure>\n",
+    "<img src=\"https://raw.githubusercontent.com/sanchit-gandhi/notebooks/main/spectrogram.jpg\" alt=\"Trulli\" style=\"width:100%\">\n",
+    "<figcaption align = \"center\"><b>Figure 2:</b> Conversion of sampled audio array to log-Mel spectrogram.\n",
+    "Left: sampled 1-dimensional audio signal. Right: corresponding log-Mel spectrogram. Figure source:\n",
+    "<a href=\"https://ai.googleblog.com/2019/04/specaugment-new-data-augmentation.html\">Google SpecAugment Blog</a>.\n",
+    "</figcaption>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b2ef54d5-b946-4c1d-9fdc-adc5d01b46aa",
+   "metadata": {
+    "id": "b2ef54d5-b946-4c1d-9fdc-adc5d01b46aa"
+   },
+   "source": [
+    "We'll load the feature extractor from the pre-trained checkpoint with the default values:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "bc77d7bb-f9e2-47f5-b663-30f7a4321ce5",
+   "metadata": {
+    "id": "bc77d7bb-f9e2-47f5-b663-30f7a4321ce5"
+   },
+   "outputs": [],
+   "source": [
+    "from transformers import WhisperFeatureExtractor\n",
+    "\n",
+    "feature_extractor = WhisperFeatureExtractor.from_pretrained(\"openai/whisper-small\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "93748af7-b917-4ecf-a0c8-7d89077ff9cb",
+   "metadata": {
+    "id": "93748af7-b917-4ecf-a0c8-7d89077ff9cb"
+   },
+   "source": [
+    "### Load WhisperTokenizer"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2bc82609-a9fb-447a-a2af-99597c864029",
+   "metadata": {
+    "id": "2bc82609-a9fb-447a-a2af-99597c864029"
+   },
+   "source": [
+    "The Whisper model outputs a sequence of _token ids_. The tokenizer maps each of these token ids to their corresponding text string. For Hindi, we can load the pre-trained tokenizer and use it for fine-tuning without any further modifications. We simply have to \n",
+    "specify the target language and the task. These arguments inform the \n",
+    "tokenizer to prefix the language and task tokens to the start of encoded \n",
+    "label sequences:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c7b07f9b-ae0e-4f89-98f0-0c50d432eab6",
+   "metadata": {
+    "id": "c7b07f9b-ae0e-4f89-98f0-0c50d432eab6",
+    "outputId": "5c004b44-86e7-4e00-88be-39e0af5eed69"
+   },
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "90d056e20b3e4f14ae0199a1a4ab1bb0",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Downloading:   0%|          | 0.00/829 [00:00<?, ?B/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "d82a88daec0e4f14add691b7b903064c",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Downloading:   0%|          | 0.00/1.04M [00:00<?, ?B/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "350acdb0f40e454099fa901e66de55f0",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Downloading:   0%|          | 0.00/494k [00:00<?, ?B/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "2e6a82a462cc411d90fa1bea4ee60790",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Downloading:   0%|          | 0.00/52.7k [00:00<?, ?B/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "c74bfee0198b4817832ea86e8e88d96c",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Downloading:   0%|          | 0.00/2.11k [00:00<?, ?B/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "04fb2d81eff646068e10475a08ae42f4",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Downloading:   0%|          | 0.00/2.06k [00:00<?, ?B/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "from transformers import WhisperTokenizer\n",
+    "\n",
+    "tokenizer = WhisperTokenizer.from_pretrained(\"openai/whisper-small\", language=\"Hindi\", task=\"transcribe\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d2ef23f3-f4a8-483a-a2dc-080a7496cb1b",
+   "metadata": {
+    "id": "d2ef23f3-f4a8-483a-a2dc-080a7496cb1b"
+   },
+   "source": [
+    "### Combine To Create A WhisperProcessor"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5ff67654-5a29-4bb8-a69d-0228946c6f8d",
+   "metadata": {
+    "id": "5ff67654-5a29-4bb8-a69d-0228946c6f8d"
+   },
+   "source": [
+    "To simplify using the feature extractor and tokenizer, we can _wrap_ \n",
+    "both into a single `WhisperProcessor` class. This processor object \n",
+    "inherits from the `WhisperFeatureExtractor` and `WhisperProcessor`, \n",
+    "and can be used on the audio inputs and model predictions as required. \n",
+    "In doing so, we only need to keep track of two objects during training: \n",
+    "the `processor` and the `model`:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "77d9f0c5-8607-4642-a8ac-c3ab2e223ea6",
+   "metadata": {
+    "id": "77d9f0c5-8607-4642-a8ac-c3ab2e223ea6"
+   },
+   "outputs": [],
+   "source": [
+    "from transformers import WhisperProcessor\n",
+    "\n",
+    "processor = WhisperProcessor.from_pretrained(\"openai/whisper-small\", language=\"Hindi\", task=\"transcribe\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "381acd09-0b0f-4d04-9eb3-f028ac0e5f2c",
+   "metadata": {
+    "id": "381acd09-0b0f-4d04-9eb3-f028ac0e5f2c"
+   },
+   "source": [
+    "### Prepare Data"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9649bf01-2e8a-45e5-8fca-441c13637b8f",
+   "metadata": {
+    "id": "9649bf01-2e8a-45e5-8fca-441c13637b8f"
+   },
+   "source": [
+    "Let's print the first example of the Common Voice dataset to see \n",
+    "what form the data is in:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6e6b0ec5-0c94-4e2c-ae24-c791be1b2255",
+   "metadata": {
+    "id": "6e6b0ec5-0c94-4e2c-ae24-c791be1b2255"
+   },
+   "outputs": [],
+   "source": [
+    "print(common_voice[\"train\"][0])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5a679f05-063d-41b3-9b58-4fc9c6ccf4fd",
+   "metadata": {
+    "id": "5a679f05-063d-41b3-9b58-4fc9c6ccf4fd"
+   },
+   "source": [
+    "Since \n",
+    "our input audio is sampled at 48kHz, we need to _downsample_ it to \n",
+    "16kHz prior to passing it to the Whisper feature extractor, 16kHz being the sampling rate expected by the Whisper model. \n",
+    "\n",
+    "We'll set the audio inputs to the correct sampling rate using dataset's \n",
+    "[`cast_column`](https://huggingface.co/docs/datasets/package_reference/main_classes.html?highlight=cast_column#datasets.DatasetDict.cast_column)\n",
+    "method. This operation does not change the audio in-place, \n",
+    "but rather signals to `datasets` to resample audio samples _on the fly_ the \n",
+    "first time that they are loaded:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f12e2e57-156f-417b-8cfb-69221cc198e8",
+   "metadata": {
+    "id": "f12e2e57-156f-417b-8cfb-69221cc198e8"
+   },
+   "outputs": [],
+   "source": [
+    "from datasets import Audio\n",
+    "\n",
+    "common_voice = common_voice.cast_column(\"audio\", Audio(sampling_rate=16000))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "00382a3e-abec-4cdd-a54c-d1aaa3ea4707",
+   "metadata": {
+    "id": "00382a3e-abec-4cdd-a54c-d1aaa3ea4707"
+   },
+   "source": [
+    "Re-loading the first audio sample in the Common Voice dataset will resample \n",
+    "it to the desired sampling rate:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "87122d71-289a-466a-afcf-fa354b18946b",
+   "metadata": {
+    "id": "87122d71-289a-466a-afcf-fa354b18946b"
+   },
+   "outputs": [],
+   "source": [
+    "print(common_voice[\"train\"][0])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3df7378a-a4c0-45d7-8d07-defbd1062ab6",
+   "metadata": {},
+   "source": [
+    "We'll define our pre-processing strategy. We advise that you **do not** lower-case the transcriptions or remove punctuation unless mixing different datasets. This will enable you to fine-tune Whisper models that can predict punctuation and casing. Later, you will see how we can evaluate the predictions without punctuation or casing, so that the models benefit from the WER improvement obtained by normalising the transcriptions while still predicting fully formatted transcriptions."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d041650e-1c48-4439-87b3-5b6f4a514107",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from transformers.models.whisper.english_normalizer import BasicTextNormalizer\n",
+    "\n",
+    "do_lower_case = False\n",
+    "do_remove_punctuation = False\n",
+    "\n",
+    "normalizer = BasicTextNormalizer()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "89e12c2e-2f14-479b-987b-f0c75c881095",
+   "metadata": {},
+   "source": [
+    "Now we can write a function to prepare our data ready for the model:\n",
+    "1. We load and resample the audio data by calling `batch[\"audio\"]`. As explained above, 🤗 Datasets performs any necessary resampling operations on the fly.\n",
+    "2. We use the feature extractor to compute the log-Mel spectrogram input features from our 1-dimensional audio array.\n",
+    "3. We perform any optional pre-processing (lower-case or remove punctuation).\n",
+    "4. We encode the transcriptions to label ids through the use of the tokenizer."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c085911c-a10a-41ef-8874-306e0503e9bb",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def prepare_dataset(batch):\n",
+    "    # load and (possibly) resample audio data to 16kHz\n",
+    "    audio = batch[\"audio\"]\n",
+    "\n",
+    "    # compute log-Mel input features from input audio array \n",
+    "    batch[\"input_features\"] = processor.feature_extractor(audio[\"array\"], sampling_rate=audio[\"sampling_rate\"]).input_features[0]\n",
+    "    # compute input length of audio sample in seconds\n",
+    "    batch[\"input_length\"] = len(audio[\"array\"]) / audio[\"sampling_rate\"]\n",
+    "    \n",
+    "    # optional pre-processing steps\n",
+    "    transcription = batch[\"sentence\"]\n",
+    "    if do_lower_case:\n",
+    "        transcription = transcription.lower()\n",
+    "    if do_remove_punctuation:\n",
+    "        transcription = normalizer(transcription).strip()\n",
+    "    \n",
+    "    # encode target text to label ids\n",
+    "    batch[\"labels\"] = processor.tokenizer(transcription).input_ids\n",
+    "    return batch"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8c960965-9fb6-466f-9dbd-c9d43e71d9d0",
+   "metadata": {
+    "id": "70b319fb-2439-4ef6-a70d-a47bf41c4a13"
+   },
+   "source": [
+    "We can apply the data preparation function to all of our training examples using dataset's `.map` method. The argument `num_proc` specifies how many CPU cores to use. Setting `num_proc` > 1 will enable multiprocessing. If the `.map` method hangs with multiprocessing, set `num_proc=1` and process the dataset sequentially."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7b73ab39-ffaf-4b9e-86e5-782963c6134b",
+   "metadata": {
+    "id": "7b73ab39-ffaf-4b9e-86e5-782963c6134b"
+   },
+   "outputs": [],
+   "source": [
+    "common_voice = common_voice.map(prepare_dataset, remove_columns=common_voice.column_names[\"train\"], num_proc=2)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "54ce0fdb-7218-4a4d-b175-383980fec0df",
+   "metadata": {},
+   "source": [
+    "Finally, we filter any training data with audio samples longer than 30s. These samples would otherwise be truncated by the Whisper feature-extractor which could affect the stability of training. We define a function that returns `True` for samples that are less than 30s, and `False` for those that are longer:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "01cb25ef-4bb0-4325-9461-f59198acadf6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "max_input_length = 30.0\n",
+    "\n",
+    "def is_audio_in_length_range(length):\n",
+    "    return length < max_input_length"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "30e676a8-7ca8-4850-8c5d-5b2b00d13fba",
+   "metadata": {},
+   "source": [
+    "We apply our filter function to all samples of our training dataset through 🤗 Datasets' `.filter` method:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "333f7f6e-6053-4d3b-8924-c733c79b82ac",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "common_voice[\"train\"] = common_voice[\"train\"].filter(\n",
+    "    is_audio_in_length_range,\n",
+    "    input_columns=[\"input_length\"],\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "263a5a58-0239-4a25-b0df-c625fc9c5810",
+   "metadata": {
+    "id": "263a5a58-0239-4a25-b0df-c625fc9c5810"
+   },
+   "source": [
+    "## Training and Evaluation"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a693e768-c5a6-453f-89a1-b601dcf7daf7",
+   "metadata": {
+    "id": "a693e768-c5a6-453f-89a1-b601dcf7daf7"
+   },
+   "source": [
+    "Now that we've prepared our data, we're ready to dive into the training pipeline. \n",
+    "The [🤗 Trainer](https://huggingface.co/transformers/master/main_classes/trainer.html?highlight=trainer)\n",
+    "will do much of the heavy lifting for us. All we have to do is:\n",
+    "\n",
+    "- Define a data collator: the data collator takes our pre-processed data and prepares PyTorch tensors ready for the model.\n",
+    "\n",
+    "- Evaluation metrics: during evaluation, we want to evaluate the model using the [word error rate (WER)](https://huggingface.co/metrics/wer) metric. We need to define a `compute_metrics` function that handles this computation.\n",
+    "\n",
+    "- Load a pre-trained checkpoint: we need to load a pre-trained checkpoint and configure it correctly for training.\n",
+    "\n",
+    "- Define the training configuration: this will be used by the 🤗 Trainer to define the training schedule.\n",
+    "\n",
+    "Once we've fine-tuned the model, we will evaluate it on the test data to verify that we have correctly trained it \n",
+    "to transcribe speech in Hindi."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8d230e6d-624c-400a-bbf5-fa660881df25",
+   "metadata": {
+    "id": "8d230e6d-624c-400a-bbf5-fa660881df25"
+   },
+   "source": [
+    "### Define a Data Collator"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "04def221-0637-4a69-b242-d3f0c1d0ee78",
+   "metadata": {
+    "id": "04def221-0637-4a69-b242-d3f0c1d0ee78"
+   },
+   "source": [
+    "The data collator for a sequence-to-sequence speech model is unique in the sense that it \n",
+    "treats the `input_features` and `labels` independently: the  `input_features` must be \n",
+    "handled by the feature extractor and the `labels` by the tokenizer.\n",
+    "\n",
+    "The `input_features` are already padded to 30s and converted to a log-Mel spectrogram \n",
+    "of fixed dimension by action of the feature extractor, so all we have to do is convert the `input_features`\n",
+    "to batched PyTorch tensors. We do this using the feature extractor's `.pad` method with `return_tensors=pt`.\n",
+    "\n",
+    "The `labels` on the other hand are un-padded. We first pad the sequences\n",
+    "to the maximum length in the batch using the tokenizer's `.pad` method. The padding tokens \n",
+    "are then replaced by `-100` so that these tokens are **not** taken into account when \n",
+    "computing the loss. We then cut the BOS token from the start of the label sequence as we \n",
+    "append it later during training.\n",
+    "\n",
+    "We can leverage the `WhisperProcessor` we defined earlier to perform both the \n",
+    "feature extractor and the tokenizer operations:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8326221e-ec13-4731-bb4e-51e5fc1486c5",
+   "metadata": {
+    "id": "8326221e-ec13-4731-bb4e-51e5fc1486c5"
+   },
+   "outputs": [],
+   "source": [
+    "import torch\n",
+    "\n",
+    "from dataclasses import dataclass\n",
+    "from typing import Any, Dict, List, Union\n",
+    "\n",
+    "@dataclass\n",
+    "class DataCollatorSpeechSeq2SeqWithPadding:\n",
+    "    processor: Any\n",
+    "\n",
+    "    def __call__(self, features: List[Dict[str, Union[List[int], torch.Tensor]]]) -> Dict[str, torch.Tensor]:\n",
+    "        # split inputs and labels since they have to be of different lengths and need different padding methods\n",
+    "        # first treat the audio inputs by simply returning torch tensors\n",
+    "        input_features = [{\"input_features\": feature[\"input_features\"]} for feature in features]\n",
+    "        batch = self.processor.feature_extractor.pad(input_features, return_tensors=\"pt\")\n",
+    "\n",
+    "        # get the tokenized label sequences\n",
+    "        label_features = [{\"input_ids\": feature[\"labels\"]} for feature in features]\n",
+    "        # pad the labels to max length\n",
+    "        labels_batch = self.processor.tokenizer.pad(label_features, return_tensors=\"pt\")\n",
+    "\n",
+    "        # replace padding with -100 to ignore loss correctly\n",
+    "        labels = labels_batch[\"input_ids\"].masked_fill(labels_batch.attention_mask.ne(1), -100)\n",
+    "\n",
+    "        # if bos token is appended in previous tokenization step,\n",
+    "        # cut bos token here as it's append later anyways\n",
+    "        if (labels[:, 0] == self.processor.tokenizer.bos_token_id).all().cpu().item():\n",
+    "            labels = labels[:, 1:]\n",
+    "\n",
+    "        batch[\"labels\"] = labels\n",
+    "\n",
+    "        return batch"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3cae7dbf-8a50-456e-a3a8-7fd005390f86",
+   "metadata": {
+    "id": "3cae7dbf-8a50-456e-a3a8-7fd005390f86"
+   },
+   "source": [
+    "Let's initialise the data collator we've just defined:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "fc834702-c0d3-4a96-b101-7b87be32bf42",
+   "metadata": {
+    "id": "fc834702-c0d3-4a96-b101-7b87be32bf42"
+   },
+   "outputs": [],
+   "source": [
+    "data_collator = DataCollatorSpeechSeq2SeqWithPadding(processor=processor)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d62bb2ab-750a-45e7-82e9-61d6f4805698",
+   "metadata": {
+    "id": "d62bb2ab-750a-45e7-82e9-61d6f4805698"
+   },
+   "source": [
+    "### Evaluation Metrics"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "66fee1a7-a44c-461e-b047-c3917221572e",
+   "metadata": {
+    "id": "66fee1a7-a44c-461e-b047-c3917221572e"
+   },
+   "source": [
+    "We'll use the word error rate (WER) metric, the 'de-facto' metric for assessing \n",
+    "ASR systems. For more information, refer to the WER [docs](https://huggingface.co/metrics/wer). We'll load the WER metric from 🤗 Evaluate:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b22b4011-f31f-4b57-b684-c52332f92890",
+   "metadata": {
+    "id": "b22b4011-f31f-4b57-b684-c52332f92890"
+   },
+   "outputs": [],
+   "source": [
+    "import evaluate\n",
+    "\n",
+    "metric = evaluate.load(\"wer\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4f32cab6-31f0-4cb9-af4c-40ba0f5fc508",
+   "metadata": {
+    "id": "4f32cab6-31f0-4cb9-af4c-40ba0f5fc508"
+   },
+   "source": [
+    "We then simply have to define a function that takes our model \n",
+    "predictions and returns the WER metric. This function, called\n",
+    "`compute_metrics`, first replaces `-100` with the `pad_token_id`\n",
+    "in the `label_ids` (undoing the step we applied in the \n",
+    "data collator to ignore padded tokens correctly in the loss).\n",
+    "It then decodes the predicted and label ids to strings. Finally,\n",
+    "it computes the WER between the predictions and reference labels. \n",
+    "Here, we have the option of evaluating with the 'normalised' transcriptions \n",
+    "and predictions. We recommend you set this to `True` to benefit from the WER \n",
+    "improvement obtained by normalising the transcriptions."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "23959a70-22d0-4ffe-9fa1-72b61e75bb52",
+   "metadata": {
+    "id": "23959a70-22d0-4ffe-9fa1-72b61e75bb52"
+   },
+   "outputs": [],
+   "source": [
+    "# evaluate with the 'normalised' WER\n",
+    "do_normalize_eval = True\n",
+    "\n",
+    "def compute_metrics(pred):\n",
+    "    pred_ids = pred.predictions\n",
+    "    label_ids = pred.label_ids\n",
+    "\n",
+    "    # replace -100 with the pad_token_id\n",
+    "    label_ids[label_ids == -100] = processor.tokenizer.pad_token_id\n",
+    "\n",
+    "    # we do not want to group tokens when computing the metrics\n",
+    "    pred_str = processor.tokenizer.batch_decode(pred_ids, skip_special_tokens=True)\n",
+    "    label_str = processor.tokenizer.batch_decode(label_ids, skip_special_tokens=True)\n",
+    "\n",
+    "    if do_normalize_eval:\n",
+    "        pred_str = [normalizer(pred) for pred in pred_str]\n",
+    "        label_str = [normalizer(label) for label in label_str]\n",
+    "\n",
+    "    wer = 100 * metric.compute(predictions=pred_str, references=label_str)\n",
+    "\n",
+    "    return {\"wer\": wer}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "daf2a825-6d9f-4a23-b145-c37c0039075b",
+   "metadata": {
+    "id": "daf2a825-6d9f-4a23-b145-c37c0039075b"
+   },
+   "source": [
+    "### Load a Pre-Trained Checkpoint"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "437a97fa-4864-476b-8abc-f28b8166cfa5",
+   "metadata": {
+    "id": "437a97fa-4864-476b-8abc-f28b8166cfa5"
+   },
+   "source": [
+    "Now let's load the pre-trained Whisper `small` checkpoint. Again, this \n",
+    "is trivial through use of 🤗 Transformers!"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "5a10cc4b-07ec-4ebd-ac1d-7c601023594f",
+   "metadata": {
+    "id": "5a10cc4b-07ec-4ebd-ac1d-7c601023594f"
+   },
+   "outputs": [],
+   "source": [
+    "from transformers import WhisperForConditionalGeneration\n",
+    "\n",
+    "model = WhisperForConditionalGeneration.from_pretrained(\"openai/whisper-small\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a15ead5f-2277-4a39-937b-585c2497b2df",
+   "metadata": {
+    "id": "a15ead5f-2277-4a39-937b-585c2497b2df"
+   },
+   "source": [
+    "Override generation arguments - no tokens are forced as decoder outputs (see [`forced_decoder_ids`](https://huggingface.co/docs/transformers/main_classes/text_generation#transformers.generation_utils.GenerationMixin.generate.forced_decoder_ids)), no tokens are suppressed during generation (see [`suppress_tokens`](https://huggingface.co/docs/transformers/main_classes/text_generation#transformers.generation_utils.GenerationMixin.generate.suppress_tokens)). Set `use_cache` to False since we're using gradient checkpointing, and the two are incompatible:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "62038ba3-88ed-4fce-84db-338f50dcd04f",
+   "metadata": {
+    "id": "62038ba3-88ed-4fce-84db-338f50dcd04f"
+   },
+   "outputs": [],
+   "source": [
+    "model.config.forced_decoder_ids = None\n",
+    "model.config.suppress_tokens = []\n",
+    "model.config.use_cache = False"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2178dea4-80ca-47b6-b6ea-ba1915c90c06",
+   "metadata": {
+    "id": "2178dea4-80ca-47b6-b6ea-ba1915c90c06"
+   },
+   "source": [
+    "### Define the Training Configuration"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c21af1e9-0188-4134-ac82-defc7bdcc436",
+   "metadata": {
+    "id": "c21af1e9-0188-4134-ac82-defc7bdcc436"
+   },
+   "source": [
+    "In the final step, we define all the parameters related to training. For more detail on the training arguments, refer to the Seq2SeqTrainingArguments [docs](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.Seq2SeqTrainingArguments)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "0ae3e9af-97b7-4aa0-ae85-20b23b5bcb3a",
+   "metadata": {
+    "id": "0ae3e9af-97b7-4aa0-ae85-20b23b5bcb3a"
+   },
+   "outputs": [],
+   "source": [
+    "from transformers import Seq2SeqTrainingArguments\n",
+    "\n",
+    "training_args = Seq2SeqTrainingArguments(\n",
+    "    output_dir=\"./\",\n",
+    "    per_device_train_batch_size=64,\n",
+    "    gradient_accumulation_steps=1,  # increase by 2x for every 2x decrease in batch size\n",
+    "    learning_rate=1e-5,\n",
+    "    warmup_steps=500,\n",
+    "    max_steps=5000,\n",
+    "    gradient_checkpointing=True,\n",
+    "    fp16=True,\n",
+    "    evaluation_strategy=\"steps\",\n",
+    "    per_device_eval_batch_size=8,\n",
+    "    predict_with_generate=True,\n",
+    "    generation_max_length=225,\n",
+    "    save_steps=1000,\n",
+    "    eval_steps=1000,\n",
+    "    logging_steps=25,\n",
+    "    report_to=[\"tensorboard\"],\n",
+    "    load_best_model_at_end=True,\n",
+    "    metric_for_best_model=\"wer\",\n",
+    "    greater_is_better=False,\n",
+    "    push_to_hub=True,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b3a944d8-3112-4552-82a0-be25988b3857",
+   "metadata": {
+    "id": "b3a944d8-3112-4552-82a0-be25988b3857"
+   },
+   "source": [
+    "**Note**: if one does not want to upload the model checkpoints to the Hub, \n",
+    "set `push_to_hub=False`."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "bac29114-d226-4f54-97cf-8718c9f94e1e",
+   "metadata": {
+    "id": "bac29114-d226-4f54-97cf-8718c9f94e1e"
+   },
+   "source": [
+    "We can forward the training arguments to the 🤗 Trainer along with our model,\n",
+    "dataset, data collator and `compute_metrics` function:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d546d7fe-0543-479a-b708-2ebabec19493",
+   "metadata": {
+    "id": "d546d7fe-0543-479a-b708-2ebabec19493"
+   },
+   "outputs": [],
+   "source": [
+    "from transformers import Seq2SeqTrainer\n",
+    "\n",
+    "trainer = Seq2SeqTrainer(\n",
+    "    args=training_args,\n",
+    "    model=model,\n",
+    "    train_dataset=common_voice[\"train\"],\n",
+    "    eval_dataset=common_voice[\"test\"],\n",
+    "    data_collator=data_collator,\n",
+    "    compute_metrics=compute_metrics,\n",
+    "    tokenizer=processor.feature_extractor,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "uOrRhDGtN5S4",
+   "metadata": {
+    "id": "uOrRhDGtN5S4"
+   },
+   "source": [
+    "We'll save the processor object once before starting training. Since the processor is not trainable, it won't change over the course of training:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "-2zQwMfEOBJq",
+   "metadata": {
+    "id": "-2zQwMfEOBJq"
+   },
+   "outputs": [],
+   "source": [
+    "processor.save_pretrained(training_args.output_dir)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7f404cf9-4345-468c-8196-4bd101d9bd51",
+   "metadata": {
+    "id": "7f404cf9-4345-468c-8196-4bd101d9bd51"
+   },
+   "source": [
+    "### Training"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5e8b8d56-5a70-4f68-bd2e-f0752d0bd112",
+   "metadata": {
+    "id": "5e8b8d56-5a70-4f68-bd2e-f0752d0bd112"
+   },
+   "source": [
+    "Training will take approximately 5-10 hours depending on your GPU. The peak GPU memory for the given training configuration is approximately 36GB. \n",
+    "Depending on your GPU, it is possible that you will encounter a CUDA `\"out-of-memory\"` error when you launch training. \n",
+    "In this case, you can reduce the `per_device_train_batch_size` incrementally by factors of 2 \n",
+    "and employ [`gradient_accumulation_steps`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.Seq2SeqTrainingArguments.gradient_accumulation_steps)\n",
+    "to compensate.\n",
+    "\n",
+    "To launch training, simply execute:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "ee8b7b8e-1c9a-4d77-9137-1778a629e6de",
+   "metadata": {
+    "id": "ee8b7b8e-1c9a-4d77-9137-1778a629e6de"
+   },
+   "outputs": [],
+   "source": [
+    "trainer.train()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "810ced54-7187-4a06-b2fe-ba6dcca94dc3",
+   "metadata": {
+    "id": "810ced54-7187-4a06-b2fe-ba6dcca94dc3"
+   },
+   "source": [
+    "We can label our checkpoint with the `whisper-event` tag on push by setting the appropriate key-word arguments (kwargs):"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c704f91e-241b-48c9-b8e0-f0da396a9663",
+   "metadata": {
+    "id": "c704f91e-241b-48c9-b8e0-f0da396a9663"
+   },
+   "outputs": [],
+   "source": [
+    "kwargs = {\n",
+    "    \"dataset_tags\": \"mozilla-foundation/common_voice_11_0\",\n",
+    "    \"dataset\": \"Common Voice 11.0\",  # a 'pretty' name for the training dataset\n",
+    "    \"language\": \"hi\",\n",
+    "    \"model_name\": \"Whisper Small Hi - Sanchit Gandhi\",  # a 'pretty' name for your model\n",
+    "    \"finetuned_from\": \"openai/whisper-small\",\n",
+    "    \"tasks\": \"automatic-speech-recognition\",\n",
+    "    \"tags\": \"whisper-event\",\n",
+    "}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "090d676a-f944-4297-a938-a40eda0b2b68",
+   "metadata": {
+    "id": "090d676a-f944-4297-a938-a40eda0b2b68"
+   },
+   "source": [
+    "The training results can now be uploaded to the Hub. To do so, execute the `push_to_hub` command and save the preprocessor object we created:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d7030622-caf7-4039-939b-6195cdaa2585",
+   "metadata": {
+    "id": "d7030622-caf7-4039-939b-6195cdaa2585"
+   },
+   "outputs": [],
+   "source": [
+    "trainer.push_to_hub(**kwargs)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ca743fbd-602c-48d4-ba8d-a2fe60af64ba",
+   "metadata": {
+    "id": "ca743fbd-602c-48d4-ba8d-a2fe60af64ba"
+   },
+   "source": [
+    "## Closing Remarks"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7f737783-2870-4e35-aa11-86a42d7d997a",
+   "metadata": {
+    "id": "7f737783-2870-4e35-aa11-86a42d7d997a"
+   },
+   "source": [
+    "In this blog, we covered a step-by-step guide on fine-tuning Whisper for multilingual ASR \n",
+    "using 🤗 Datasets, Transformers and the Hugging Face Hub. For more details on the Whisper model, the Common Voice dataset and the theory behind fine-tuning, refere to the accompanying [blog post](https://huggingface.co/blog/fine-tune-whisper). If you're interested in fine-tuning other \n",
+    "Transformers models, both for English and multilingual ASR, be sure to check out the \n",
+    "examples scripts at [examples/pytorch/speech-recognition](https://github.com/huggingface/transformers/tree/main/examples/pytorch/speech-recognition)."
+   ]
+  }
+ ],
+ "metadata": {
+  "colab": {
+   "include_colab_link": true,
+   "provenance": []
+  },
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

fine-tune-whisper-streaming.ipynb

@@ -0,0 +1,883 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "75b58048-7d14-4fc6-8085-1fc08c81b4a6",
+   "metadata": {},
+   "source": [
+    "# Fine-Tune Whisper With 🤗 Transformers and Streaming Mode"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "fbfa8ad5-4cdc-4512-9058-836cbbf65e1a",
+   "metadata": {},
+   "source": [
+    "In this Colab, we present a step-by-step guide on fine-tuning Whisper with Hugging Face 🤗 Transformers on 400 hours of speech data! Using streaming mode, we'll show how you can train a speech recongition model on any dataset, irrespective of size. With streaming mode, storage requirements are no longer a consideration: you can train a model on whatever dataset you want, even if it's download size exceeds your devices disk space. How can this be possible? It simply seems too good to be true! Well, rest assured it's not 😉 Carry on reading to find out more."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "afe0d503-ae4e-4aa7-9af4-dbcba52db41e",
+   "metadata": {},
+   "source": [
+    "## Introduction"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9ae91ed4-9c3e-4ade-938e-f4c2dcfbfdc0",
+   "metadata": {},
+   "source": [
+    "Speech recognition datasets are large. A typical speech dataset consists of approximately 100 hours of audio-transcription data, requiring upwards of 130GB of storage space for download and preparation. For most ASR researchers, this is already at the upper limit of what is feasible for disk space. So what happens when we want to train on a larger dataset? The full [LibriSpeech](https://huggingface.co/datasets/librispeech_asr) dataset consists of 960 hours of audio data. Kensho's [SPGISpeech](https://huggingface.co/datasets/kensho/spgispeech) contains 5,000 hours of audio data. ML Commons [People's Speech](https://huggingface.co/datasets/MLCommons/peoples_speech) contains **30,000+** hours of audio data! Do we need to bite the bullet and buy additional storage? Or is there a way we can train on all of these datasets with no disk drive requirements?\n",
+    "\n",
+    "When training machine learning systems, we rarely use the entire dataset at once. We typically _batch_ our data into smaller subsets of data, and pass these incrementally through our training pipeline. This is because we train our system on an accelerator device, such as a GPU or TPU, which has a memory limit typically around 16GB. We have to fit our model, optimiser and training data all on the same accelerator device, so we usually have to divide the dataset up into smaller batches and move them from the CPU to the GPU when required.\n",
+    "\n",
+    "Consequently, we don't require the entire dataset to be downloaded at once; we simply need the batch of data that we pass to our model at any one go. We can leverage this principle of partial dataset loading when preparing our dataset: rather than downloading the entire dataset at the start, we can load each piece of data as and when we need it. For each batch, we load the relevant data from a remote server and pass it through the training pipeline. For the next batch, we load the next items and again pass them through the training pipeline. At no point do we have to save data to our disk drive, we simply load them in memory and use them in our pipeline. In doing so, we only ever need as much memory as each individual batch requires.\n",
+    "\n",
+    "This is analogous to downloading a TV show versus streaming it 📺 When we download a TV show, we download the entire video offline and save it to our disk. Compare this to when we stream a TV show. Here, we don't download any part of the video to memory, but iterate over the video file and load each part in real-time as required. It's this same principle that we can apply to our ML training pipeline! We want to iterate over the dataset and load each sample of data as required.\n",
+    "\n",
+    "While the principle of partial dataset loading sounds ideal, it also seems **pretty** difficult to do. Luckily for us, 🤗 Datasets allows us to do this with minimal code changes! We'll make use of the principle of [_streaming_](https://huggingface.co/docs/datasets/stream), depicted graphically in Figure 1. Streaming does exactly this: the data is loaded progressively as we iterate over the dataset, meaning it is only loaded as and when we need it. If you're familiar with 🤗 Transformers and Datasets, the content of this notebook will be very familiar, with some small extensions to support streaming mode."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1c87f76e-47be-4a5d-bc52-7b1c2e9d4f5a",
+   "metadata": {},
+   "source": [
+    "<figure>\n",
+    "<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/datasets/streaming.gif\" alt=\"Trulli\" style=\"width:100%\">\n",
+    "<figcaption align = \"center\"><b>Figure 1:</b> Streaming mode. The dataset is divided into smaller subsets, with subsets loaded progressively as we iterate over the dataset. </figcaption>\n",
+    "</figure>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "21b6316e-8a55-4549-a154-66d3da2ab74a",
+   "metadata": {},
+   "source": [
+    "This notebook provides a guide to fine-tuning on the task of _speech recognition_, which involves learning a\n",
+    "mapping from speech to text. Speech recognition is divided into two categories: English-only or multilingual (all other languages). \n",
+    "This notebook applies to both categories, with pointers for changing between languages and datasets.\n",
+    "\n",
+    "As for our model, we'll fine-tune the Whisper model released in [September 2022](https://openai.com/blog/whisper/) by the authors \n",
+    "Alec Radford et al. from OpenAI. Whisper is an encoder-decoder model pre-trained on 680k hours of labelled audio-transcription data. \n",
+    "It achieves strong performance on many speech recognition and speech translation datasets without fine-tuning. With fine-tuning, \n",
+    "we aim to improve upon these results further, with many SoTA results up for grabs! For a full explanation on the Whisper model, the \n",
+    "reader is advised to read the blog post [Fine-Tune Whisper with 🤗 Transformers](https://huggingface.co/blog/fine-tune-whisper#introduction).\n",
+    "\n",
+    "The Whisper checkpoints come in five configurations of varying model sizes.\n",
+    "The smallest four are trained on either English-only or multilingual data.\n",
+    "The largest checkpoint is multilingual only. All nine of the pre-trained checkpoints \n",
+    "are available on the [Hugging Face Hub](https://huggingface.co/models?search=openai/whisper). The \n",
+    "checkpoints are summarised in the following table with links to the models on the Hub:\n",
+    "\n",
+    "| Size   | Layers | Width | Heads | Parameters | English-only                                         | Multilingual                                      |\n",
+    "|--------|--------|-------|-------|------------|------------------------------------------------------|---------------------------------------------------|\n",
+    "| tiny   | 4      | 384   | 6     | 39 M       | [✓](https://huggingface.co/openai/whisper-tiny.en)   | [✓](https://huggingface.co/openai/whisper-tiny.)  |\n",
+    "| base   | 6      | 512   | 8     | 74 M       | [✓](https://huggingface.co/openai/whisper-base.en)   | [✓](https://huggingface.co/openai/whisper-base)   |\n",
+    "| small  | 12     | 768   | 12    | 244 M      | [✓](https://huggingface.co/openai/whisper-small.en)  | [✓](https://huggingface.co/openai/whisper-small)  |\n",
+    "| medium | 24     | 1024  | 16    | 769 M      | [✓](https://huggingface.co/openai/whisper-medium.en) | [✓](https://huggingface.co/openai/whisper-medium) |\n",
+    "| large  | 32     | 1280  | 20    | 1550 M     | x                                                    | [✓](https://huggingface.co/openai/whisper-large)  |\n",
+    "\n",
+    "When fine-tuning on an English dataset for speech recognition, it is recommeneded to select one of the English-only checkpoints. For any other language, it is recommended to select a multilingual checkpoint.\n",
+    "\n",
+    "For demonstration purposes, we'll fine-tune the multilingual version of the \n",
+    "[`\"small\"`](https://huggingface.co/openai/whisper-small) checkpoint with 244M params (~= 1GB). \n",
+    "As for our data, we'll train and evaluate our system on 400 hours of multilingual speech recognition data\n",
+    "taken from the [Common Voice 11](https://huggingface.co/datasets/mozilla-foundation/common_voice_11_0)\n",
+    "dataset. We'll show how we can train a model on 400 hours of training data using the default disk space \n",
+    "that comes with a standard GPU device or Google Colab."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b219c9dd-39b6-4a95-b2a1-3f547a1e7bc0",
+   "metadata": {},
+   "source": [
+    "## Load Dataset with Streaming"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b17a4763-4381-4157-ae38-b04a8b5f1c43",
+   "metadata": {},
+   "source": [
+    "This is where the magic happens! We'll first write a wrapper function around 🤗 Datasets `load_dataset` method. This function downloads the required splits using streaming mode by forcing `streaming=True` in the `load_dataset` method. Multiple splits can be combined (interleaved) by concatenating them with the \"+\" symbol when specifying the split name, e.g. `split=train+validation` will return a single split with the training and validation splits interleaved together. The function has the same arguments and key-word arguments as 🤗 Datasets `load_dataset` method, so we can use it in exactly the same way!"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "065a8cf7-e54f-4ac3-900e-609c80714fca",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from datasets import interleave_datasets, load_dataset\n",
+    "\n",
+    "def load_streaming_dataset(dataset_name, dataset_config_name, split, **kwargs):\n",
+    "    if \"+\" in split:\n",
+    "        # load multiple splits separated by the `+` symbol *with* streaming mode\n",
+    "        dataset_splits = [load_dataset(dataset_name, dataset_config_name, split=split_name, streaming=True, **kwargs) for split_name in split.split(\"+\")]\n",
+    "        # interleave multiple splits to form one dataset\n",
+    "        interleaved_dataset = interleave_datasets(dataset_splits)\n",
+    "        return interleaved_dataset\n",
+    "    else:\n",
+    "        # load a single split *with* streaming mode\n",
+    "        dataset = load_dataset(dataset_name, dataset_config_name, split=split, streaming=True, **kwargs)\n",
+    "        return dataset"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "674429c5-0ab4-4adf-975b-621bb69eca38",
+   "metadata": {},
+   "source": [
+    "We'll train our system on the Spanish split of [Common Voice 11](https://huggingface.co/datasets/mozilla-foundation/common_voice_11_0). We can see how much training data we have by viewing the [language page](https://commonvoice.mozilla.org/en/datasets) on the Common Voice website. The Spanish split has over 400 hours of labelled training data - that's enourmous! More than we could ever fit on a Google Colab or a standard workstation. But with streaming mode, we'll only download data as and when we need it, making training on this dataset possible!\n",
+    "\n",
+    "Since Spanish is relatively high-resource, we'll only use the `train` split for training and the `test` split for evaluation. If you're training on a low-resource language, such as the Hindi split of Common Voice 11, it's worth combining the `train` and `validation` splits to give a larger training set. You can achieve this by setting: `split=\"train+validation\"` for the training split.\n",
+    "\n",
+    "If you're using a gated dataset, like Common Voice 11, ensure you have accepted the terms of use on the Hugging Face Hub: [mozilla-foundation/common_voice_11_0](https://huggingface.co/datasets/mozilla-foundation/common_voice_11_0). Once you have accepted the terms, you will have full access to the dataset and be able to load the data locally."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a2787582-554f-44ce-9f38-4180a5ed6b44",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from datasets import IterableDatasetDict\n",
+    "\n",
+    "raw_datasets = IterableDatasetDict()\n",
+    "\n",
+    "raw_datasets[\"train\"] = load_streaming_dataset(\"mozilla-foundation/common_voice_11_0\", \"es\", split=\"train\", use_auth_token=True)  # set split=\"train+validation\" for low-resource\n",
+    "raw_datasets[\"test\"] = load_streaming_dataset(\"mozilla-foundation/common_voice_11_0\", \"es\", split=\"test\", use_auth_token=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2d63b2d2-f68a-4d74-b7f1-5127f6d16605",
+   "metadata": {},
+   "source": [
+    "## Prepare Processor and Pre-Process Data"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "601c3099-1026-439e-93e2-5635b3ba5a73",
+   "metadata": {},
+   "source": [
+    "The ASR pipeline can be de-composed into three stages: \n",
+    "1) A feature extractor which pre-processes the raw audio-inputs\n",
+    "2) The model which performs the sequence-to-sequence mapping \n",
+    "3) A tokenizer which post-processes the model outputs to text format\n",
+    "\n",
+    "In 🤗 Transformers, the Whisper model has an associated feature extractor and tokenizer, \n",
+    "called [WhisperFeatureExtractor](https://huggingface.co/docs/transformers/main/model_doc/whisper#transformers.WhisperFeatureExtractor)\n",
+    "and [WhisperTokenizer](https://huggingface.co/docs/transformers/main/model_doc/whisper#transformers.WhisperTokenizer) \n",
+    "respectively. To make our lives simple, these two objects are wrapped under a single class, called the [WhisperProcessor](https://huggingface.co/docs/transformers/model_doc/whisper#transformers.WhisperProcessor). We can call the WhisperProcessor to perform \n",
+    "both the audio pre-processing and the text token post-processing. In doing so, we only need to keep track of two objects during training: \n",
+    "the `processor` and the `model`.\n",
+    "\n",
+    "If using a multilingual checkpoint, you should set the `\"language\"` to your target text language. You should also set the task to `\"transcribe\"` for speech recogntition and `\"translate\"` for speech translation. These arguments modify the behaviour of the tokenizer - they should be set correctly to ensure the target labels are encoded properly. These arguments should be omitted for English-only fine-tuning."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "77d9f0c5-8607-4642-a8ac-c3ab2e223ea6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from transformers import WhisperProcessor\n",
+    "\n",
+    "processor = WhisperProcessor.from_pretrained(\"openai/whisper-small\", language=\"Spanish\", task=\"transcribe\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "381acd09-0b0f-4d04-9eb3-f028ac0e5f2c",
+   "metadata": {},
+   "source": [
+    "### Pre-Process Data"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "bf10cd3e-924e-44fc-8790-46e413de7b3d",
+   "metadata": {},
+   "source": [
+    "Let's have a look at the dataset features. Pay particular attention to the `\"audio\"` column - this details the sampling rate of our audio inputs:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "ab5a13b4-9bd4-4aa0-aef2-b3de9b762988",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "raw_datasets[\"train\"].features"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5a679f05-063d-41b3-9b58-4fc9c6ccf4fd",
+   "metadata": {},
+   "source": [
+    "Since our input audio is sampled at 48kHz, we need to _downsample_ it to\n",
+    "16kHz prior to passing it to the Whisper feature extractor, 16kHz being the sampling rate expected by the Whisper model. \n",
+    "\n",
+    "We'll set the audio inputs to the correct sampling rate using dataset's \n",
+    "[`cast_column`](https://huggingface.co/docs/datasets/package_reference/main_classes.html?highlight=cast_column#datasets.DatasetDict.cast_column)\n",
+    "method. This operation does not change the audio in-place, \n",
+    "but rather signals to `datasets` to resample audio samples _on the fly_ the \n",
+    "first time that they are loaded:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "3ab6a724-3d1e-478b-a9e9-d2f85feb6c39",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from datasets import Audio\n",
+    "\n",
+    "raw_datasets = raw_datasets.cast_column(\"audio\", Audio(sampling_rate=16000))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "161322c2-94f3-4d26-9e1d-d9d5202ca3cf",
+   "metadata": {},
+   "source": [
+    "We'll define our pre-processing strategy. We advise that you **do not** lower-case the transcriptions or remove punctuation unless mixing different datasets. This will enable you to fine-tune Whisper models that can predict punctuation and casing. Later, you will see how we can evaluate the predictions without punctuation or casing, so that the models benefit from the WER improvement obtained by normalising the transcriptions while still predicting fully formatted transcriptions."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d041650e-1c48-4439-87b3-5b6f4a514107",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from transformers.models.whisper.english_normalizer import BasicTextNormalizer\n",
+    "\n",
+    "do_lower_case = False\n",
+    "do_remove_punctuation = False\n",
+    "\n",
+    "normalizer = BasicTextNormalizer()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "bfaa935b-a11d-497c-88c1-0c4d1bb3247b",
+   "metadata": {},
+   "source": [
+    "Now we can write a function to prepare our data ready for the model:\n",
+    "1. We load and resample the audio data by calling `batch[\"audio\"]`. As explained above, 🤗 Datasets performs any necessary resampling operations on the fly.\n",
+    "2. We use the feature extractor to compute the log-Mel spectrogram input features from our 1-dimensional audio array.\n",
+    "3. We perform any optional pre-processing (lower-case or remove punctuation).\n",
+    "4. We encode the transcriptions to label ids through the use of the tokenizer."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c085911c-a10a-41ef-8874-306e0503e9bb",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def prepare_dataset(batch):\n",
+    "    # load and (possibly) resample audio data to 16kHz\n",
+    "    audio = batch[\"audio\"]\n",
+    "\n",
+    "    # compute log-Mel input features from input audio array \n",
+    "    batch[\"input_features\"] = processor.feature_extractor(audio[\"array\"], sampling_rate=audio[\"sampling_rate\"]).input_features[0]\n",
+    "    # compute input length of audio sample in seconds\n",
+    "    batch[\"input_length\"] = len(audio[\"array\"]) / audio[\"sampling_rate\"]\n",
+    "    \n",
+    "    # optional pre-processing steps\n",
+    "    transcription = batch[\"sentence\"]\n",
+    "    if do_lower_case:\n",
+    "        transcription = transcription.lower()\n",
+    "    if do_remove_punctuation:\n",
+    "        transcription = normalizer(transcription).strip()\n",
+    "    \n",
+    "    # encode target text to label ids\n",
+    "    batch[\"labels\"] = processor.tokenizer(transcription).input_ids\n",
+    "    return batch"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "70b319fb-2439-4ef6-a70d-a47bf41c4a13",
+   "metadata": {},
+   "source": [
+    "We can apply the data preparation function to all of our training examples using 🤗 Datasets' `.map` method. We'll remove all of the columns from the raw training data, leaving just the `input_features` and `labels` defined in the `prepare_dataset` function:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a37a7cdb-9013-427f-8de9-6a8d0e9dc684",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "vectorized_datasets = raw_datasets.map(prepare_dataset, remove_columns=list(next(iter(raw_datasets.values())).features)).with_format(\"torch\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3d59b37e-4950-47ec-9e3e-2cf2ec7fc750",
+   "metadata": {},
+   "source": [
+    "We can now define how we shuffle the data in the train split. The size of the subset we load is set by the variable `buffer_size`. You can increase or decrease this depending on your memory constraints. In this example, the `buffer_size` is set to 500, meaning 500 samples are loaded before shuffling across the subset. The larger we set this value, the closer to True offline shuffling. The `seed` is set for reproducibility:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1b145699-acfc-4b1d-93a2-a2ad3d62674c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "vectorized_datasets[\"train\"] = vectorized_datasets[\"train\"].shuffle(\n",
+    "    buffer_size=500,\n",
+    "    seed=0,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "666b9ef0-7909-4e1e-a419-87604d233e29",
+   "metadata": {},
+   "source": [
+    "Finally, we filter any training data with audio samples longer than 30s. These samples would otherwise be truncated by the Whisper feature-extractor which could affect the stability of training. We define a function that returns `True` for samples that are less than 30s, and `False` for those that are longer:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "01cb25ef-4bb0-4325-9461-f59198acadf6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "max_input_length = 30.0\n",
+    "\n",
+    "def is_audio_in_length_range(length):\n",
+    "    return length < max_input_length"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "28e37ac3-b1c5-465b-8586-7cfd8d76b0f1",
+   "metadata": {},
+   "source": [
+    "We apply our filter function to all samples of our training dataset through 🤗 Datasets' `.filter` method:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "333f7f6e-6053-4d3b-8924-c733c79b82ac",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "vectorized_datasets[\"train\"] = vectorized_datasets[\"train\"].filter(\n",
+    "    is_audio_in_length_range,\n",
+    "    input_columns=[\"input_length\"],\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "263a5a58-0239-4a25-b0df-c625fc9c5810",
+   "metadata": {},
+   "source": [
+    "## Training and Evaluation"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a693e768-c5a6-453f-89a1-b601dcf7daf7",
+   "metadata": {},
+   "source": [
+    "Now that we've prepared our data, we're ready to dive into the training pipeline. \n",
+    "The [🤗 Trainer](https://huggingface.co/transformers/master/main_classes/trainer.html?highlight=trainer)\n",
+    "will do much of the heavy lifting for us. All we have to do is:\n",
+    "\n",
+    "- Define a data collator: the data collator takes our pre-processed data and prepares PyTorch tensors ready for the model.\n",
+    "\n",
+    "- Evaluation metrics: during evaluation, we want to evaluate the model using the [word error rate (WER)](https://huggingface.co/metrics/wer) metric. We need to define a `compute_metrics` function that handles this computation.\n",
+    "\n",
+    "- Load a pre-trained checkpoint: we need to load a pre-trained checkpoint and configure it correctly for training.\n",
+    "\n",
+    "- Define the training configuration: this will be used by the 🤗 Trainer to define the training schedule."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8d230e6d-624c-400a-bbf5-fa660881df25",
+   "metadata": {},
+   "source": [
+    "### Define a Data Collator"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "04def221-0637-4a69-b242-d3f0c1d0ee78",
+   "metadata": {},
+   "source": [
+    "The data collator for a sequence-to-sequence speech model is unique in the sense that it \n",
+    "treats the `input_features` and `labels` independently: the  `input_features` must be \n",
+    "handled by the feature extractor and the `labels` by the tokenizer.\n",
+    "\n",
+    "The `input_features` are already padded to 30s and converted to a log-Mel spectrogram \n",
+    "of fixed dimension by action of the feature extractor, so all we have to do is convert the `input_features`\n",
+    "to batched PyTorch tensors. We do this using the feature extractor's `.pad` method with `return_tensors=pt`.\n",
+    "\n",
+    "The `labels` on the other hand are un-padded. We first pad the sequences\n",
+    "to the maximum length in the batch using the tokenizer's `.pad` method. The padding tokens \n",
+    "are then replaced by `-100` so that these tokens are **not** taken into account when \n",
+    "computing the loss. We then cut the BOS token from the start of the label sequence as we \n",
+    "append it later during training.\n",
+    "\n",
+    "We can leverage the `WhisperProcessor` we defined earlier to perform both the \n",
+    "feature extractor and the tokenizer operations:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8326221e-ec13-4731-bb4e-51e5fc1486c5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import torch\n",
+    "\n",
+    "from dataclasses import dataclass\n",
+    "from typing import Any, Dict, List, Union\n",
+    "\n",
+    "@dataclass\n",
+    "class DataCollatorSpeechSeq2SeqWithPadding:\n",
+    "    processor: Any\n",
+    "\n",
+    "    def __call__(self, features: List[Dict[str, Union[List[int], torch.Tensor]]]) -> Dict[str, torch.Tensor]:\n",
+    "        # split inputs and labels since they have to be of different lengths and need different padding methods\n",
+    "        # first treat the audio inputs by simply returning torch tensors\n",
+    "        input_features = [{\"input_features\": feature[\"input_features\"]} for feature in features]\n",
+    "        batch = self.processor.feature_extractor.pad(input_features, return_tensors=\"pt\")\n",
+    "\n",
+    "        # get the tokenized label sequences\n",
+    "        label_features = [{\"input_ids\": feature[\"labels\"]} for feature in features]\n",
+    "        # pad the labels to max length\n",
+    "        labels_batch = self.processor.tokenizer.pad(label_features, return_tensors=\"pt\")\n",
+    "\n",
+    "        # replace padding with -100 to ignore loss correctly\n",
+    "        labels = labels_batch[\"input_ids\"].masked_fill(labels_batch.attention_mask.ne(1), -100)\n",
+    "\n",
+    "        # if bos token is appended in previous tokenization step,\n",
+    "        # cut bos token here as it's append later anyways\n",
+    "        if (labels[:, 0] == self.processor.tokenizer.bos_token_id).all().cpu().item():\n",
+    "            labels = labels[:, 1:]\n",
+    "\n",
+    "        batch[\"labels\"] = labels\n",
+    "\n",
+    "        return batch"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3cae7dbf-8a50-456e-a3a8-7fd005390f86",
+   "metadata": {},
+   "source": [
+    "Let's initialise the data collator we've just defined:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "fc834702-c0d3-4a96-b101-7b87be32bf42",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "data_collator = DataCollatorSpeechSeq2SeqWithPadding(processor=processor)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d62bb2ab-750a-45e7-82e9-61d6f4805698",
+   "metadata": {},
+   "source": [
+    "### Evaluation Metrics"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "66fee1a7-a44c-461e-b047-c3917221572e",
+   "metadata": {},
+   "source": [
+    "We'll use the word error rate (WER) metric, the 'de-facto' metric for assessing \n",
+    "ASR systems. For more information, refer to the WER [docs](https://huggingface.co/metrics/wer). We'll load the WER metric from 🤗 Evaluate:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b22b4011-f31f-4b57-b684-c52332f92890",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import evaluate\n",
+    "\n",
+    "metric = evaluate.load(\"wer\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "509f96d7-3f11-4f37-add9-f74a0c44f3fc",
+   "metadata": {},
+   "source": [
+    "We then simply have to define a function that takes our model \n",
+    "predictions and returns the WER metric. This function, called\n",
+    "`compute_metrics`, first replaces `-100` with the `pad_token_id`\n",
+    "in the `label_ids` (undoing the step we applied in the \n",
+    "data collator to ignore padded tokens correctly in the loss).\n",
+    "It then decodes the predicted and label ids to strings. Finally,\n",
+    "it computes the WER between the predictions and reference labels. \n",
+    "Here, we have the option of evaluating with the 'normalised' transcriptions \n",
+    "and predictions. We recommend you set this to `True` to benefit from the WER \n",
+    "improvement obtained by normalising the transcriptions."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a11d1bfc-9e28-460f-a287-72d8f7bc1acb",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# evaluate with the 'normalised' WER\n",
+    "do_normalize_eval = True\n",
+    "\n",
+    "def compute_metrics(pred):\n",
+    "    pred_ids = pred.predictions\n",
+    "    label_ids = pred.label_ids\n",
+    "\n",
+    "    # replace -100 with the pad_token_id\n",
+    "    label_ids[label_ids == -100] = processor.tokenizer.pad_token_id\n",
+    "\n",
+    "    # we do not want to group tokens when computing the metrics\n",
+    "    pred_str = processor.tokenizer.batch_decode(pred_ids, skip_special_tokens=True)\n",
+    "    label_str = processor.tokenizer.batch_decode(label_ids, skip_special_tokens=True)\n",
+    "\n",
+    "    if do_normalize_eval:\n",
+    "        pred_str = [normalizer(pred) for pred in pred_str]\n",
+    "        label_str = [normalizer(label) for label in label_str]\n",
+    "        # filtering step to only evaluate the samples that correspond to non-zero references:\n",
+    "        pred_str = [pred_str[i] for i in range(len(pred_str)) if len(label_str[i]) > 0]\n",
+    "        label_str = [label_str[i] for i in range(len(label_str)) if len(label_str[i]) > 0]\n",
+    "\n",
+    "    wer = 100 * metric.compute(predictions=pred_str, references=label_str)\n",
+    "\n",
+    "    return {\"wer\": wer}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "daf2a825-6d9f-4a23-b145-c37c0039075b",
+   "metadata": {},
+   "source": [
+    "### Load a Pre-Trained Checkpoint"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "437a97fa-4864-476b-8abc-f28b8166cfa5",
+   "metadata": {},
+   "source": [
+    "Now let's load the pre-trained Whisper `small` checkpoint. Again, this \n",
+    "is trivial through use of 🤗 Transformers!"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "5a10cc4b-07ec-4ebd-ac1d-7c601023594f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from transformers import WhisperForConditionalGeneration\n",
+    "\n",
+    "model = WhisperForConditionalGeneration.from_pretrained(\"openai/whisper-small\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a15ead5f-2277-4a39-937b-585c2497b2df",
+   "metadata": {},
+   "source": [
+    "Override generation arguments - no tokens are forced as decoder outputs (see [`forced_decoder_ids`](https://huggingface.co/docs/transformers/main_classes/text_generation#transformers.generation_utils.GenerationMixin.generate.forced_decoder_ids)), no tokens are suppressed during generation (see [`suppress_tokens`](https://huggingface.co/docs/transformers/main_classes/text_generation#transformers.generation_utils.GenerationMixin.generate.suppress_tokens)). Set `use_cache` to False since we're using gradient checkpointing, and the two are incompatible:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "62038ba3-88ed-4fce-84db-338f50dcd04f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "model.config.forced_decoder_ids = None\n",
+    "model.config.suppress_tokens = []\n",
+    "model.config.use_cache = False"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2178dea4-80ca-47b6-b6ea-ba1915c90c06",
+   "metadata": {},
+   "source": [
+    "### Define the Training Configuration"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c21af1e9-0188-4134-ac82-defc7bdcc436",
+   "metadata": {},
+   "source": [
+    "In the final step, we define all the parameters related to training. Here, you can set the `max_steps` to train for longer. For more detail on the training arguments, refer to the Seq2SeqTrainingArguments [docs](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.Seq2SeqTrainingArguments)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "0ae3e9af-97b7-4aa0-ae85-20b23b5bcb3a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from transformers import Seq2SeqTrainingArguments\n",
+    "\n",
+    "training_args = Seq2SeqTrainingArguments(\n",
+    "    output_dir=\"./\",\n",
+    "    per_device_train_batch_size=64,\n",
+    "    gradient_accumulation_steps=1,  # increase by 2x for every 2x decrease in batch size\n",
+    "    learning_rate=1e-5,\n",
+    "    warmup_steps=500,\n",
+    "    max_steps=5000,\n",
+    "    gradient_checkpointing=True,\n",
+    "    fp16=True,\n",
+    "    evaluation_strategy=\"steps\",\n",
+    "    per_device_eval_batch_size=8,\n",
+    "    predict_with_generate=True,\n",
+    "    generation_max_length=225,\n",
+    "    save_steps=1000,\n",
+    "    eval_steps=1000,\n",
+    "    logging_steps=25,\n",
+    "    report_to=[\"tensorboard\"],\n",
+    "    load_best_model_at_end=True,\n",
+    "    metric_for_best_model=\"wer\",\n",
+    "    greater_is_better=False,\n",
+    "    push_to_hub=True,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b3a944d8-3112-4552-82a0-be25988b3857",
+   "metadata": {},
+   "source": [
+    "**Note**: if one does not want to upload the model checkpoints to the Hub, \n",
+    "set `push_to_hub=False`."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "393c883e-3e50-492c-bd58-f51dbf15ee56",
+   "metadata": {},
+   "source": [
+    "We then define a custom [Callback](https://huggingface.co/docs/transformers/main_classes/callback) that is called by the 🤗 Trainer on the end of each epoch. The Callback reinitialises and reshuffles the streaming dataset at the beginning of each new epoch - this gives different shuffling across our subsets for every epoch."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "3ac16b62-b3c0-4c68-8f3d-9ecf471534b2",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from transformers import TrainerCallback\n",
+    "from transformers.trainer_pt_utils import IterableDatasetShard\n",
+    "from torch.utils.data import IterableDataset\n",
+    "\n",
+    "# trainer callback to reinitialise and reshuffle the streamable datasets at the beginning of each epoch\n",
+    "class ShuffleCallback(TrainerCallback):\n",
+    "    def on_epoch_begin(self, args, state, control, train_dataloader, **kwargs):\n",
+    "        if isinstance(train_dataloader.dataset, IterableDatasetShard):\n",
+    "            pass  # set_epoch() is handled by the Trainer\n",
+    "        elif isinstance(train_dataloader.dataset, IterableDataset):\n",
+    "            train_dataloader.dataset.set_epoch(train_dataloader.dataset._epoch + 1)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "bac29114-d226-4f54-97cf-8718c9f94e1e",
+   "metadata": {},
+   "source": [
+    "We can forward the training arguments to the 🤗 Trainer along with our model,\n",
+    "dataset, data collator, `compute_metrics` function and custom callback:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d546d7fe-0543-479a-b708-2ebabec19493",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from transformers import Seq2SeqTrainer\n",
+    "\n",
+    "trainer = Seq2SeqTrainer(\n",
+    "    args=training_args,\n",
+    "    model=model,\n",
+    "    train_dataset=vectorized_datasets[\"train\"],\n",
+    "    eval_dataset=vectorized_datasets[\"test\"],\n",
+    "    data_collator=data_collator,\n",
+    "    compute_metrics=compute_metrics,\n",
+    "    tokenizer=processor,\n",
+    "    callbacks=[ShuffleCallback()],\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "67ab88c3-7091-4e51-8ad5-f5cacbe18449",
+   "metadata": {},
+   "source": [
+    "We'll save the model and processor to the output directory before training:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a1ccb9ed-cbc8-4419-91c0-651e9424b672",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "model.save_pretrained(training_args.output_dir)\n",
+    "processor.save_pretrained(training_args.output_dir)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7f404cf9-4345-468c-8196-4bd101d9bd51",
+   "metadata": {},
+   "source": [
+    "### Training"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5e8b8d56-5a70-4f68-bd2e-f0752d0bd112",
+   "metadata": {},
+   "source": [
+    "Training will take approximately 5-10 hours depending on your GPU. The peak GPU memory for the given training configuration is approximately 36GB. \n",
+    "Depending on your GPU, it is possible that you will encounter a CUDA `\"out-of-memory\"` error when you launch training. \n",
+    "In this case, you can reduce the `per_device_train_batch_size` incrementally by factors of 2 \n",
+    "and employ [`gradient_accumulation_steps`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.Seq2SeqTrainingArguments.gradient_accumulation_steps)\n",
+    "to compensate.\n",
+    "\n",
+    "To launch training, simply execute:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "ee8b7b8e-1c9a-4d77-9137-1778a629e6de",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "trainer.train()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "747c6a6e",
+   "metadata": {
+    "pycharm": {
+     "name": "#%% md\n"
+    }
+   },
+   "source": [
+    "(note that training may take some time to commence as we load the first training data samples with streaming mode)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "810ced54-7187-4a06-b2fe-ba6dcca94dc3",
+   "metadata": {},
+   "source": [
+    "We can label our checkpoint with the `whisper-event` tag on push by setting the appropriate key-word arguments (kwargs):"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6dd0e310-9b07-4133-ac14-2ed2d7524e22",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "kwargs = {\n",
+    "    \"dataset_tags\": \"mozilla-foundation/common_voice_11_0\",\n",
+    "    \"dataset\": \"Common Voice 11.0\",  # a 'pretty' name for the training dataset\n",
+    "    \"language\": \"es\",\n",
+    "    \"model_name\": \"Whisper Small Es - Sanchit Gandhi\",  # a 'pretty' name for your model\n",
+    "    \"finetuned_from\": \"openai/whisper-small\",\n",
+    "    \"tasks\": \"automatic-speech-recognition\",\n",
+    "    \"tags\": \"whisper-event\",\n",
+    "}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "090d676a-f944-4297-a938-a40eda0b2b68",
+   "metadata": {},
+   "source": [
+    "The training results can now be uploaded to the Hub. To do so, execute the `push_to_hub` command:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "95737cda-c5dd-4887-a4d0-dfcb0d61d977",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "trainer.push_to_hub(**kwargs)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

interleave_streaming_datasets.ipynb

@@ -0,0 +1,358 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "6a5c0357",
+   "metadata": {
+    "collapsed": false,
+    "jupyter": {
+     "outputs_hidden": false
+    },
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "# Ensure datasets is installed from main. Uncomment the following line if you face issues running this script:\n",
+    "# !pip install git+https://github.com/huggingface/datasets"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "794aaced",
+   "metadata": {
+    "collapsed": false,
+    "jupyter": {
+     "outputs_hidden": false
+    },
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "from datasets import Audio, interleave_datasets, IterableDataset, load_dataset\n",
+    "from typing import List, Optional"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f210ca9a-486b-46a2-a675-2526a9bd83f5",
+   "metadata": {},
+   "source": [
+    "### Define the dataset attributes"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "fc07293f-3ba4-4e89-a4ca-8e39409a8373",
+   "metadata": {},
+   "source": [
+    "In this example, we'll show to combine the Common Voice 11, VoxPopuli, Mulitlingual LibriSpeech and FLEURS datasets for Spanish, giving a training corpus equal to the sum of the individual datasets. This is particularly beneficial in low-resource settings, where any one of the datasets alone might have insufficient data to train a model.\n",
+    "\n",
+    "We need to specify the dataset names on the Hub, the corresponding configs and finally the text column names for the transcriptions:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "c53344f3-c315-430a-a2f3-57aea6bb0e17",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "dataset_names = [\"mozilla-foundation/common_voice_11_0\", \"facebook/voxpopuli\", \"facebook/multilingual_librispeech\", \"google/fleurs\"]\n",
+    "dataset_config_names = [\"es\", \"es\", \"spanish\", \"es_419\"]\n",
+    "text_column_names = [\"sentence\", \"normalized_text\", \"text\", \"transcription\"]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "215541f6-ee1c-4104-b43c-fa3f7fce0494",
+   "metadata": {},
+   "source": [
+    "### Define the merging function"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b722a48b-c576-4a63-b2a2-3c264890a75f",
+   "metadata": {},
+   "source": [
+    "We define a function, `load_multiple_streaming_datasets`, that takes as argument a list of datasets, configs, splits (optional) and text column names (optional). It sets them to a specified sampling rate and interleaves them together, giving one merged dataset. This is all \n",
+    "done in _streaming mode_: as we iterate over the merged dataset we load samples one-by-one on the fly. No data is\n",
+    "saved to disk.\n",
+    "\n",
+    "We can also specify our strategy for interleaving datasets. The default strategy, `all_exhausted` is an oversampling \n",
+    "strategy. In this case, the dataset construction is stopped as soon as every samples in every dataset \n",
+    "has been added at least once. In practice, it means that if a dataset is exhausted, it will return to the \n",
+    "beginning of this dataset until the stop criterion has been reached. You can specify `stopping_strategy=first_exhausted` \n",
+    "for a subsampling strategy, i.e the dataset construction is stopped as soon one of the dataset runs out of samples. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "61eb4cb1-ee27-4270-a474-1bb33e1df65f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def load_multiple_streaming_datasets(\n",
+    "    dataset_names: List,\n",
+    "    dataset_config_names: List,\n",
+    "    splits: Optional[List] = None,\n",
+    "    text_column_names: Optional[List] = None,\n",
+    "    sampling_rate: Optional[int] = 16000,\n",
+    "    stopping_strategy: Optional[str] = \"all_exhausted\",\n",
+    "    **kwargs\n",
+    ") -> IterableDataset:\n",
+    "\n",
+    "    if len(dataset_names) != len(dataset_config_names):\n",
+    "        raise ValueError(\n",
+    "            f\"Ensure one config is passed for each dataset, got {len(dataset_names)} datasets and\"\n",
+    "            f\" {len(dataset_config_names)} configs.\"\n",
+    "        )\n",
+    "\n",
+    "    if splits is not None and len(splits) != len(dataset_names):\n",
+    "        raise ValueError(\n",
+    "            f\"Ensure one split is passed for each dataset, got {len(dataset_names)} datasets and {len(splits)} splits.\"\n",
+    "        )\n",
+    "\n",
+    "    if text_column_names is not None and len(text_column_names) != len(dataset_names):\n",
+    "        raise ValueError(\n",
+    "            f\"Ensure one text column name is passed for each dataset, got {len(dataset_names)} datasets and\"\n",
+    "            f\" {len(text_column_names)} text column names.\"\n",
+    "        )\n",
+    "\n",
+    "    splits = splits if splits is not None else [\"train\" for i in range(len(dataset_names))]\n",
+    "    text_column_names = (\n",
+    "        text_column_names if text_column_names is not None else [\"text\" for i in range(len(dataset_names))]\n",
+    "    )\n",
+    "\n",
+    "    all_datasets = []\n",
+    "    # iterate over the datasets we want to interleave\n",
+    "    for i, dataset_name in enumerate(dataset_names):\n",
+    "        dataset = load_dataset(dataset_name, dataset_config_names[i], split=splits[i], streaming=True, **kwargs)\n",
+    "        # resample to specified sampling rate\n",
+    "        dataset = dataset.cast_column(\"audio\", Audio(sampling_rate))\n",
+    "        #  normalise columns to [\"audio\", \"sentence\"]\n",
+    "        if text_column_names[i] != \"sentence\":\n",
+    "            dataset = dataset.rename_column(text_column_names[i], \"sentence\")\n",
+    "        dataset = dataset.remove_columns(set(dataset.features.keys()) - set([\"audio\", \"sentence\"]))\n",
+    "        all_datasets.append(dataset)\n",
+    "\n",
+    "    interleaved_dataset = interleave_datasets(all_datasets, stopping_strategy=stopping_strategy)\n",
+    "    return interleaved_dataset"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "29bc228b-ce9b-4cee-9092-1223ddfa51ad",
+   "metadata": {},
+   "source": [
+    "Let's apply this function to load and merge our four datasets:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "8ae90f83-4ecd-46a3-98be-bd75706e0d88",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ds = load_multiple_streaming_datasets(dataset_names, dataset_config_names=dataset_config_names, text_column_names=text_column_names, use_auth_token=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6056a693-1fb0-45f4-ad43-be5f1812c1a5",
+   "metadata": {},
+   "source": [
+    "### Iterate over the dataset"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7ffe011f-f905-4027-ab67-5c9c3b2b5ac0",
+   "metadata": {},
+   "source": [
+    "We iterate over the dataset, loading and merging samples on the fly. Let's print the transcriptions for the first 10 samples of our merged dataset:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "75b3355a-3c06-4d23-af43-2b93b1ad70b2",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Reading metadata...: 230467it [00:41, 5545.80it/s]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "0 ¿ Qué tal a tres de cinco ?\n",
+      "1 y desde luego esa razón no puede tener que ver con la explicación surrealista que hemos escuchado más de una vez de que se trata de una conspiración izquierdista.\n",
+      "2 para exclamar con voz de acción de gracias y para contar todas tus maravillas jehová la habitación de tu casa he amado y el lugar del tabernáculo de tu gloria no juntes con los pecadores mi alma ni con los hombres de sangres mi vida\n",
+      "3 el uso de internet y de la red informática mundial permite que los estudiantes tengan acceso a la información en todo momento\n",
+      "4 vamos , quiero decir , que no soy de citas especiales .\n",
+      "5 si bien esta lista no es perfecta sí que resulta necesario que las entidades financieras refuercen sus controles.\n",
+      "6 oye oh jehová mi voz con que á ti clamo y ten misericordia de mí respóndeme mi corazón ha dicho de ti buscad mi rostro tu rostro buscaré oh jehová\n",
+      "7 los deportes de nieve en descenso como el esquí y la tablanieve son disciplinas populares que consisten en deslizarse con esquís o una tabla fijada a los pies sobre un terreno nevado\n",
+      "8 fray Lope , en aquel momento , colmaba otro vaso igual :\n",
+      "9 señora presidenta la competitividad es importante pero no puede ser el único criterio.\n"
+     ]
+    }
+   ],
+   "source": [
+    "for i, sample in enumerate(ds):\n",
+    "    print(i, sample[\"sentence\"])\n",
+    "    if i == 9:\n",
+    "        break"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "42d5ad08-b20e-4cba-a1a9-909fdbf030d4",
+   "metadata": {},
+   "source": [
+    "We can see that the transcriptions take several different formats. Those from Common Voice 11 are cased and punctuated. Those from VoxPopuli are punctuated only. Those from Multilingual LibriSpeech and FLEURS are neither cased not punctuated. We need to normalise the transcriptions to a uniform format before training our model. \n",
+    "\n",
+    "The following code cell is lifted from the Whisper training notebook: https://github.com/huggingface/community-events/blob/main/whisper-fine-tuning-event/fine-tune-whisper-streaming.ipynb"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "ed20e9cd-31c2-44cb-872b-333378a92fd1",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/Users/sanchitgandhi/venv/lib/python3.8/site-packages/jax/_src/lib/__init__.py:33: UserWarning: JAX on Mac ARM machines is experimental and minimally tested. Please see https://github.com/google/jax/issues/5501 in the event of problems.\n",
+      "  warnings.warn(\"JAX on Mac ARM machines is experimental and minimally tested. \"\n"
+     ]
+    }
+   ],
+   "source": [
+    "from transformers.models.whisper.english_normalizer import BasicTextNormalizer\n",
+    "\n",
+    "do_lower_case = True\n",
+    "do_remove_punctuation = True\n",
+    "\n",
+    "normalizer = BasicTextNormalizer()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "01d13029-c24f-4a51-aff2-9251a2ceb4ce",
+   "metadata": {},
+   "source": [
+    "Now we define a function to normalise our transcriptions:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "26e42417-4bd2-46f8-914e-3a6f9f3471ac",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def normalize_transcriptions(batch):\n",
+    "    # optional pre-processing steps\n",
+    "    transcription = batch[\"sentence\"]\n",
+    "    if do_lower_case:\n",
+    "        transcription = transcription.lower()\n",
+    "    if do_remove_punctuation:\n",
+    "        transcription = normalizer(transcription).strip()\n",
+    "    batch[\"sentence\"] = transcription\n",
+    "    return batch"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3b1c67fe-be4b-4ee5-9a1f-0d444f2b5c62",
+   "metadata": {},
+   "source": [
+    "Let's apply the data pre-processing steps to our dataset and view the first 10 samples again:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "0babac71-9157-4d0f-a8a8-184547bdf501",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Reading metadata...: 230467it [00:32, 6984.59it/s] \n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "0  qué tal a tres de cinco \n",
+      "1 y desde luego esa razón no puede tener que ver con la explicación surrealista que hemos escuchado más de una vez de que se trata de una conspiración izquierdista \n",
+      "2 para exclamar con voz de acción de gracias y para contar todas tus maravillas jehová la habitación de tu casa he amado y el lugar del tabernáculo de tu gloria no juntes con los pecadores mi alma ni con los hombres de sangres mi vida\n",
+      "3 el uso de internet y de la red informática mundial permite que los estudiantes tengan acceso a la información en todo momento\n",
+      "4 vamos quiero decir que no soy de citas especiales \n",
+      "5 si bien esta lista no es perfecta sí que resulta necesario que las entidades financieras refuercen sus controles \n",
+      "6 oye oh jehová mi voz con que á ti clamo y ten misericordia de mí respóndeme mi corazón ha dicho de ti buscad mi rostro tu rostro buscaré oh jehová\n",
+      "7 los deportes de nieve en descenso como el esquí y la tablanieve son disciplinas populares que consisten en deslizarse con esquís o una tabla fijada a los pies sobre un terreno nevado\n",
+      "8 fray lope en aquel momento colmaba otro vaso igual \n",
+      "9 señora presidenta la competitividad es importante pero no puede ser el único criterio \n"
+     ]
+    }
+   ],
+   "source": [
+    "ds = ds.map(normalize_transcriptions)\n",
+    "\n",
+    "for i, sample in enumerate(ds):\n",
+    "    print(i, sample[\"sentence\"])\n",
+    "    if i == 9:\n",
+    "        break"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d135627a-a7aa-458c-94b8-57ddeae74a72",
+   "metadata": {},
+   "source": [
+    "This time the transcriptions are in a consistent format. We can use this data to fine-tune our Whisper model. Note that since we've removed punctuation and casing, the Whisper model won't learn to predict these features."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

normalizer.json

@@ -0,0 +1,1742 @@
+{
+  "accessorise": "accessorize",
+  "accessorised": "accessorized",
+  "accessorises": "accessorizes",
+  "accessorising": "accessorizing",
+  "acclimatisation": "acclimatization",
+  "acclimatise": "acclimatize",
+  "acclimatised": "acclimatized",
+  "acclimatises": "acclimatizes",
+  "acclimatising": "acclimatizing",
+  "accoutrements": "accouterments",
+  "aeon": "eon",
+  "aeons": "eons",
+  "aerogramme": "aerogram",
+  "aerogrammes": "aerograms",
+  "aeroplane": "airplane",
+  "aeroplanes": "airplanes",
+  "aesthete": "esthete",
+  "aesthetes": "esthetes",
+  "aesthetic": "esthetic",
+  "aesthetically": "esthetically",
+  "aesthetics": "esthetics",
+  "aetiology": "etiology",
+  "ageing": "aging",
+  "aggrandisement": "aggrandizement",
+  "agonise": "agonize",
+  "agonised": "agonized",
+  "agonises": "agonizes",
+  "agonising": "agonizing",
+  "agonisingly": "agonizingly",
+  "almanack": "almanac",
+  "almanacks": "almanacs",
+  "aluminium": "aluminum",
+  "amortisable": "amortizable",
+  "amortisation": "amortization",
+  "amortisations": "amortizations",
+  "amortise": "amortize",
+  "amortised": "amortized",
+  "amortises": "amortizes",
+  "amortising": "amortizing",
+  "amphitheatre": "amphitheater",
+  "amphitheatres": "amphitheaters",
+  "anaemia": "anemia",
+  "anaemic": "anemic",
+  "anaesthesia": "anesthesia",
+  "anaesthetic": "anesthetic",
+  "anaesthetics": "anesthetics",
+  "anaesthetise": "anesthetize",
+  "anaesthetised": "anesthetized",
+  "anaesthetises": "anesthetizes",
+  "anaesthetising": "anesthetizing",
+  "anaesthetist": "anesthetist",
+  "anaesthetists": "anesthetists",
+  "anaesthetize": "anesthetize",
+  "anaesthetized": "anesthetized",
+  "anaesthetizes": "anesthetizes",
+  "anaesthetizing": "anesthetizing",
+  "analogue": "analog",
+  "analogues": "analogs",
+  "analyse": "analyze",
+  "analysed": "analyzed",
+  "analyses": "analyzes",
+  "analysing": "analyzing",
+  "anglicise": "anglicize",
+  "anglicised": "anglicized",
+  "anglicises": "anglicizes",
+  "anglicising": "anglicizing",
+  "annualised": "annualized",
+  "antagonise": "antagonize",
+  "antagonised": "antagonized",
+  "antagonises": "antagonizes",
+  "antagonising": "antagonizing",
+  "apologise": "apologize",
+  "apologised": "apologized",
+  "apologises": "apologizes",
+  "apologising": "apologizing",
+  "appal": "appall",
+  "appals": "appalls",
+  "appetiser": "appetizer",
+  "appetisers": "appetizers",
+  "appetising": "appetizing",
+  "appetisingly": "appetizingly",
+  "arbour": "arbor",
+  "arbours": "arbors",
+  "archaeologically": "archeologically",
+  "archaeologist": "archeologist",
+  "archaeologists": "archeologists",
+  "archaeology": "archeology</span>",
+  "archeological": "archaeological",
+  "ardour": "ardor",
+  "armour": "armor",
+  "armoured": "armored",
+  "armourer": "armorer",
+  "armourers": "armorers",
+  "armouries": "armories",
+  "armoury": "armory",
+  "artefact": "artifact",
+  "artefacts": "artifacts",
+  "authorise": "authorize",
+  "authorised": "authorized",
+  "authorises": "authorizes",
+  "authorising": "authorizing",
+  "axe": "ax",
+  "backpedalled": "backpedaled",
+  "backpedalling": "backpedaling",
+  "bannister": "banister",
+  "bannisters": "banisters",
+  "baptise": "baptize",
+  "baptised": "baptized",
+  "baptises": "baptizes",
+  "baptising": "baptizing",
+  "bastardise": "bastardize",
+  "bastardised": "bastardized",
+  "bastardises": "bastardizes",
+  "bastardising": "bastardizing",
+  "battleax": "battleaxe",
+  "baulk": "balk",
+  "baulked": "balked",
+  "baulking": "balking",
+  "baulks": "balks",
+  "bedevilled": "bedeviled",
+  "bedevilling": "bedeviling",
+  "behaviour": "behavior",
+  "behavioural": "behavioral",
+  "behaviourism": "behaviorism",
+  "behaviourist": "behaviorist",
+  "behaviourists": "behaviorists",
+  "behaviours": "behaviors",
+  "behove": "behoove",
+  "behoved": "behooved",
+  "behoves": "behooves",
+  "bejewelled": "bejeweled",
+  "belabour": "belabor",
+  "belaboured": "belabored",
+  "belabouring": "belaboring",
+  "belabours": "belabors",
+  "bevelled": "beveled",
+  "bevvies": "bevies",
+  "bevvy": "bevy",
+  "biassed": "biased",
+  "biassing": "biasing",
+  "bingeing": "binging",
+  "bougainvillaea": "bougainvillea",
+  "bougainvillaeas": "bougainvilleas",
+  "bowdlerise": "bowdlerize",
+  "bowdlerised": "bowdlerized",
+  "bowdlerises": "bowdlerizes",
+  "bowdlerising": "bowdlerizing",
+  "breathalyse": "breathalyze",
+  "breathalysed": "breathalyzed",
+  "breathalyser": "breathalyzer",
+  "breathalysers": "breathalyzers",
+  "breathalyses": "breathalyzes",
+  "breathalysing": "breathalyzing",
+  "brutalise": "brutalize",
+  "brutalised": "brutalized",
+  "brutalises": "brutalizes",
+  "brutalising": "brutalizing",
+  "busses": "buses",
+  "bussing": "busing",
+  "caesarean": "cesarean",
+  "caesareans": "cesareans",
+  "calibre": "caliber",
+  "calibres": "calibers",
+  "calliper": "caliper",
+  "callipers": "calipers",
+  "callisthenics": "calisthenics",
+  "canalise": "canalize",
+  "canalised": "canalized",
+  "canalises": "canalizes",
+  "canalising": "canalizing",
+  "cancelation": "cancellation",
+  "cancelations": "cancellations",
+  "cancelled": "canceled",
+  "cancelling": "canceling",
+  "candour": "candor",
+  "cannibalise": "cannibalize",
+  "cannibalised": "cannibalized",
+  "cannibalises": "cannibalizes",
+  "cannibalising": "cannibalizing",
+  "canonise": "canonize",
+  "canonised": "canonized",
+  "canonises": "canonizes",
+  "canonising": "canonizing",
+  "capitalise": "capitalize",
+  "capitalised": "capitalized",
+  "capitalises": "capitalizes",
+  "capitalising": "capitalizing",
+  "caramelise": "caramelize",
+  "caramelised": "caramelized",
+  "caramelises": "caramelizes",
+  "caramelising": "caramelizing",
+  "carbonise": "carbonize",
+  "carbonised": "carbonized",
+  "carbonises": "carbonizes",
+  "carbonising": "carbonizing",
+  "carolled": "caroled",
+  "carolling": "caroling",
+  "catalogue": "catalog",
+  "catalogued": "cataloged",
+  "catalogues": "catalogs",
+  "cataloguing": "cataloging",
+  "catalyse": "catalyze",
+  "catalysed": "catalyzed",
+  "catalyses": "catalyzes",
+  "catalysing": "catalyzing",
+  "categorise": "categorize",
+  "categorised": "categorized",
+  "categorises": "categorizes",
+  "categorising": "categorizing",
+  "cauterise": "cauterize",
+  "cauterised": "cauterized",
+  "cauterises": "cauterizes",
+  "cauterising": "cauterizing",
+  "cavilled": "caviled",
+  "cavilling": "caviling",
+  "centigramme": "centigram",
+  "centigrammes": "centigrams",
+  "centilitre": "centiliter",
+  "centilitres": "centiliters",
+  "centimetre": "centimeter",
+  "centimetres": "centimeters",
+  "centralise": "centralize",
+  "centralised": "centralized",
+  "centralises": "centralizes",
+  "centralising": "centralizing",
+  "centre": "center",
+  "centred": "centered",
+  "centrefold": "centerfold",
+  "centrefolds": "centerfolds",
+  "centrepiece": "centerpiece",
+  "centrepieces": "centerpieces",
+  "centres": "centers",
+  "channelled": "channeled",
+  "channelling": "channeling",
+  "characterise": "characterize",
+  "characterised": "characterized",
+  "characterises": "characterizes",
+  "characterising": "characterizing",
+  "cheque": "check",
+  "chequebook": "checkbook",
+  "chequebooks": "checkbooks",
+  "chequered": "checkered",
+  "cheques": "checks",
+  "chilli": "chili",
+  "chimaera": "chimera",
+  "chimaeras": "chimeras",
+  "chiselled": "chiseled",
+  "chiselling": "chiseling",
+  "circularise": "circularize",
+  "circularised": "circularized",
+  "circularises": "circularizes",
+  "circularising": "circularizing",
+  "civilise": "civilize",
+  "civilised": "civilized",
+  "civilises": "civilizes",
+  "civilising": "civilizing",
+  "clamour": "clamor",
+  "clamoured": "clamored",
+  "clamouring": "clamoring",
+  "clamours": "clamors",
+  "clangour": "clangor",
+  "clarinettist": "clarinetist",
+  "clarinettists": "clarinetists",
+  "collectivise": "collectivize",
+  "collectivised": "collectivized",
+  "collectivises": "collectivizes",
+  "collectivising": "collectivizing",
+  "colonisation": "colonization",
+  "colonise": "colonize",
+  "colonised": "colonized",
+  "coloniser": "colonizer",
+  "colonisers": "colonizers",
+  "colonises": "colonizes",
+  "colonising": "colonizing",
+  "colour": "color",
+  "colourant": "colorant",
+  "colourants": "colorants",
+  "coloured": "colored",
+  "coloureds": "coloreds",
+  "colourful": "colorful",
+  "colourfully": "colorfully",
+  "colouring": "coloring",
+  "colourize": "colorize",
+  "colourized": "colorized",
+  "colourizes": "colorizes",
+  "colourizing": "colorizing",
+  "colourless": "colorless",
+  "colours": "colors",
+  "commercialise": "commercialize",
+  "commercialised": "commercialized",
+  "commercialises": "commercializes",
+  "commercialising": "commercializing",
+  "compartmentalise": "compartmentalize",
+  "compartmentalised": "compartmentalized",
+  "compartmentalises": "compartmentalizes",
+  "compartmentalising": "compartmentalizing",
+  "computerise": "computerize",
+  "computerised": "computerized",
+  "computerises": "computerizes",
+  "computerising": "computerizing",
+  "conceptualise": "conceptualize",
+  "conceptualised": "conceptualized",
+  "conceptualises": "conceptualizes",
+  "conceptualising": "conceptualizing",
+  "connexion": "connection",
+  "connexions": "connections",
+  "contextualise": "contextualize",
+  "contextualised": "contextualized",
+  "contextualises": "contextualizes",
+  "contextualising": "contextualizing",
+  "cosier": "cozier",
+  "cosies": "cozies",
+  "cosiest": "coziest",
+  "cosily": "cozily",
+  "cosiness": "coziness",
+  "cosy": "cozy",
+  "councillor": "councilor",
+  "councillors": "councilors",
+  "counselled": "counseled",
+  "counselling": "counseling",
+  "counsellor": "counselor",
+  "counsellors": "counselors",
+  "crenelated": "crenellated",
+  "criminalise": "criminalize",
+  "criminalised": "criminalized",
+  "criminalises": "criminalizes",
+  "criminalising": "criminalizing",
+  "criticise": "criticize",
+  "criticised": "criticized",
+  "criticises": "criticizes",
+  "criticising": "criticizing",
+  "crueller": "crueler",
+  "cruellest": "cruelest",
+  "crystallisation": "crystallization",
+  "crystallise": "crystallize",
+  "crystallised": "crystallized",
+  "crystallises": "crystallizes",
+  "crystallising": "crystallizing",
+  "cudgelled": "cudgeled",
+  "cudgelling": "cudgeling",
+  "customise": "customize",
+  "customised": "customized",
+  "customises": "customizes",
+  "customising": "customizing",
+  "cypher": "cipher",
+  "cyphers": "ciphers",
+  "decentralisation": "decentralization",
+  "decentralise": "decentralize",
+  "decentralised": "decentralized",
+  "decentralises": "decentralizes",
+  "decentralising": "decentralizing",
+  "decriminalisation": "decriminalization",
+  "decriminalise": "decriminalize",
+  "decriminalised": "decriminalized",
+  "decriminalises": "decriminalizes",
+  "decriminalising": "decriminalizing",
+  "defence": "defense",
+  "defenceless": "defenseless",
+  "defences": "defenses",
+  "dehumanisation": "dehumanization",
+  "dehumanise": "dehumanize",
+  "dehumanised": "dehumanized",
+  "dehumanises": "dehumanizes",
+  "dehumanising": "dehumanizing",
+  "demeanour": "demeanor",
+  "demilitarisation": "demilitarization",
+  "demilitarise": "demilitarize",
+  "demilitarised": "demilitarized",
+  "demilitarises": "demilitarizes",
+  "demilitarising": "demilitarizing",
+  "demobilisation": "demobilization",
+  "demobilise": "demobilize",
+  "demobilised": "demobilized",
+  "demobilises": "demobilizes",
+  "demobilising": "demobilizing",
+  "democratisation": "democratization",
+  "democratise": "democratize",
+  "democratised": "democratized",
+  "democratises": "democratizes",
+  "democratising": "democratizing",
+  "demonise": "demonize",
+  "demonised": "demonized",
+  "demonises": "demonizes",
+  "demonising": "demonizing",
+  "demoralisation": "demoralization",
+  "demoralise": "demoralize",
+  "demoralised": "demoralized",
+  "demoralises": "demoralizes",
+  "demoralising": "demoralizing",
+  "denationalisation": "denationalization",
+  "denationalise": "denationalize",
+  "denationalised": "denationalized",
+  "denationalises": "denationalizes",
+  "denationalising": "denationalizing",
+  "deodorise": "deodorize",
+  "deodorised": "deodorized",
+  "deodorises": "deodorizes",
+  "deodorising": "deodorizing",
+  "depersonalise": "depersonalize",
+  "depersonalised": "depersonalized",
+  "depersonalises": "depersonalizes",
+  "depersonalising": "depersonalizing",
+  "deputise": "deputize",
+  "deputised": "deputized",
+  "deputises": "deputizes",
+  "deputising": "deputizing",
+  "desensitisation": "desensitization",
+  "desensitise": "desensitize",
+  "desensitised": "desensitized",
+  "desensitises": "desensitizes",
+  "desensitising": "desensitizing",
+  "destabilisation": "destabilization",
+  "destabilise": "destabilize",
+  "destabilised": "destabilized",
+  "destabilises": "destabilizes",
+  "destabilising": "destabilizing",
+  "dialled": "dialed",
+  "dialling": "dialing",
+  "dialogue": "dialog",
+  "dialogues": "dialogs",
+  "diarrhoea": "diarrhea",
+  "digitise": "digitize",
+  "digitised": "digitized",
+  "digitises": "digitizes",
+  "digitising": "digitizing",
+  "disc": "disk",
+  "discolour": "discolor",
+  "discoloured": "discolored",
+  "discolouring": "discoloring",
+  "discolours": "discolors",
+  "discs": "disks",
+  "disembowelled": "disemboweled",
+  "disembowelling": "disemboweling",
+  "disfavour": "disfavor",
+  "dishevelled": "disheveled",
+  "dishonour": "dishonor",
+  "dishonourable": "dishonorable",
+  "dishonourably": "dishonorably",
+  "dishonoured": "dishonored",
+  "dishonouring": "dishonoring",
+  "dishonours": "dishonors",
+  "disorganisation": "disorganization",
+  "disorganised": "disorganized",
+  "distil": "distill",
+  "distils": "distills",
+  "dramatisation": "dramatization",
+  "dramatisations": "dramatizations",
+  "dramatise": "dramatize",
+  "dramatised": "dramatized",
+  "dramatises": "dramatizes",
+  "dramatising": "dramatizing",
+  "draught": "draft",
+  "draughtboard": "draftboard",
+  "draughtboards": "draftboards",
+  "draughtier": "draftier",
+  "draughtiest": "draftiest",
+  "draughts": "drafts",
+  "draughtsman": "draftsman",
+  "draughtsmanship": "draftsmanship",
+  "draughtsmen": "draftsmen",
+  "draughtswoman": "draftswoman",
+  "draughtswomen": "draftswomen",
+  "draughty": "drafty",
+  "drivelled": "driveled",
+  "drivelling": "driveling",
+  "duelled": "dueled",
+  "duelling": "dueling",
+  "economise": "economize",
+  "economised": "economized",
+  "economises": "economizes",
+  "economising": "economizing",
+  "editorialise": "editorialize",
+  "editorialised": "editorialized",
+  "editorialises": "editorializes",
+  "editorialising": "editorializing",
+  "edoema": "edema",
+  "empathise": "empathize",
+  "empathised": "empathized",
+  "empathises": "empathizes",
+  "empathising": "empathizing",
+  "emphasise": "emphasize",
+  "emphasised": "emphasized",
+  "emphasises": "emphasizes",
+  "emphasising": "emphasizing",
+  "enamelled": "enameled",
+  "enamelling": "enameling",
+  "enamoured": "enamored",
+  "encyclopaedia": "encyclopedia",
+  "encyclopaedias": "encyclopedias",
+  "encyclopaedic": "encyclopedic",
+  "endeavour": "endeavor",
+  "endeavoured": "endeavored",
+  "endeavouring": "endeavoring",
+  "endeavours": "endeavors",
+  "energise": "energize",
+  "energised": "energized",
+  "energises": "energizes",
+  "energising": "energizing",
+  "enrol": "enroll",
+  "enrols": "enrolls",
+  "enthral": "enthrall",
+  "enthrals": "enthralls",
+  "epaulette": "epaulet",
+  "epaulettes": "epaulets",
+  "epicentre": "epicenter",
+  "epicentres": "epicenters",
+  "epilogue": "epilog",
+  "epilogues": "epilogs",
+  "epitomise": "epitomize",
+  "epitomised": "epitomized",
+  "epitomises": "epitomizes",
+  "epitomising": "epitomizing",
+  "equalisation": "equalization",
+  "equalise": "equalize",
+  "equalised": "equalized",
+  "equaliser": "equalizer",
+  "equalisers": "equalizers",
+  "equalises": "equalizes",
+  "equalising": "equalizing",
+  "eulogise": "eulogize",
+  "eulogised": "eulogized",
+  "eulogises": "eulogizes",
+  "eulogising": "eulogizing",
+  "evangelise": "evangelize",
+  "evangelised": "evangelized",
+  "evangelises": "evangelizes",
+  "evangelising": "evangelizing",
+  "exorcise": "exorcize",
+  "exorcised": "exorcized",
+  "exorcises": "exorcizes",
+  "exorcising": "exorcizing",
+  "extemporisation": "extemporization",
+  "extemporise": "extemporize",
+  "extemporised": "extemporized",
+  "extemporises": "extemporizes",
+  "extemporising": "extemporizing",
+  "externalisation": "externalization",
+  "externalisations": "externalizations",
+  "externalise": "externalize",
+  "externalised": "externalized",
+  "externalises": "externalizes",
+  "externalising": "externalizing",
+  "factorise": "factorize",
+  "factorised": "factorized",
+  "factorises": "factorizes",
+  "factorising": "factorizing",
+  "faecal": "fecal",
+  "faeces": "feces",
+  "familiarisation": "familiarization",
+  "familiarise": "familiarize",
+  "familiarised": "familiarized",
+  "familiarises": "familiarizes",
+  "familiarising": "familiarizing",
+  "fantasise": "fantasize",
+  "fantasised": "fantasized",
+  "fantasises": "fantasizes",
+  "fantasising": "fantasizing",
+  "favour": "favor",
+  "favourable": "favorable",
+  "favourably": "favorably",
+  "favoured": "favored",
+  "favouring": "favoring",
+  "favourite": "favorite",
+  "favourites": "favorites",
+  "favouritism": "favoritism",
+  "favours": "favors",
+  "feminise": "feminize",
+  "feminised": "feminized",
+  "feminises": "feminizes",
+  "feminising": "feminizing",
+  "fertilisation": "fertilization",
+  "fertilise": "fertilize",
+  "fertilised": "fertilized",
+  "fertiliser": "fertilizer",
+  "fertilisers": "fertilizers",
+  "fertilises": "fertilizes",
+  "fertilising": "fertilizing",
+  "fervour": "fervor",
+  "fibre": "fiber",
+  "fibreglass": "fiberglass",
+  "fibres": "fibers",
+  "fictionalisation": "fictionalization",
+  "fictionalisations": "fictionalizations",
+  "fictionalise": "fictionalize",
+  "fictionalised": "fictionalized",
+  "fictionalises": "fictionalizes",
+  "fictionalising": "fictionalizing",
+  "fillet": "filet",
+  "filleted": "fileted",
+  "filleting": "fileting",
+  "fillets": "filets",
+  "finalisation": "finalization",
+  "finalise": "finalize",
+  "finalised": "finalized",
+  "finalises": "finalizes",
+  "finalising": "finalizing",
+  "flautist": "flutist",
+  "flautists": "flutists",
+  "flavour": "flavor",
+  "flavoured": "flavored",
+  "flavouring": "flavoring",
+  "flavourings": "flavorings",
+  "flavourless": "flavorless",
+  "flavours": "flavors",
+  "flavoursome": "flavorsome",
+  "flyer / flier": "flier / flyer",
+  "foetal": "fetal",
+  "foetid": "fetid",
+  "foetus": "fetus",
+  "foetuses": "fetuses",
+  "formalisation": "formalization",
+  "formalise": "formalize",
+  "formalised": "formalized",
+  "formalises": "formalizes",
+  "formalising": "formalizing",
+  "fossilisation": "fossilization",
+  "fossilise": "fossilize",
+  "fossilised": "fossilized",
+  "fossilises": "fossilizes",
+  "fossilising": "fossilizing",
+  "fraternisation": "fraternization",
+  "fraternise": "fraternize",
+  "fraternised": "fraternized",
+  "fraternises": "fraternizes",
+  "fraternising": "fraternizing",
+  "fulfil": "fulfill",
+  "fulfilment": "fulfillment",
+  "fulfils": "fulfills",
+  "funnelled": "funneled",
+  "funnelling": "funneling",
+  "gage": "gauge",
+  "gaged": "gauged",
+  "gages": "gauges",
+  "gaging": "gauging",
+  "galvanise": "galvanize",
+  "galvanised": "galvanized",
+  "galvanises": "galvanizes",
+  "galvanising": "galvanizing",
+  "gambolled": "gamboled",
+  "gambolling": "gamboling",
+  "gaol": "jail",
+  "gaolbird": "jailbird",
+  "gaolbirds": "jailbirds",
+  "gaolbreak": "jailbreak",
+  "gaolbreaks": "jailbreaks",
+  "gaoled": "jailed",
+  "gaoler": "jailer",
+  "gaolers": "jailers",
+  "gaoling": "jailing",
+  "gaols": "jails",
+  "gasses": "gases",
+  "generalisation": "generalization",
+  "generalisations": "generalizations",
+  "generalise": "generalize",
+  "generalised": "generalized",
+  "generalises": "generalizes",
+  "generalising": "generalizing",
+  "ghettoise": "ghettoize",
+  "ghettoised": "ghettoized",
+  "ghettoises": "ghettoizes",
+  "ghettoising": "ghettoizing",
+  "gipsies": "gypsies",
+  "glamor": "glamour",
+  "glamorise": "glamorize",
+  "glamorised": "glamorized",
+  "glamorises": "glamorizes",
+  "glamorising": "glamorizing",
+  "globalisation": "globalization",
+  "globalise": "globalize",
+  "globalised": "globalized",
+  "globalises": "globalizes",
+  "globalising": "globalizing",
+  "glueing": "gluing",
+  "goitre": "goiter",
+  "goitres": "goiters",
+  "gonorrhoea": "gonorrhea",
+  "gramme": "gram",
+  "grammes": "grams",
+  "gravelled": "graveled",
+  "grey": "gray",
+  "greyed": "grayed",
+  "greying": "graying",
+  "greyish": "grayish",
+  "greyness": "grayness",
+  "greys": "grays",
+  "grovelled": "groveled",
+  "grovelling": "groveling",
+  "groyne": "groin",
+  "groynes": "groins",
+  "gruelling": "grueling",
+  "gruellingly": "gruelingly",
+  "gryphon": "griffin",
+  "gryphons": "griffins",
+  "gynaecological": "gynecological",
+  "gynaecologist": "gynecologist",
+  "gynaecologists": "gynecologists",
+  "gynaecology": "gynecology",
+  "haematological": "hematological",
+  "haematologist": "hematologist",
+  "haematologists": "hematologists",
+  "haematology": "hematology",
+  "haemoglobin": "hemoglobin",
+  "haemophilia": "hemophilia",
+  "haemophiliac": "hemophiliac",
+  "haemophiliacs": "hemophiliacs",
+  "haemorrhage": "hemorrhage",
+  "haemorrhaged": "hemorrhaged",
+  "haemorrhages": "hemorrhages",
+  "haemorrhaging": "hemorrhaging",
+  "haemorrhoids": "hemorrhoids",
+  "harbour": "harbor",
+  "harboured": "harbored",
+  "harbouring": "harboring",
+  "harbours": "harbors",
+  "harmonisation": "harmonization",
+  "harmonise": "harmonize",
+  "harmonised": "harmonized",
+  "harmonises": "harmonizes",
+  "harmonising": "harmonizing",
+  "homoeopath": "homeopath",
+  "homoeopathic": "homeopathic",
+  "homoeopaths": "homeopaths",
+  "homoeopathy": "homeopathy",
+  "homogenise": "homogenize",
+  "homogenised": "homogenized",
+  "homogenises": "homogenizes",
+  "homogenising": "homogenizing",
+  "honour": "honor",
+  "honourable": "honorable",
+  "honourably": "honorably",
+  "honoured": "honored",
+  "honouring": "honoring",
+  "honours": "honors",
+  "hospitalisation": "hospitalization",
+  "hospitalise": "hospitalize",
+  "hospitalised": "hospitalized",
+  "hospitalises": "hospitalizes",
+  "hospitalising": "hospitalizing",
+  "humanise": "humanize",
+  "humanised": "humanized",
+  "humanises": "humanizes",
+  "humanising": "humanizing",
+  "humour": "humor",
+  "humoured": "humored",
+  "humouring": "humoring",
+  "humourless": "humorless",
+  "humours": "humors",
+  "hybridise": "hybridize",
+  "hybridised": "hybridized",
+  "hybridises": "hybridizes",
+  "hybridising": "hybridizing",
+  "hypnotise": "hypnotize",
+  "hypnotised": "hypnotized",
+  "hypnotises": "hypnotizes",
+  "hypnotising": "hypnotizing",
+  "hypothesise": "hypothesize",
+  "hypothesised": "hypothesized",
+  "hypothesises": "hypothesizes",
+  "hypothesising": "hypothesizing",
+  "idealisation": "idealization",
+  "idealise": "idealize",
+  "idealised": "idealized",
+  "idealises": "idealizes",
+  "idealising": "idealizing",
+  "idolise": "idolize",
+  "idolised": "idolized",
+  "idolises": "idolizes",
+  "idolising": "idolizing",
+  "immobilisation": "immobilization",
+  "immobilise": "immobilize",
+  "immobilised": "immobilized",
+  "immobiliser": "immobilizer",
+  "immobilisers": "immobilizers",
+  "immobilises": "immobilizes",
+  "immobilising": "immobilizing",
+  "immortalise": "immortalize",
+  "immortalised": "immortalized",
+  "immortalises": "immortalizes",
+  "immortalising": "immortalizing",
+  "immunisation": "immunization",
+  "immunise": "immunize",
+  "immunised": "immunized",
+  "immunises": "immunizes",
+  "immunising": "immunizing",
+  "impanelled": "impaneled",
+  "impanelling": "impaneling",
+  "imperilled": "imperiled",
+  "imperilling": "imperiling",
+  "individualise": "individualize",
+  "individualised": "individualized",
+  "individualises": "individualizes",
+  "individualising": "individualizing",
+  "industrialise": "industrialize",
+  "industrialised": "industrialized",
+  "industrialises": "industrializes",
+  "industrialising": "industrializing",
+  "inflexion": "inflection",
+  "inflexions": "inflections",
+  "initialise": "initialize",
+  "initialised": "initialized",
+  "initialises": "initializes",
+  "initialising": "initializing",
+  "initialled": "initialed",
+  "initialling": "initialing",
+  "instal": "install",
+  "instalment": "installment",
+  "instalments": "installments",
+  "instals": "installs",
+  "instil": "instill",
+  "instils": "instills",
+  "institutionalisation": "institutionalization",
+  "institutionalise": "institutionalize",
+  "institutionalised": "institutionalized",
+  "institutionalises": "institutionalizes",
+  "institutionalising": "institutionalizing",
+  "intellectualise": "intellectualize",
+  "intellectualised": "intellectualized",
+  "intellectualises": "intellectualizes",
+  "intellectualising": "intellectualizing",
+  "internalisation": "internalization",
+  "internalise": "internalize",
+  "internalised": "internalized",
+  "internalises": "internalizes",
+  "internalising": "internalizing",
+  "internationalisation": "internationalization",
+  "internationalise": "internationalize",
+  "internationalised": "internationalized",
+  "internationalises": "internationalizes",
+  "internationalising": "internationalizing",
+  "ionisation": "ionization",
+  "ionise": "ionize",
+  "ionised": "ionized",
+  "ioniser": "ionizer",
+  "ionisers": "ionizers",
+  "ionises": "ionizes",
+  "ionising": "ionizing",
+  "italicise": "italicize",
+  "italicised": "italicized",
+  "italicises": "italicizes",
+  "italicising": "italicizing",
+  "itemise": "itemize",
+  "itemised": "itemized",
+  "itemises": "itemizes",
+  "itemising": "itemizing",
+  "jeopardise": "jeopardize",
+  "jeopardised": "jeopardized",
+  "jeopardises": "jeopardizes",
+  "jeopardising": "jeopardizing",
+  "jewelled": "jeweled",
+  "jeweller": "jeweler",
+  "jewellers": "jewelers",
+  "jewellery": "jewelry",
+  "judgement": "judgment",
+  "kilogramme": "kilogram",
+  "kilogrammes": "kilograms",
+  "kilometre": "kilometer",
+  "kilometres": "kilometers",
+  "labelled": "labeled",
+  "labelling": "labeling",
+  "labour": "labor",
+  "laboured": "labored",
+  "labourer": "laborer",
+  "labourers": "laborers",
+  "labouring": "laboring",
+  "labours": "labors",
+  "lacklustre": "lackluster",
+  "legalisation": "legalization",
+  "legalise": "legalize",
+  "legalised": "legalized",
+  "legalises": "legalizes",
+  "legalising": "legalizing",
+  "legitimise": "legitimize",
+  "legitimised": "legitimized",
+  "legitimises": "legitimizes",
+  "legitimising": "legitimizing",
+  "leukaemia": "leukemia",
+  "levelled": "leveled",
+  "leveller": "leveler",
+  "levellers": "levelers",
+  "levelling": "leveling",
+  "libelled": "libeled",
+  "libelling": "libeling",
+  "libellous": "libelous",
+  "liberalisation": "liberalization",
+  "liberalise": "liberalize",
+  "liberalised": "liberalized",
+  "liberalises": "liberalizes",
+  "liberalising": "liberalizing",
+  "licence": "license",
+  "licenced": "licensed",
+  "licences": "licenses",
+  "licencing": "licensing",
+  "likeable": "likable",
+  "lionisation": "lionization",
+  "lionise": "lionize",
+  "lionised": "lionized",
+  "lionises": "lionizes",
+  "lionising": "lionizing",
+  "liquidise": "liquidize",
+  "liquidised": "liquidized",
+  "liquidiser": "liquidizer",
+  "liquidisers": "liquidizers",
+  "liquidises": "liquidizes",
+  "liquidising": "liquidizing",
+  "litre": "liter",
+  "litres": "liters",
+  "localise": "localize",
+  "localised": "localized",
+  "localises": "localizes",
+  "localising": "localizing",
+  "louvre": "louver",
+  "louvred": "louvered",
+  "louvres": "louvers",
+  "lustre": "luster",
+  "magnetise": "magnetize",
+  "magnetised": "magnetized",
+  "magnetises": "magnetizes",
+  "magnetising": "magnetizing",
+  "manoeuvrability": "maneuverability",
+  "manoeuvrable": "maneuverable",
+  "manoeuvre": "maneuver",
+  "manoeuvred": "maneuvered",
+  "manoeuvres": "maneuvers",
+  "manoeuvring": "maneuvering",
+  "manoeuvrings": "maneuverings",
+  "marginalisation": "marginalization",
+  "marginalise": "marginalize",
+  "marginalised": "marginalized",
+  "marginalises": "marginalizes",
+  "marginalising": "marginalizing",
+  "marshalled": "marshaled",
+  "marshalling": "marshaling",
+  "marvelled": "marveled",
+  "marvelling": "marveling",
+  "marvellous": "marvelous",
+  "marvellously": "marvelously",
+  "materialisation": "materialization",
+  "materialise": "materialize",
+  "materialised": "materialized",
+  "materialises": "materializes",
+  "materialising": "materializing",
+  "maximisation": "maximization",
+  "maximise": "maximize",
+  "maximised": "maximized",
+  "maximises": "maximizes",
+  "maximising": "maximizing",
+  "meagre": "meager",
+  "mechanisation": "mechanization",
+  "mechanise": "mechanize",
+  "mechanised": "mechanized",
+  "mechanises": "mechanizes",
+  "mechanising": "mechanizing",
+  "mediaeval": "medieval",
+  "memorialise": "memorialize",
+  "memorialised": "memorialized",
+  "memorialises": "memorializes",
+  "memorialising": "memorializing",
+  "memorise": "memorize",
+  "memorised": "memorized",
+  "memorises": "memorizes",
+  "memorising": "memorizing",
+  "mesmerise": "mesmerize",
+  "mesmerised": "mesmerized",
+  "mesmerises": "mesmerizes",
+  "mesmerising": "mesmerizing",
+  "metabolise": "metabolize",
+  "metabolised": "metabolized",
+  "metabolises": "metabolizes",
+  "metabolising": "metabolizing",
+  "metre": "meter",
+  "metres": "meters",
+  "mhm": "hmm",
+  "micrometre": "micrometer",
+  "micrometres": "micrometers",
+  "militarise": "militarize",
+  "militarised": "militarized",
+  "militarises": "militarizes",
+  "militarising": "militarizing",
+  "milligramme": "milligram",
+  "milligrammes": "milligrams",
+  "millilitre": "milliliter",
+  "millilitres": "milliliters",
+  "millimetre": "millimeter",
+  "millimetres": "millimeters",
+  "miniaturisation": "miniaturization",
+  "miniaturise": "miniaturize",
+  "miniaturised": "miniaturized",
+  "miniaturises": "miniaturizes",
+  "miniaturising": "miniaturizing",
+  "minibusses": "minibuses",
+  "minimise": "minimize",
+  "minimised": "minimized",
+  "minimises": "minimizes",
+  "minimising": "minimizing",
+  "misbehaviour": "misbehavior",
+  "misdemeanour": "misdemeanor",
+  "misdemeanours": "misdemeanors",
+  "misspelt": "misspelled",
+  "mitre": "miter",
+  "mitres": "miters",
+  "mm": "hmm",
+  "mmm": "hmm",
+  "mobilisation": "mobilization",
+  "mobilise": "mobilize",
+  "mobilised": "mobilized",
+  "mobilises": "mobilizes",
+  "mobilising": "mobilizing",
+  "modelled": "modeled",
+  "modeller": "modeler",
+  "modellers": "modelers",
+  "modelling": "modeling",
+  "modernise": "modernize",
+  "modernised": "modernized",
+  "modernises": "modernizes",
+  "modernising": "modernizing",
+  "moisturise": "moisturize",
+  "moisturised": "moisturized",
+  "moisturiser": "moisturizer",
+  "moisturisers": "moisturizers",
+  "moisturises": "moisturizes",
+  "moisturising": "moisturizing",
+  "monologue": "monolog",
+  "monologues": "monologs",
+  "monopolisation": "monopolization",
+  "monopolise": "monopolize",
+  "monopolised": "monopolized",
+  "monopolises": "monopolizes",
+  "monopolising": "monopolizing",
+  "moralise": "moralize",
+  "moralised": "moralized",
+  "moralises": "moralizes",
+  "moralising": "moralizing",
+  "motorised": "motorized",
+  "mould": "mold",
+  "moulded": "molded",
+  "moulder": "molder",
+  "mouldered": "moldered",
+  "mouldering": "moldering",
+  "moulders": "molders",
+  "mouldier": "moldier",
+  "mouldiest": "moldiest",
+  "moulding": "molding",
+  "mouldings": "moldings",
+  "moulds": "molds",
+  "mouldy": "moldy",
+  "moult": "molt",
+  "moulted": "molted",
+  "moulting": "molting",
+  "moults": "molts",
+  "moustache": "mustache",
+  "moustached": "mustached",
+  "moustaches": "mustaches",
+  "moustachioed": "mustachioed",
+  "multicoloured": "multicolored",
+  "nationalisation": "nationalization",
+  "nationalisations": "nationalizations",
+  "nationalise": "nationalize",
+  "nationalised": "nationalized",
+  "nationalises": "nationalizes",
+  "nationalising": "nationalizing",
+  "naturalisation": "naturalization",
+  "naturalise": "naturalize",
+  "naturalised": "naturalized",
+  "naturalises": "naturalizes",
+  "naturalising": "naturalizing",
+  "neighbour": "neighbor",
+  "neighbourhood": "neighborhood",
+  "neighbourhoods": "neighborhoods",
+  "neighbouring": "neighboring",
+  "neighbourliness": "neighborliness",
+  "neighbourly": "neighborly",
+  "neighbours": "neighbors",
+  "neutralisation": "neutralization",
+  "neutralise": "neutralize",
+  "neutralised": "neutralized",
+  "neutralises": "neutralizes",
+  "neutralising": "neutralizing",
+  "normalisation": "normalization",
+  "normalise": "normalize",
+  "normalised": "normalized",
+  "normalises": "normalizes",
+  "normalising": "normalizing",
+  "odour": "odor",
+  "odourless": "odorless",
+  "odours": "odors",
+  "oesophagus": "esophagus",
+  "oesophaguses": "esophaguses",
+  "oestrogen": "estrogen",
+  "offence": "offense",
+  "offences": "offenses",
+  "omelette": "omelet",
+  "omelettes": "omelets",
+  "optimise": "optimize",
+  "optimised": "optimized",
+  "optimises": "optimizes",
+  "optimising": "optimizing",
+  "organisation": "organization",
+  "organisational": "organizational",
+  "organisations": "organizations",
+  "organise": "organize",
+  "organised": "organized",
+  "organiser": "organizer",
+  "organisers": "organizers",
+  "organises": "organizes",
+  "organising": "organizing",
+  "orthopaedic": "orthopedic",
+  "orthopaedics": "orthopedics",
+  "ostracise": "ostracize",
+  "ostracised": "ostracized",
+  "ostracises": "ostracizes",
+  "ostracising": "ostracizing",
+  "outmanoeuvre": "outmaneuver",
+  "outmanoeuvred": "outmaneuvered",
+  "outmanoeuvres": "outmaneuvers",
+  "outmanoeuvring": "outmaneuvering",
+  "overemphasise": "overemphasize",
+  "overemphasised": "overemphasized",
+  "overemphasises": "overemphasizes",
+  "overemphasising": "overemphasizing",
+  "oxidisation": "oxidization",
+  "oxidise": "oxidize",
+  "oxidised": "oxidized",
+  "oxidises": "oxidizes",
+  "oxidising": "oxidizing",
+  "paederast": "pederast",
+  "paederasts": "pederasts",
+  "paediatric": "pediatric",
+  "paediatrician": "pediatrician",
+  "paediatricians": "pediatricians",
+  "paediatrics": "pediatrics",
+  "paedophile": "pedophile",
+  "paedophiles": "pedophiles",
+  "paedophilia": "pedophilia",
+  "palaeolithic": "paleolithic",
+  "palaeontologist": "paleontologist",
+  "palaeontologists": "paleontologists",
+  "palaeontology": "paleontology",
+  "panelled": "paneled",
+  "panelling": "paneling",
+  "panellist": "panelist",
+  "panellists": "panelists",
+  "paralyse": "paralyze",
+  "paralysed": "paralyzed",
+  "paralyses": "paralyzes",
+  "paralysing": "paralyzing",
+  "parcelled": "parceled",
+  "parcelling": "parceling",
+  "parlour": "parlor",
+  "parlours": "parlors",
+  "particularise": "particularize",
+  "particularised": "particularized",
+  "particularises": "particularizes",
+  "particularising": "particularizing",
+  "passivisation": "passivization",
+  "passivise": "passivize",
+  "passivised": "passivized",
+  "passivises": "passivizes",
+  "passivising": "passivizing",
+  "pasteurisation": "pasteurization",
+  "pasteurise": "pasteurize",
+  "pasteurised": "pasteurized",
+  "pasteurises": "pasteurizes",
+  "pasteurising": "pasteurizing",
+  "patronise": "patronize",
+  "patronised": "patronized",
+  "patronises": "patronizes",
+  "patronising": "patronizing",
+  "patronisingly": "patronizingly",
+  "pedalled": "pedaled",
+  "pedalling": "pedaling",
+  "pedestrianisation": "pedestrianization",
+  "pedestrianise": "pedestrianize",
+  "pedestrianised": "pedestrianized",
+  "pedestrianises": "pedestrianizes",
+  "pedestrianising": "pedestrianizing",
+  "penalise": "penalize",
+  "penalised": "penalized",
+  "penalises": "penalizes",
+  "penalising": "penalizing",
+  "pencilled": "penciled",
+  "pencilling": "penciling",
+  "personalise": "personalize",
+  "personalised": "personalized",
+  "personalises": "personalizes",
+  "personalising": "personalizing",
+  "pharmacopoeia": "pharmacopeia",
+  "pharmacopoeias": "pharmacopeias",
+  "philosophise": "philosophize",
+  "philosophised": "philosophized",
+  "philosophises": "philosophizes",
+  "philosophising": "philosophizing",
+  "philtre": "filter",
+  "philtres": "filters",
+  "phoney": "phony",
+  "plagiarise": "plagiarize",
+  "plagiarised": "plagiarized",
+  "plagiarises": "plagiarizes",
+  "plagiarising": "plagiarizing",
+  "plough": "plow",
+  "ploughed": "plowed",
+  "ploughing": "plowing",
+  "ploughman": "plowman",
+  "ploughmen": "plowmen",
+  "ploughs": "plows",
+  "ploughshare": "plowshare",
+  "ploughshares": "plowshares",
+  "polarisation": "polarization",
+  "polarise": "polarize",
+  "polarised": "polarized",
+  "polarises": "polarizes",
+  "polarising": "polarizing",
+  "politicisation": "politicization",
+  "politicise": "politicize",
+  "politicised": "politicized",
+  "politicises": "politicizes",
+  "politicising": "politicizing",
+  "popularisation": "popularization",
+  "popularise": "popularize",
+  "popularised": "popularized",
+  "popularises": "popularizes",
+  "popularising": "popularizing",
+  "pouffe": "pouf",
+  "pouffes": "poufs",
+  "practise": "practice",
+  "practised": "practiced",
+  "practises": "practices",
+  "practising": "practicing",
+  "praesidium": "presidium",
+  "praesidiums": "presidiums",
+  "pressurisation": "pressurization",
+  "pressurise": "pressurize",
+  "pressurised": "pressurized",
+  "pressurises": "pressurizes",
+  "pressurising": "pressurizing",
+  "pretence": "pretense",
+  "pretences": "pretenses",
+  "primaeval": "primeval",
+  "prioritisation": "prioritization",
+  "prioritise": "prioritize",
+  "prioritised": "prioritized",
+  "prioritises": "prioritizes",
+  "prioritising": "prioritizing",
+  "privatisation": "privatization",
+  "privatisations": "privatizations",
+  "privatise": "privatize",
+  "privatised": "privatized",
+  "privatises": "privatizes",
+  "privatising": "privatizing",
+  "professionalisation": "professionalization",
+  "professionalise": "professionalize",
+  "professionalised": "professionalized",
+  "professionalises": "professionalizes",
+  "professionalising": "professionalizing",
+  "programme": "program",
+  "programmes": "programs",
+  "prologue": "prolog",
+  "prologues": "prologs",
+  "propagandise": "propagandize",
+  "propagandised": "propagandized",
+  "propagandises": "propagandizes",
+  "propagandising": "propagandizing",
+  "proselytise": "proselytize",
+  "proselytised": "proselytized",
+  "proselytiser": "proselytizer",
+  "proselytisers": "proselytizers",
+  "proselytises": "proselytizes",
+  "proselytising": "proselytizing",
+  "psychoanalyse": "psychoanalyze",
+  "psychoanalysed": "psychoanalyzed",
+  "psychoanalyses": "psychoanalyzes",
+  "psychoanalysing": "psychoanalyzing",
+  "publicise": "publicize",
+  "publicised": "publicized",
+  "publicises": "publicizes",
+  "publicising": "publicizing",
+  "pulverisation": "pulverization",
+  "pulverise": "pulverize",
+  "pulverised": "pulverized",
+  "pulverises": "pulverizes",
+  "pulverising": "pulverizing",
+  "pummelled": "pummel",
+  "pummelling": "pummeled",
+  "pyjama": "pajama",
+  "pyjamas": "pajamas",
+  "pzazz": "pizzazz",
+  "quarrelled": "quarreled",
+  "quarrelling": "quarreling",
+  "radicalise": "radicalize",
+  "radicalised": "radicalized",
+  "radicalises": "radicalizes",
+  "radicalising": "radicalizing",
+  "rancour": "rancor",
+  "randomise": "randomize",
+  "randomised": "randomized",
+  "randomises": "randomizes",
+  "randomising": "randomizing",
+  "rationalisation": "rationalization",
+  "rationalisations": "rationalizations",
+  "rationalise": "rationalize",
+  "rationalised": "rationalized",
+  "rationalises": "rationalizes",
+  "rationalising": "rationalizing",
+  "ravelled": "raveled",
+  "ravelling": "raveling",
+  "realisable": "realizable",
+  "realisation": "realization",
+  "realisations": "realizations",
+  "realise": "realize",
+  "realised": "realized",
+  "realises": "realizes",
+  "realising": "realizing",
+  "recognisable": "recognizable",
+  "recognisably": "recognizably",
+  "recognisance": "recognizance",
+  "recognise": "recognize",
+  "recognised": "recognized",
+  "recognises": "recognizes",
+  "recognising": "recognizing",
+  "reconnoitre": "reconnoiter",
+  "reconnoitred": "reconnoitered",
+  "reconnoitres": "reconnoiters",
+  "reconnoitring": "reconnoitering",
+  "refuelled": "refueled",
+  "refuelling": "refueling",
+  "regularisation": "regularization",
+  "regularise": "regularize",
+  "regularised": "regularized",
+  "regularises": "regularizes",
+  "regularising": "regularizing",
+  "remodelled": "remodeled",
+  "remodelling": "remodeling",
+  "remould": "remold",
+  "remoulded": "remolded",
+  "remoulding": "remolding",
+  "remoulds": "remolds",
+  "reorganisation": "reorganization",
+  "reorganisations": "reorganizations",
+  "reorganise": "reorganize",
+  "reorganised": "reorganized",
+  "reorganises": "reorganizes",
+  "reorganising": "reorganizing",
+  "revelled": "reveled",
+  "reveller": "reveler",
+  "revellers": "revelers",
+  "revelling": "reveling",
+  "revitalise": "revitalize",
+  "revitalised": "revitalized",
+  "revitalises": "revitalizes",
+  "revitalising": "revitalizing",
+  "revolutionise": "revolutionize",
+  "revolutionised": "revolutionized",
+  "revolutionises": "revolutionizes",
+  "revolutionising": "revolutionizing",
+  "rhapsodise": "rhapsodize",
+  "rhapsodised": "rhapsodized",
+  "rhapsodises": "rhapsodizes",
+  "rhapsodising": "rhapsodizing",
+  "rigour": "rigor",
+  "rigours": "rigors",
+  "ritualised": "ritualized",
+  "rivalled": "rivaled",
+  "rivalling": "rivaling",
+  "romanticise": "romanticize",
+  "romanticised": "romanticized",
+  "romanticises": "romanticizes",
+  "romanticising": "romanticizing",
+  "rumour": "rumor",
+  "rumoured": "rumored",
+  "rumours": "rumors",
+  "sabre": "saber",
+  "sabres": "sabers",
+  "saltpetre": "saltpeter",
+  "sanitise": "sanitize",
+  "sanitised": "sanitized",
+  "sanitises": "sanitizes",
+  "sanitising": "sanitizing",
+  "satirise": "satirize",
+  "satirised": "satirized",
+  "satirises": "satirizes",
+  "satirising": "satirizing",
+  "saviour": "savior",
+  "saviours": "saviors",
+  "savour": "savor",
+  "savoured": "savored",
+  "savouries": "savories",
+  "savouring": "savoring",
+  "savours": "savors",
+  "savoury": "savory",
+  "scandalise": "scandalize",
+  "scandalised": "scandalized",
+  "scandalises": "scandalizes",
+  "scandalising": "scandalizing",
+  "sceptic": "skeptic",
+  "sceptical": "skeptical",
+  "sceptically": "skeptically",
+  "scepticism": "skepticism",
+  "sceptics": "skeptics",
+  "sceptre": "scepter",
+  "sceptres": "scepters",
+  "scrutinise": "scrutinize",
+  "scrutinised": "scrutinized",
+  "scrutinises": "scrutinizes",
+  "scrutinising": "scrutinizing",
+  "secularisation": "secularization",
+  "secularise": "secularize",
+  "secularised": "secularized",
+  "secularises": "secularizes",
+  "secularising": "secularizing",
+  "sensationalise": "sensationalize",
+  "sensationalised": "sensationalized",
+  "sensationalises": "sensationalizes",
+  "sensationalising": "sensationalizing",
+  "sensitise": "sensitize",
+  "sensitised": "sensitized",
+  "sensitises": "sensitizes",
+  "sensitising": "sensitizing",
+  "sentimentalise": "sentimentalize",
+  "sentimentalised": "sentimentalized",
+  "sentimentalises": "sentimentalizes",
+  "sentimentalising": "sentimentalizing",
+  "sepulchre": "sepulcher",
+  "sepulchres": "sepulchers",
+  "serialisation": "serialization",
+  "serialisations": "serializations",
+  "serialise": "serialize",
+  "serialised": "serialized",
+  "serialises": "serializes",
+  "serialising": "serializing",
+  "sermonise": "sermonize",
+  "sermonised": "sermonized",
+  "sermonises": "sermonizes",
+  "sermonising": "sermonizing",
+  "sheikh": "sheik",
+  "shovelled": "shoveled",
+  "shovelling": "shoveling",
+  "shrivelled": "shriveled",
+  "shrivelling": "shriveling",
+  "signalise": "signalize",
+  "signalised": "signalized",
+  "signalises": "signalizes",
+  "signalising": "signalizing",
+  "signalled": "signaled",
+  "signalling": "signaling",
+  "smoulder": "smolder",
+  "smouldered": "smoldered",
+  "smouldering": "smoldering",
+  "smoulders": "smolders",
+  "snivelled": "sniveled",
+  "snivelling": "sniveling",
+  "snorkelled": "snorkeled",
+  "snorkelling": "snorkeling",
+  "snowplough": "snowplow",
+  "snowploughs": "snowplow",
+  "socialisation": "socialization",
+  "socialise": "socialize",
+  "socialised": "socialized",
+  "socialises": "socializes",
+  "socialising": "socializing",
+  "sodomise": "sodomize",
+  "sodomised": "sodomized",
+  "sodomises": "sodomizes",
+  "sodomising": "sodomizing",
+  "solemnise": "solemnize",
+  "solemnised": "solemnized",
+  "solemnises": "solemnizes",
+  "solemnising": "solemnizing",
+  "sombre": "somber",
+  "specialisation": "specialization",
+  "specialisations": "specializations",
+  "specialise": "specialize",
+  "specialised": "specialized",
+  "specialises": "specializes",
+  "specialising": "specializing",
+  "spectre": "specter",
+  "spectres": "specters",
+  "spiralled": "spiraled",
+  "spiralling": "spiraling",
+  "splendour": "splendor",
+  "splendours": "splendors",
+  "squirrelled": "squirreled",
+  "squirrelling": "squirreling",
+  "stabilisation": "stabilization",
+  "stabilise": "stabilize",
+  "stabilised": "stabilized",
+  "stabiliser": "stabilizer",
+  "stabilisers": "stabilizers",
+  "stabilises": "stabilizes",
+  "stabilising": "stabilizing",
+  "standardisation": "standardization",
+  "standardise": "standardize",
+  "standardised": "standardized",
+  "standardises": "standardizes",
+  "standardising": "standardizing",
+  "stencilled": "stenciled",
+  "stencilling": "stenciling",
+  "sterilisation": "sterilization",
+  "sterilisations": "sterilizations",
+  "sterilise": "sterilize",
+  "sterilised": "sterilized",
+  "steriliser": "sterilizer",
+  "sterilisers": "sterilizers",
+  "sterilises": "sterilizes",
+  "sterilising": "sterilizing",
+  "stigmatisation": "stigmatization",
+  "stigmatise": "stigmatize",
+  "stigmatised": "stigmatized",
+  "stigmatises": "stigmatizes",
+  "stigmatising": "stigmatizing",
+  "storey": "story",
+  "storeys": "stories",
+  "subsidisation": "subsidization",
+  "subsidise": "subsidize",
+  "subsidised": "subsidized",
+  "subsidiser": "subsidizer",
+  "subsidisers": "subsidizers",
+  "subsidises": "subsidizes",
+  "subsidising": "subsidizing",
+  "succour": "succor",
+  "succoured": "succored",
+  "succouring": "succoring",
+  "succours": "succors",
+  "sulphate": "sulfate",
+  "sulphates": "sulfates",
+  "sulphide": "sulfide",
+  "sulphides": "sulfides",
+  "sulphur": "sulfur",
+  "sulphurous": "sulfurous",
+  "summarise": "summarize",
+  "summarised": "summarized",
+  "summarises": "summarizes",
+  "summarising": "summarizing",
+  "swivelled": "swiveled",
+  "swivelling": "swiveling",
+  "symbolise": "symbolize",
+  "symbolised": "symbolized",
+  "symbolises": "symbolizes",
+  "symbolising": "symbolizing",
+  "sympathise": "sympathize",
+  "sympathised": "sympathized",
+  "sympathiser": "sympathizer",
+  "sympathisers": "sympathizers",
+  "sympathises": "sympathizes",
+  "sympathising": "sympathizing",
+  "synchronisation": "synchronization",
+  "synchronise": "synchronize",
+  "synchronised": "synchronized",
+  "synchronises": "synchronizes",
+  "synchronising": "synchronizing",
+  "synthesise": "synthesize",
+  "synthesised": "synthesized",
+  "synthesiser": "synthesizer",
+  "synthesisers": "synthesizers",
+  "synthesises": "synthesizes",
+  "synthesising": "synthesizing",
+  "syphon": "siphon",
+  "syphoned": "siphoned",
+  "syphoning": "siphoning",
+  "syphons": "siphons",
+  "systematisation": "systematization",
+  "systematise": "systematize",
+  "systematised": "systematized",
+  "systematises": "systematizes",
+  "systematising": "systematizing",
+  "tantalise": "tantalize",
+  "tantalised": "tantalized",
+  "tantalises": "tantalizes",
+  "tantalising": "tantalizing",
+  "tantalisingly": "tantalizingly",
+  "tasselled": "tasseled",
+  "technicolour": "technicolor",
+  "temporise": "temporize",
+  "temporised": "temporized",
+  "temporises": "temporizes",
+  "temporising": "temporizing",
+  "tenderise": "tenderize",
+  "tenderised": "tenderized",
+  "tenderises": "tenderizes",
+  "tenderising": "tenderizing",
+  "terrorise": "terrorize",
+  "terrorised": "terrorized",
+  "terrorises": "terrorizes",
+  "terrorising": "terrorizing",
+  "theatre": "theater",
+  "theatregoer": "theatergoer",
+  "theatregoers": "theatergoers",
+  "theatres": "theaters",
+  "theorise": "theorize",
+  "theorised": "theorized",
+  "theorises": "theorizes",
+  "theorising": "theorizing",
+  "tonne": "ton",
+  "tonnes": "tons",
+  "towelled": "toweled",
+  "towelling": "toweling",
+  "toxaemia": "toxemia",
+  "tranquillise": "tranquilize",
+  "tranquillised": "tranquilized",
+  "tranquilliser": "tranquilizer",
+  "tranquillisers": "tranquilizers",
+  "tranquillises": "tranquilizes",
+  "tranquillising": "tranquilizing",
+  "tranquillity": "tranquility",
+  "tranquillize": "tranquilize",
+  "tranquillized": "tranquilized",
+  "tranquillizer": "tranquilizer",
+  "tranquillizers": "tranquilizers",
+  "tranquillizes": "tranquilizes",
+  "tranquillizing": "tranquilizing",
+  "tranquilly": "tranquility",
+  "transistorised": "transistorized",
+  "traumatise": "traumatize",
+  "traumatised": "traumatized",
+  "traumatises": "traumatizes",
+  "traumatising": "traumatizing",
+  "travelled": "traveled",
+  "traveller": "traveler",
+  "travellers": "travelers",
+  "travelling": "traveling",
+  "travelog": "travelogue",
+  "travelogs": "travelogues",
+  "trialled": "trialed",
+  "trialling": "trialing",
+  "tricolour": "tricolor",
+  "tricolours": "tricolors",
+  "trivialise": "trivialize",
+  "trivialised": "trivialized",
+  "trivialises": "trivializes",
+  "trivialising": "trivializing",
+  "tumour": "tumor",
+  "tumours": "tumors",
+  "tunnelled": "tunneled",
+  "tunnelling": "tunneling",
+  "tyrannise": "tyrannize",
+  "tyrannised": "tyrannized",
+  "tyrannises": "tyrannizes",
+  "tyrannising": "tyrannizing",
+  "tyre": "tire",
+  "tyres": "tires",
+  "unauthorised": "unauthorized",
+  "uncivilised": "uncivilized",
+  "underutilised": "underutilized",
+  "unequalled": "unequaled",
+  "unfavourable": "unfavorable",
+  "unfavourably": "unfavorably",
+  "unionisation": "unionization",
+  "unionise": "unionize",
+  "unionised": "unionized",
+  "unionises": "unionizes",
+  "unionising": "unionizing",
+  "unorganised": "unorganized",
+  "unravelled": "unraveled",
+  "unravelling": "unraveling",
+  "unrecognisable": "unrecognizable",
+  "unrecognised": "unrecognized",
+  "unrivalled": "unrivaled",
+  "unsavoury": "unsavory",
+  "untrammelled": "untrammeled",
+  "urbanisation": "urbanization",
+  "urbanise": "urbanize",
+  "urbanised": "urbanized",
+  "urbanises": "urbanizes",
+  "urbanising": "urbanizing",
+  "utilisable": "utilizable",
+  "utilisation": "utilization",
+  "utilise": "utilize",
+  "utilised": "utilized",
+  "utilises": "utilizes",
+  "utilising": "utilizing",
+  "valour": "valor",
+  "vandalise": "vandalize",
+  "vandalised": "vandalized",
+  "vandalises": "vandalizes",
+  "vandalising": "vandalizing",
+  "vaporisation": "vaporization",
+  "vaporise": "vaporize",
+  "vaporised": "vaporized",
+  "vaporises": "vaporizes",
+  "vaporising": "vaporizing",
+  "vapour": "vapor",
+  "vapours": "vapors",
+  "verbalise": "verbalize",
+  "verbalised": "verbalized",
+  "verbalises": "verbalizes",
+  "verbalising": "verbalizing",
+  "victimisation": "victimization",
+  "victimise": "victimize",
+  "victimised": "victimized",
+  "victimises": "victimizes",
+  "victimising": "victimizing",
+  "videodisc": "videodisk",
+  "videodiscs": "videodisks",
+  "vigour": "vigor",
+  "visualisation": "visualization",
+  "visualisations": "visualizations",
+  "visualise": "visualize",
+  "visualised": "visualized",
+  "visualises": "visualizes",
+  "visualising": "visualizing",
+  "vocalisation": "vocalization",
+  "vocalisations": "vocalizations",
+  "vocalise": "vocalize",
+  "vocalised": "vocalized",
+  "vocalises": "vocalizes",
+  "vocalising": "vocalizing",
+  "vulcanised": "vulcanized",
+  "vulgarisation": "vulgarization",
+  "vulgarise": "vulgarize",
+  "vulgarised": "vulgarized",
+  "vulgarises": "vulgarizes",
+  "vulgarising": "vulgarizing",
+  "waggon": "wagon",
+  "waggons": "wagons",
+  "watercolour": "watercolor",
+  "watercolours": "watercolors",
+  "weaselled": "weaseled",
+  "weaselling": "weaseling",
+  "westernisation": "westernization",
+  "westernise": "westernize",
+  "westernised": "westernized",
+  "westernises": "westernizes",
+  "westernising": "westernizing",
+  "womanise": "womanize",
+  "womanised": "womanized",
+  "womaniser": "womanizer",
+  "womanisers": "womanizers",
+  "womanises": "womanizes",
+  "womanising": "womanizing",
+  "woollen": "woolen",
+  "woollens": "woolens",
+  "woollies": "woolies",
+  "woolly": "wooly",
+  "worshipped": "worshiped",
+  "worshipper": "worshiper",
+  "worshipping": "worshiping",
+  "yodelled": "yodeled",
+  "yodelling": "yodeling",
+  "yoghourt": "yogurt",
+  "yoghourts": "yogurts",
+  "yoghurt": "yogurt",
+  "yoghurts": "yogurts"
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3f005a4dad4989a1c813c1ead90f882357b184d9a25c972b814e45f94bd255da
+size 151098921

requirements.txt

@@ -0,0 +1,9 @@
+torch>=1.7
+torchaudio
+git+https://github.com/huggingface/transformers
+git+https://github.com/huggingface/datasets
+librosa
+jiwer
+evaluate>=0.3.0
+more-itertools
+tensorboard

run.sh

@@ -0,0 +1,42 @@
+python run_speech_recognition_seq2seq_streaming.py \
+--model_name_or_path="openai/whisper-tiny" \
+--dataset_name="mozilla-foundation/common_voice_11_0" \
+--dataset_config_name="bn" \
+--language="bengali" \
+--train_split_name="train+validation" \
+--eval_split_name="test" \
+--model_index_name="Whisper Tiny Bengali" \
+--output_dir="./" \
+--overwrite_output_dir \
+--max_steps="5000" \
+--per_device_train_batch_size="16" \
+--per_device_eval_batch_size="8" \
+--gradient_accumulation_steps="1" \
+--gradient_checkpointing="False" \
+--evaluation_strategy="steps" \
+--eval_steps="1000" \
+--save_strategy="steps" \
+--save_steps="1000" \
+--save_total_limit="5" \
+--learning_rate="1e-5" \
+--warmup_steps="500" \
+--logging_steps="25" \
+--weight_decay="0.01" \
+--load_best_model_at_end="True" \
+--metric_for_best_model="wer" \
+--greater_is_better="False" \
+--fp16="True" \
+--tf32="True" \
+--streaming="False" \
+--generation_max_length="225" \
+--length_column_name="input_length" \
+--max_duration_in_seconds="30" \
+--text_column_name="sentence" \
+--freeze_feature_encoder="False" \
+--report_to="tensorboard" \
+--do_train \
+--do_eval \
+--predict_with_generate \
+--do_normalize_eval \
+--use_auth_token \
+--push_to_hub

run_eval_whisper_streaming.py

@@ -0,0 +1,162 @@
+import argparse
+
+from transformers import pipeline
+from transformers.models.whisper.english_normalizer import BasicTextNormalizer
+from datasets import load_dataset, Audio
+import evaluate
+
+wer_metric = evaluate.load("wer")
+
+
+def is_target_text_in_range(ref):
+    if ref.strip() == "ignore time segment in scoring":
+        return False
+    else:
+        return ref.strip() != ""
+
+
+def get_text(sample):
+    if "text" in sample:
+        return sample["text"]
+    elif "sentence" in sample:
+        return sample["sentence"]
+    elif "normalized_text" in sample:
+        return sample["normalized_text"]
+    elif "transcript" in sample:
+        return sample["transcript"]
+    elif "transcription" in sample:
+        return sample["transcription"]
+    else:
+        raise ValueError(
+            f"Expected transcript column of either 'text', 'sentence', 'normalized_text' or 'transcript'. Got sample of "
+            ".join{sample.keys()}. Ensure a text column name is present in the dataset."
+        )
+
+
+whisper_norm = BasicTextNormalizer()
+
+
+def normalise(batch):
+    batch["norm_text"] = whisper_norm(get_text(batch))
+    return batch
+
+
+def data(dataset):
+    for i, item in enumerate(dataset):
+        yield {**item["audio"], "reference": item["norm_text"]}
+
+
+def main(args):
+    batch_size = args.batch_size
+    whisper_asr = pipeline(
+        "automatic-speech-recognition", model=args.model_id, device=args.device
+    )
+
+    whisper_asr.model.config.forced_decoder_ids = (
+        whisper_asr.tokenizer.get_decoder_prompt_ids(
+            language=args.language, task="transcribe"
+        )
+    )
+
+    dataset = load_dataset(
+        args.dataset,
+        args.config,
+        split=args.split,
+        streaming=args.streaming,
+        use_auth_token=True,
+    )
+
+    # Only uncomment for debugging
+    dataset = dataset.take(args.max_eval_samples)
+
+    dataset = dataset.cast_column("audio", Audio(sampling_rate=16000))
+    dataset = dataset.map(normalise)
+    dataset = dataset.filter(is_target_text_in_range, input_columns=["norm_text"])
+
+    predictions = []
+    references = []
+
+    # run streamed inference
+    for out in whisper_asr(data(dataset), batch_size=batch_size):
+        predictions.append(whisper_norm(out["text"]))
+        references.append(out["reference"][0])
+
+    wer = wer_metric.compute(references=references, predictions=predictions)
+    wer = round(100 * wer, 2)
+
+    print("WER:", wer)
+    evaluate.push_to_hub(
+        model_id=args.model_id,
+        metric_value=wer,
+        metric_type="wer",
+        metric_name="WER",
+        dataset_name=args.dataset,
+        dataset_type=args.dataset,
+        dataset_split=args.split,
+        dataset_config=args.config,
+        task_type="automatic-speech-recognition",
+        task_name="Automatic Speech Recognition"
+    )
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+
+    parser.add_argument(
+        "--model_id",
+        type=str,
+        required=True,
+        help="Model identifier. Should be loadable with 🤗 Transformers",
+    )
+    parser.add_argument(
+        "--dataset",
+        type=str,
+        default="mozilla-foundation/common_voice_11_0",
+        help="Dataset name to evaluate the `model_id`. Should be loadable with 🤗 Datasets",
+    )
+    parser.add_argument(
+        "--config",
+        type=str,
+        required=True,
+        help="Config of the dataset. *E.g.* `'en'` for the English split of Common Voice",
+    )
+    parser.add_argument(
+        "--split",
+        type=str,
+        default="test",
+        help="Split of the dataset. *E.g.* `'test'`",
+    )
+
+    parser.add_argument(
+        "--device",
+        type=int,
+        default=-1,
+        help="The device to run the pipeline on. -1 for CPU (default), 0 for the first GPU and so on.",
+    )
+    parser.add_argument(
+        "--batch_size",
+        type=int,
+        default=16,
+        help="Number of samples to go through each streamed batch.",
+    )
+    parser.add_argument(
+        "--max_eval_samples",
+        type=int,
+        default=None,
+        help="Number of samples to be evaluated. Put a lower number e.g. 64 for testing this script.",
+    )
+    parser.add_argument(
+        "--streaming",
+        type=bool,
+        default=True,
+        help="Choose whether you'd like to download the entire dataset or stream it during the evaluation.",
+    )
+    parser.add_argument(
+        "--language",
+        type=str,
+        required=True,
+        help="Two letter language code for the transcription language, e.g. use 'en' for English.",
+    )
+    args = parser.parse_args()
+
+    main(args)

run_speech_recognition_seq2seq_streaming.py

@@ -0,0 +1,635 @@
+#!/usr/bin/env python
+# coding=utf-8
+# Copyright 2022 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+Fine-tuning the library models for sequence to sequence speech recognition
+with 🤗 Datasets' streaming mode.
+"""
+# You can also adapt this script for your own sequence to sequence speech
+# recognition task. Pointers for this are left as comments.
+
+import logging
+import os
+import sys
+from dataclasses import dataclass, field
+from typing import Any, Dict, List, Optional, Union
+
+import datasets
+import torch
+from datasets import DatasetDict, IterableDatasetDict, interleave_datasets, load_dataset
+from torch.utils.data import IterableDataset
+
+abs_path = os.path.abspath('.')
+base_dir = os.path.dirname(abs_path)
+os.environ['TRANSFORMERS_CACHE'] = os.path.join(base_dir, 'models_cache')
+os.environ['HF_DATASETS_CACHE'] = os.path.join(base_dir, 'datasets_cache')
+
+
+import evaluate
+import transformers
+from transformers import (
+    AutoConfig,
+    AutoFeatureExtractor,
+    AutoModelForSpeechSeq2Seq,
+    AutoProcessor,
+    AutoTokenizer,
+    HfArgumentParser,
+    Seq2SeqTrainer,
+    Seq2SeqTrainingArguments,
+    TrainerCallback,
+    set_seed,
+)
+from transformers.models.whisper.english_normalizer import BasicTextNormalizer
+from transformers.trainer_pt_utils import IterableDatasetShard
+from transformers.trainer_utils import get_last_checkpoint, is_main_process
+from transformers.utils import check_min_version, send_example_telemetry
+from transformers.utils.versions import require_version
+
+
+# Will error if the minimal version of Transformers is not installed. Remove at your own risks.
+check_min_version("4.25.0.dev0")
+
+require_version("datasets>=1.18.2", "To fix: pip install -r examples/pytorch/speech-recognition/requirements.txt")
+
+logger = logging.getLogger(__name__)
+
+
+@dataclass
+class ModelArguments:
+    """
+    Arguments pertaining to which model/config/tokenizer we are going to fine-tune from.
+    """
+
+    model_name_or_path: str = field(
+        metadata={"help": "Path to pretrained model or model identifier from huggingface.co/models"}
+    )
+    config_name: Optional[str] = field(
+        default=None, metadata={"help": "Pretrained config name or path if not the same as model_name"}
+    )
+    tokenizer_name: Optional[str] = field(
+        default=None, metadata={"help": "Pretrained tokenizer name or path if not the same as model_name"}
+    )
+    feature_extractor_name: Optional[str] = field(
+        default=None, metadata={"help": "feature extractor name or path if not the same as model_name"}
+    )
+    cache_dir: Optional[str] = field(
+        default=None,
+        metadata={"help": "Where to store the pretrained models downloaded from huggingface.co"},
+    )
+    use_fast_tokenizer: bool = field(
+        default=True,
+        metadata={"help": "Whether to use one of the fast tokenizer (backed by the tokenizers library) or not."},
+    )
+    model_revision: str = field(
+        default="main",
+        metadata={"help": "The specific model version to use (can be a branch name, tag name or commit id)."},
+    )
+    use_auth_token: bool = field(
+        default=False,
+        metadata={
+            "help": (
+                "Will use the token generated when running `huggingface-cli login` (necessary to use this script "
+                "with private models)."
+            )
+        },
+    )
+    freeze_feature_encoder: bool = field(
+        default=True, metadata={"help": "Whether to freeze the feature encoder layers of the model."}
+    )
+    freeze_encoder: bool = field(
+        default=False, metadata={"help": "Whether to freeze the entire encoder of the seq2seq model."}
+    )
+    forced_decoder_ids: List[List[int]] = field(
+        default=None,
+        metadata={
+            "help": (
+                "A list of pairs of integers which indicates a mapping from generation indices to token indices "
+                "that will be forced before sampling. For example, [[0, 123]] means the first generated token "
+                "will always be a token of index 123."
+            )
+        },
+    )
+    suppress_tokens: List[int] = field(
+        default=None, metadata={"help": "A list of tokens that will be suppressed at generation."}
+    )
+    model_index_name: str = field(default=None, metadata={"help": "Pretty name for the model card."})
+
+
+@dataclass
+class DataTrainingArguments:
+    """
+    Arguments pertaining to what data we are going to input our model for training and eval.
+    """
+
+    dataset_name: str = field(
+        default=None, metadata={"help": "The name of the dataset to use (via the datasets library)."}
+    )
+    dataset_config_name: Optional[str] = field(
+        default=None, metadata={"help": "The configuration name of the dataset to use (via the datasets library)."}
+    )
+    text_column: Optional[str] = field(
+        default=None,
+        metadata={"help": "The name of the column in the datasets containing the full texts (for summarization)."},
+    )
+    max_train_samples: Optional[int] = field(
+        default=None,
+        metadata={
+            "help": (
+                "For debugging purposes or quicker training, truncate the number of training examples to this "
+                "value if set."
+            )
+        },
+    )
+    max_eval_samples: Optional[int] = field(
+        default=None,
+        metadata={
+            "help": (
+                "For debugging purposes or quicker training, truncate the number of evaluation examples to this "
+                "value if set."
+            )
+        },
+    )
+    audio_column_name: str = field(
+        default="audio",
+        metadata={"help": "The name of the dataset column containing the audio data. Defaults to 'audio'"},
+    )
+    text_column_name: str = field(
+        default="text",
+        metadata={"help": "The name of the dataset column containing the text data. Defaults to 'text'"},
+    )
+    max_duration_in_seconds: float = field(
+        default=20.0,
+        metadata={
+            "help": (
+                "Truncate audio files that are longer than `max_duration_in_seconds` seconds to"
+                " 'max_duration_in_seconds`"
+            )
+        },
+    )
+    min_duration_in_seconds: float = field(
+        default=0.0, metadata={"help": "Filter audio files that are shorter than `min_duration_in_seconds` seconds"}
+    )
+    train_split_name: str = field(
+        default="train",
+        metadata={
+            "help": "The name of the training data set split to use (via the datasets library). Defaults to 'train'"
+        },
+    )
+    eval_split_name: str = field(
+        default="test",
+        metadata={
+            "help": "The name of the training data set split to use (via the datasets library). Defaults to 'train'"
+        },
+    )
+    do_lower_case: bool = field(
+        default=False,
+        metadata={"help": "Whether the target text should be lower cased."},
+    )
+    do_remove_punctuation: bool = field(
+        default=False,
+        metadata={"help": "Whether the target text should be striped of punctuation."},
+    )
+    do_normalize_eval: bool = field(
+        default=True,
+        metadata={"help": "Whether to normalise the references and predictions in the eval WER calculation."},
+    )
+    language: str = field(
+        default=None,
+        metadata={
+            "help": (
+                "Language for multilingual fine-tuning. This argument should be set for multilingual fine-tuning "
+                "only. For English speech recognition, it should be set to `None`."
+            )
+        },
+    )
+    task: str = field(
+        default="transcribe",
+        metadata={"help": "Task, either `transcribe` for speech recognition or `translate` for speech translation."},
+    )
+    shuffle_buffer_size: Optional[int] = field(
+        default=500,
+        metadata={
+            "help": (
+                "The number of streamed examples to download before shuffling them. The large the buffer, "
+                "the closer it is to real offline shuffling."
+            )
+        },
+    )
+    streaming: bool = field(
+        default=True,
+        metadata={"help": "Whether to use streaming mode to load and pre-process the data."},
+    )
+
+
+@dataclass
+class DataCollatorSpeechSeq2SeqWithPadding:
+    """
+    Data collator that will dynamically pad the inputs received.
+    Args:
+        processor ([`WhisperProcessor`])
+            The processor used for processing the data.
+        decoder_start_token_id (`int`)
+            The begin-of-sentence of the decoder.
+    """
+
+    processor: Any
+    decoder_start_token_id: int
+
+    def __call__(self, features: List[Dict[str, Union[List[int], torch.Tensor]]]) -> Dict[str, torch.Tensor]:
+        # split inputs and labels since they have to be of different lengths and need
+        # different padding methods
+        model_input_name = self.processor.model_input_names[0]
+        input_features = [{model_input_name: feature[model_input_name]} for feature in features]
+        label_features = [{"input_ids": feature["labels"]} for feature in features]
+
+        batch = self.processor.feature_extractor.pad(input_features, return_tensors="pt")
+
+        labels_batch = self.processor.tokenizer.pad(label_features, return_tensors="pt")
+
+        # replace padding with -100 to ignore loss correctly
+        labels = labels_batch["input_ids"].masked_fill(labels_batch.attention_mask.ne(1), -100)
+
+        # if bos token is appended in previous tokenization step,
+        # cut bos token here as it's append later anyways
+        if (labels[:, 0] == self.decoder_start_token_id).all().cpu().item():
+            labels = labels[:, 1:]
+
+        batch["labels"] = labels
+
+        return batch
+
+
+def load_maybe_streaming_dataset(dataset_name, dataset_config_name, split="train", streaming=True, **kwargs):
+    """
+    Utility function to load a dataset in streaming mode. For datasets with multiple splits,
+    each split is loaded individually and then splits combined by taking alternating examples from
+    each (interleaving).
+    """
+    if "+" in split:
+        # load multiple splits separated by the `+` symbol with streaming mode
+        dataset_splits = [
+            load_dataset(dataset_name, dataset_config_name, split=split_name, streaming=streaming, **kwargs)
+            for split_name in split.split("+")
+        ]
+        # interleave multiple splits to form one dataset
+        interleaved_dataset = interleave_datasets(dataset_splits)
+        return interleaved_dataset
+    else:
+        # load a single split *with* streaming mode
+        dataset = load_dataset(dataset_name, dataset_config_name, split=split, streaming=streaming, **kwargs)
+        return dataset
+
+
+def main():
+    # 1. Parse input arguments
+    # See all possible arguments in src/transformers/training_args.py
+    # or by passing the --help flag to this script.
+    # We now keep distinct sets of args, for a cleaner separation of concerns.
+    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, Seq2SeqTrainingArguments))
+
+    if len(sys.argv) == 2 and sys.argv[1].endswith(".json"):
+        # If we pass only one argument to the script and it's the path to a json file,
+        # let's parse it to get our arguments.
+        model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))
+    else:
+        model_args, data_args, training_args = parser.parse_args_into_dataclasses()
+
+    # Sending telemetry. Tracking the example usage helps us better allocate resources to maintain them. The
+    # information sent is the one passed as arguments along with your Python/PyTorch versions.
+    send_example_telemetry("run_speech_recognition_seq2seq_streaming", model_args, data_args)
+
+    # 2. Setup logging
+    logging.basicConfig(
+        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
+        datefmt="%m/%d/%Y %H:%M:%S",
+        handlers=[logging.StreamHandler(sys.stdout)],
+    )
+    log_level = training_args.get_process_log_level()
+    logger.setLevel(log_level)
+    datasets.utils.logging.set_verbosity(log_level)
+    transformers.utils.logging.set_verbosity(log_level)
+    transformers.utils.logging.enable_default_handler()
+    transformers.utils.logging.enable_explicit_format()
+
+    logger.setLevel(logging.INFO if is_main_process(training_args.local_rank) else logging.WARN)
+
+    # Log on each process the small summary:
+    logger.warning(
+        f"Process rank: {training_args.local_rank}, device: {training_args.device}, n_gpu: {training_args.n_gpu}"
+        f"distributed training: {bool(training_args.local_rank != -1)}, 16-bits training: {training_args.fp16}"
+    )
+    logger.info(f"Training/evaluation parameters {training_args}")
+
+    # Set the verbosity to info of the Transformers logger (on main process only):
+    if is_main_process(training_args.local_rank):
+        transformers.utils.logging.set_verbosity_info()
+    logger.info("Training/evaluation parameters %s", training_args)
+
+    # 3. Detecting last checkpoint and eventually continue from last checkpoint
+    last_checkpoint = None
+    if os.path.isdir(training_args.output_dir) and training_args.do_train and not training_args.overwrite_output_dir:
+        last_checkpoint = get_last_checkpoint(training_args.output_dir)
+        if last_checkpoint is None and len(os.listdir(training_args.output_dir)) > 0:
+            raise ValueError(
+                f"Output directory ({training_args.output_dir}) already exists and is not empty. "
+                "Use --overwrite_output_dir to overcome."
+            )
+        elif last_checkpoint is not None and training_args.resume_from_checkpoint is None:
+            logger.info(
+                f"Checkpoint detected, resuming training at {last_checkpoint}. To avoid this behavior, change "
+                "the `--output_dir` or add `--overwrite_output_dir` to train from scratch."
+            )
+
+    # Set seed before initializing model.
+    set_seed(training_args.seed)
+
+    # 4. Load dataset
+    raw_datasets = IterableDatasetDict() if data_args.streaming else DatasetDict()
+
+    if training_args.do_train:
+        raw_datasets["train"] = load_maybe_streaming_dataset(
+            data_args.dataset_name,
+            data_args.dataset_config_name,
+            split=data_args.train_split_name,
+            use_auth_token=True if model_args.use_auth_token else None,
+            streaming=data_args.streaming,
+        )
+
+    if training_args.do_eval:
+        raw_datasets["eval"] = load_maybe_streaming_dataset(
+            data_args.dataset_name,
+            data_args.dataset_config_name,
+            split=data_args.eval_split_name,
+            use_auth_token=True if model_args.use_auth_token else None,
+            streaming=data_args.streaming,
+        )
+
+    raw_datasets_features = list(next(iter(raw_datasets.values())).features.keys())
+
+    if data_args.audio_column_name not in raw_datasets_features:
+        raise ValueError(
+            f"--audio_column_name '{data_args.audio_column_name}' not found in dataset '{data_args.dataset_name}'. "
+            "Make sure to set `--audio_column_name` to the correct audio column - one of "
+            f"{', '.join(raw_datasets_features)}."
+        )
+
+    if data_args.text_column_name not in raw_datasets_features:
+        raise ValueError(
+            f"--text_column_name {data_args.text_column_name} not found in dataset '{data_args.dataset_name}'. "
+            "Make sure to set `--text_column_name` to the correct text column - one of "
+            f"{', '.join(raw_datasets_features)}."
+        )
+
+    # 5. Load pretrained model, tokenizer, and feature extractor
+    #
+    # Distributed training:
+    # The .from_pretrained methods guarantee that only one local process can concurrently
+    config = AutoConfig.from_pretrained(
+        model_args.config_name if model_args.config_name else model_args.model_name_or_path,
+        cache_dir=model_args.cache_dir,
+        revision=model_args.model_revision,
+        use_auth_token=True if model_args.use_auth_token else None,
+    )
+
+    config.update({"forced_decoder_ids": model_args.forced_decoder_ids, "suppress_tokens": model_args.suppress_tokens})
+
+    if training_args.gradient_checkpointing:
+        config.update({"use_cache": False})
+
+    feature_extractor = AutoFeatureExtractor.from_pretrained(
+        model_args.feature_extractor_name if model_args.feature_extractor_name else model_args.model_name_or_path,
+        cache_dir=model_args.cache_dir,
+        revision=model_args.model_revision,
+        use_auth_token=True if model_args.use_auth_token else None,
+    )
+    tokenizer = AutoTokenizer.from_pretrained(
+        model_args.tokenizer_name if model_args.tokenizer_name else model_args.model_name_or_path,
+        cache_dir=model_args.cache_dir,
+        use_fast=model_args.use_fast_tokenizer,
+        revision=model_args.model_revision,
+        use_auth_token=True if model_args.use_auth_token else None,
+    )
+    model = AutoModelForSpeechSeq2Seq.from_pretrained(
+        model_args.model_name_or_path,
+        config=config,
+        cache_dir=model_args.cache_dir,
+        revision=model_args.model_revision,
+        use_auth_token=True if model_args.use_auth_token else None,
+    )
+
+    if model.config.decoder_start_token_id is None:
+        raise ValueError("Make sure that `config.decoder_start_token_id` is correctly defined")
+
+    if model_args.freeze_feature_encoder:
+        model.freeze_feature_encoder()
+
+    if model_args.freeze_encoder:
+        model.freeze_encoder()
+
+    if data_args.language is not None:
+        # We only need to set the task id when the language is specified (i.e. in a multilingual setting)
+        tokenizer.set_prefix_tokens(language=data_args.language, task=data_args.task)
+
+    # 6. Resample speech dataset if necessary
+    dataset_sampling_rate = next(iter(raw_datasets.values())).features[data_args.audio_column_name].sampling_rate
+    if dataset_sampling_rate != feature_extractor.sampling_rate:
+        raw_datasets = raw_datasets.cast_column(
+            data_args.audio_column_name, datasets.features.Audio(sampling_rate=feature_extractor.sampling_rate)
+        )
+
+    # 7. Preprocessing the datasets.
+    # We need to read the audio files as arrays and tokenize the targets.
+    max_input_length = data_args.max_duration_in_seconds * feature_extractor.sampling_rate
+    min_input_length = data_args.min_duration_in_seconds * feature_extractor.sampling_rate
+    audio_column_name = data_args.audio_column_name
+    text_column_name = data_args.text_column_name
+    model_input_name = feature_extractor.model_input_names[0]
+    do_lower_case = data_args.do_lower_case
+    do_remove_punctuation = data_args.do_remove_punctuation
+    normalizer = BasicTextNormalizer()  # 'official' text normalizer from OpenAI
+
+    if data_args.max_train_samples is not None:
+        raw_datasets["train"] = (
+            raw_datasets["train"].take(data_args.max_train_samples)
+            if data_args.streaming
+            else raw_datasets["train"].select(range(data_args.max_train_samples))
+        )
+
+    if data_args.max_eval_samples is not None:
+        raw_datasets["eval"] = (
+            raw_datasets["eval"].take(data_args.max_eval_samples)
+            if data_args.streaming
+            else raw_datasets["eval"].select(range(data_args.max_eval_samples))
+        )
+
+    def prepare_dataset(batch):
+        # process audio
+        sample = batch[audio_column_name]
+        inputs = feature_extractor(sample["array"], sampling_rate=sample["sampling_rate"])
+        # process audio length
+        batch[model_input_name] = inputs.get(model_input_name)[0]
+        batch["input_length"] = len(sample["array"])
+
+        # process targets
+        input_str = batch[text_column_name].lower() if do_lower_case else batch[text_column_name]
+        if do_remove_punctuation:
+            input_str = normalizer(input_str).strip()
+        batch["labels"] = tokenizer(input_str).input_ids
+        return batch
+
+    with training_args.main_process_first(desc="dataset map pre-processing"):
+        vectorized_datasets = raw_datasets.map(
+            prepare_dataset,
+            remove_columns=raw_datasets_features,
+        ).with_format("torch")
+
+        if training_args.do_train and data_args.streaming:
+            # manually shuffle if streaming (done by the trainer for non-streaming)
+            vectorized_datasets["train"] = vectorized_datasets["train"].shuffle(
+                buffer_size=data_args.shuffle_buffer_size,
+                seed=training_args.seed,
+            )
+
+    # filter training data that is shorter than min_input_length or longer than
+    # max_input_length
+    def is_audio_in_length_range(length):
+        return min_input_length < length < max_input_length
+
+    if training_args.do_train:
+        vectorized_datasets["train"] = vectorized_datasets["train"].filter(
+            is_audio_in_length_range,
+            input_columns=["input_length"],
+        )
+
+    # 8. Load Metric
+    metric = evaluate.load("wer")
+    do_normalize_eval = data_args.do_normalize_eval
+
+    def compute_metrics(pred):
+        pred_ids = pred.predictions
+
+        pred.label_ids[pred.label_ids == -100] = tokenizer.pad_token_id
+
+        pred_str = tokenizer.batch_decode(pred_ids, skip_special_tokens=True)
+        # we do not want to group tokens when computing the metrics
+        label_str = tokenizer.batch_decode(pred.label_ids, skip_special_tokens=True)
+
+        if do_normalize_eval:
+            pred_str = [normalizer(pred) for pred in pred_str]
+            label_str = [normalizer(label) for label in label_str]
+            # filtering step to only evaluate the samples that correspond to non-zero references:
+            pred_str = [pred_str[i] for i in range(len(pred_str)) if len(label_str[i]) > 0]
+            label_str = [label_str[i] for i in range(len(label_str)) if len(label_str[i]) > 0]
+
+        wer = 100 * metric.compute(predictions=pred_str, references=label_str)
+
+        return {"wer": wer}
+
+    # 9. Create a single speech processor
+    if is_main_process(training_args.local_rank):
+        # save feature extractor, tokenizer and config
+        feature_extractor.save_pretrained(training_args.output_dir)
+        tokenizer.save_pretrained(training_args.output_dir)
+        config.save_pretrained(training_args.output_dir)
+
+    processor = AutoProcessor.from_pretrained(training_args.output_dir)
+
+    # 10. Define data collator
+    data_collator = DataCollatorSpeechSeq2SeqWithPadding(
+        processor=processor,
+        decoder_start_token_id=model.config.decoder_start_token_id,
+    )
+
+    # 11. Configure Trainer
+    # Trainer callback to reinitialise and reshuffle the streamable datasets at the beginning of each epoch
+    # Only required for streaming: Trainer automatically shuffles non-streaming datasets
+    class ShuffleCallback(TrainerCallback):
+        def on_epoch_begin(self, args, state, control, train_dataloader, **kwargs):
+            if isinstance(train_dataloader.dataset, IterableDatasetShard):
+                pass  # set_epoch() is handled by the Trainer
+            elif isinstance(train_dataloader.dataset, IterableDataset):
+                train_dataloader.dataset.set_epoch(train_dataloader.dataset._epoch + 1)
+
+    # Initialize Trainer
+    trainer = Seq2SeqTrainer(
+        model=model,
+        args=training_args,
+        train_dataset=vectorized_datasets["train"] if training_args.do_train else None,
+        eval_dataset=vectorized_datasets["eval"] if training_args.do_eval else None,
+        tokenizer=feature_extractor,
+        data_collator=data_collator,
+        compute_metrics=compute_metrics if training_args.predict_with_generate else None,
+        callbacks=[ShuffleCallback()] if data_args.streaming else None,
+    )
+
+    # 12. Training
+    if training_args.do_train:
+        checkpoint = None
+        if training_args.resume_from_checkpoint is not None:
+            checkpoint = training_args.resume_from_checkpoint
+        elif last_checkpoint is not None:
+            checkpoint = last_checkpoint
+        train_result = trainer.train(resume_from_checkpoint=checkpoint)
+        trainer.save_model()  # Saves the feature extractor too for easy upload
+
+        metrics = train_result.metrics
+        if data_args.max_train_samples:
+            metrics["train_samples"] = data_args.max_train_samples
+        trainer.log_metrics("train", metrics)
+        trainer.save_metrics("train", metrics)
+        trainer.save_state()
+
+    # 13. Evaluation
+    results = {}
+    if training_args.do_eval:
+        logger.info("*** Evaluate ***")
+        metrics = trainer.evaluate(
+            metric_key_prefix="eval",
+            max_length=training_args.generation_max_length,
+            num_beams=training_args.generation_num_beams,
+        )
+        if data_args.max_eval_samples:
+            metrics["eval_samples"] = data_args.max_eval_samples
+
+        trainer.log_metrics("eval", metrics)
+        trainer.save_metrics("eval", metrics)
+
+    # 14. Write Training Stats
+    kwargs = {
+        "finetuned_from": model_args.model_name_or_path,
+        "tasks": "automatic-speech-recognition",
+        "tags": "whisper-event",
+    }
+    if data_args.dataset_name is not None:
+        kwargs["dataset_tags"] = data_args.dataset_name
+        if data_args.dataset_config_name is not None:
+            kwargs["dataset"] = f"{data_args.dataset_name} {data_args.dataset_config_name}"
+        else:
+            kwargs["dataset"] = data_args.dataset_name
+        if "common_voice" in data_args.dataset_name:
+            kwargs["language"] = data_args.dataset_config_name.split('-')[0]
+        if model_args.model_index_name is not None:
+            kwargs["model_name"] = model_args.model_index_name
+
+    if training_args.push_to_hub:
+        trainer.push_to_hub(**kwargs)
+    else:
+        trainer.create_model_card(**kwargs)
+
+    return results
+
+
+if __name__ == "__main__":
+    main()

runs/Jan18_12-56-26_mamun-desktop/1674026928.3192365/events.out.tfevents.1674026928.mamun-desktop.5642.1

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a752d0c2162d3e9226e27a8d37822d8007f28f7178b485e4bd466dc60f788d8f
+size 5875

runs/Jan18_12-56-26_mamun-desktop/events.out.tfevents.1674026928.mamun-desktop.5642.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:07806849165ddea662c9780f7c0bb11b65c191d3ccc0320ad5600c414d7dc361
+size 4260

runs/Jan18_13-42-16_mamun-desktop/1674028391.1765018/events.out.tfevents.1674028391.mamun-desktop.414182.1

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a63bac39bc5a3d29bca6e7a401e360dafb8813717c9890267a8f395ca7e256e3
+size 5875

runs/Jan18_13-42-16_mamun-desktop/events.out.tfevents.1674028391.mamun-desktop.414182.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:17f067288b429c209aaf2cbd12763ab01c1f021781c474e4068281c93470d1b6
+size 4260

runs/Jan18_13-54-03_mamun-desktop/1674028489.236082/events.out.tfevents.1674028489.mamun-desktop.550378.1

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f211e73303fc1098f02ed7737614676c2e4f3d917186cf2b33a88e1117049567
+size 5875

runs/Jan18_13-54-03_mamun-desktop/events.out.tfevents.1674028489.mamun-desktop.550378.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:51c2b7954ef29f9fe55c7761c5dd2b7827864b827daf0f949452740dcf007367
+size 4258

runs/Jan18_13-55-30_mamun-desktop/1674028577.9689484/events.out.tfevents.1674028577.mamun-desktop.550597.1

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e020f11e8fffdc3cd9e039164e9943a9c4a7ac8e5aef998e3e751c3daaa3f498
+size 5875

runs/Jan18_13-55-30_mamun-desktop/events.out.tfevents.1674028577.mamun-desktop.550597.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dab3d94e5d3f3655f9784d5e92226b2181945d4e9f36a220dc506598a6bd8fec
+size 4258

runs/Jan18_13-56-56_mamun-desktop/1674028663.0261796/events.out.tfevents.1674028663.mamun-desktop.550835.1

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:002842f0fca227a5e8973c4aeec7752b247f405e3663209eed1ca80e4c796d7b
+size 5875

runs/Jan18_13-56-56_mamun-desktop/events.out.tfevents.1674028663.mamun-desktop.550835.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6e8eb361652a2b8cbd2a8fd404c4777c537344f5486517f294b434cd30d1193f
+size 10840

special_tokens_map.json

@@ -0,0 +1,133 @@
+{
+  "additional_special_tokens": [
+    "<|endoftext|>",
+    "<|startoftranscript|>",
+    "<|en|>",
+    "<|zh|>",
+    "<|de|>",
+    "<|es|>",
+    "<|ru|>",
+    "<|ko|>",
+    "<|fr|>",
+    "<|ja|>",
+    "<|pt|>",
+    "<|tr|>",
+    "<|pl|>",
+    "<|ca|>",
+    "<|nl|>",
+    "<|ar|>",
+    "<|sv|>",
+    "<|it|>",
+    "<|id|>",
+    "<|hi|>",
+    "<|fi|>",
+    "<|vi|>",
+    "<|iw|>",
+    "<|uk|>",
+    "<|el|>",
+    "<|ms|>",
+    "<|cs|>",
+    "<|ro|>",
+    "<|da|>",
+    "<|hu|>",
+    "<|ta|>",
+    "<|no|>",
+    "<|th|>",
+    "<|ur|>",
+    "<|hr|>",
+    "<|bg|>",
+    "<|lt|>",
+    "<|la|>",
+    "<|mi|>",
+    "<|ml|>",
+    "<|cy|>",
+    "<|sk|>",
+    "<|te|>",
+    "<|fa|>",
+    "<|lv|>",
+    "<|bn|>",
+    "<|sr|>",
+    "<|az|>",
+    "<|sl|>",
+    "<|kn|>",
+    "<|et|>",
+    "<|mk|>",
+    "<|br|>",
+    "<|eu|>",
+    "<|is|>",
+    "<|hy|>",
+    "<|ne|>",
+    "<|mn|>",
+    "<|bs|>",
+    "<|kk|>",
+    "<|sq|>",
+    "<|sw|>",
+    "<|gl|>",
+    "<|mr|>",
+    "<|pa|>",
+    "<|si|>",
+    "<|km|>",
+    "<|sn|>",
+    "<|yo|>",
+    "<|so|>",
+    "<|af|>",
+    "<|oc|>",
+    "<|ka|>",
+    "<|be|>",
+    "<|tg|>",
+    "<|sd|>",
+    "<|gu|>",
+    "<|am|>",
+    "<|yi|>",
+    "<|lo|>",
+    "<|uz|>",
+    "<|fo|>",
+    "<|ht|>",
+    "<|ps|>",
+    "<|tk|>",
+    "<|nn|>",
+    "<|mt|>",
+    "<|sa|>",
+    "<|lb|>",
+    "<|my|>",
+    "<|bo|>",
+    "<|tl|>",
+    "<|mg|>",
+    "<|as|>",
+    "<|tt|>",
+    "<|haw|>",
+    "<|ln|>",
+    "<|ha|>",
+    "<|ba|>",
+    "<|jw|>",
+    "<|su|>",
+    "<|translate|>",
+    "<|transcribe|>",
+    "<|startoflm|>",
+    "<|startofprev|>",
+    "<|nocaptions|>",
+    "<|notimestamps|>"
+  ],
+  "bos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<|endoftext|>",
+  "unk_token": {
+    "content": "",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,36 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "bos_token": {
+    "__type": "AddedToken",
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "__type": "AddedToken",
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "errors": "replace",
+  "model_max_length": 1024,
+  "name_or_path": "openai/whisper-tiny",
+  "pad_token": null,
+  "processor_class": "WhisperProcessor",
+  "return_attention_mask": false,
+  "special_tokens_map_file": null,
+  "tokenizer_class": "WhisperTokenizer",
+  "unk_token": {
+    "__type": "AddedToken",
+    "content": "",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6d7d372554ba8ea6218aeec2b8a0315e296c1d8621c01416e476391211ff1dc8
+size 3579