Fine-tuning Mozilla DeepSpeech for the Indian Accent | by Abhiroop Talasila | Towards Data Science

One of the first attempts to make computers recognize speech was focused on recognizing numbers! Bell Laboratories in 1952 designed the Audrey System, which could recognize a single voice speaking digits. There have been numerous other experiments since then which are well documented in this Wikipedia article. Fast forward to today, we have state-of-the-art Automatic Speech Recognition Engines (ASR) like Apple’s Siri, Google Assistant, and Amazon’s Alexa.
For a long-time, Google’s Speech-to-text API (STT) was the de facto choice for any ASR task. This slowly changed when open-source alternatives like Mozilla DeepSpeech came out in late 2017. It’s based on the original Deep Speech research paper by Baidu and is trained using (mostly) American English datasets, resulting in poor generalizability to other English accents.
For a recent internship, I had to integrate an ASR engine for a video-conferencing platform, which was to be used majorly by the Indian population. We were preferably looking for open-source alternatives, but most of the general ones performed poorly in real-time meetings. That’s when I came across DeepSpeech and the Indic TTS project by IITM.
The Indic dataset contains more than 50 GB of speech samples with speakers from 13 Indian states. It comprises of 10000+ spoken English sentences of both Male and Female native speakers. These files are available in .wav format along with the corresponding text.
In this article, I’ll walk you through the process of fine-tuning DeepSpeech using the Indic dataset, but you can easily follow this for other English datasets too. You can signup on the IITM website and request them for the dataset.
Prerequisites: Familiarity with ASR Engines, Speech Processing, and a basic understanding of Recurrent Neural Networks and TensorFlow.
Note: All my training and pre-processing was done on Google Colab with DeepSpeech version 0.7.4
After you request the dataset, IITM will give you access to their Google Drive links for seven days. Because I needed the data for an extended period, I transferred all the ZIP files to a Google Cloud Bucket. Each ZIP file will have a folder containing the .wav files and the corresponding metadata file named txt.done.data.
We need to process the metadata file and generate train/dev/test splits for the dataset. We can either train the model for a single state at a time or group a couple of states and then train the model. The diagram below shows how the metadata is to be processed.
The GitHub gist given below contains the entire code to generate a single CSV file, which we later need to split. Most of the code is self-explanatory, with sufficient comments. Make sure to install Librosa and num2words first.
The script is executed with the wav parameter pointing to the folder containing all the audio files and the meta parameter pointing to the folder containing the txt.done.data file. Here, the first part copies the data from Google Bucket to Colab. The second part creates a CSV file, and the last command appends to this CSV and so on. If you want to train each ZIP file individually, run only one command and proceed to split the CSV (although I don’t recommend it). Take care to comment out Line 45 if appending to the same file.
$ gsutil cp gs://bucket-name/hindi_female_english.zip /content/hindi_female_english.zip$ gsutil cp gs://bucket-name/hindi_male_english.zip /content/hindi_male_english.zip$ unzip hindi_female_english.zip -d /content/hindi_female_english$ unzip hindi_male_english.zip -d /content/hindi_male_english--------------------------------------------------------------------$ python preProcess.py --wav /content/hindi_female_english/english/wav --meta /content/hindi_female_english/english$ python preProcess.py --wav /content/hindi_male_english/english/wav --meta /content/hindi_male_english/english
So now, we have a CSV file and need to split it into three individual files. In the gist to the left, we first split the main CSV file into train and intermediate and then split intermediate into dev and test. Then we finally have three files corresponding to the three splits DeepSpeech requires for training.
The official training documentation is given here. It is very detailed but skips over some fine details that might hold you up for a frustratingly long time 😩. I’m assuming that you’ve transferred the dataset to a Google Cloud bucket and have a proper internet connection as Colab terminates your instance if you don’t reconnect quickly. All the steps below are taken from the training notebook available here on Colab.
Authorize Colab to access your Google Cloud Bucket: Download your project’s IAM access credentials file and upload them to Colab.
Authorize Colab to access your Google Cloud Bucket: Download your project’s IAM access credentials file and upload them to Colab.
import osfrom google.colab import authauth.authenticate_user()os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "creds.json"#ensure the path is set correctly!echo $GOOGLE_APPLICATION_CREDENTIALSproject_id = '<gcloud-project-id>'bucket_name = 'gs://<bucket-name>'!gcloud config set project {project_id}
2. Clone DeepSpeech v0.7.4 from GitHub: And download the corresponding checkpoint.
!git clone --branch v0.7.4 https://github.com/mozilla/DeepSpeech!wget https://github.com/mozilla/DeepSpeech/releases/download/v0.7.4/deepspeech-0.7.4-checkpoint.tar.gz!tar -xvf deepspeech-0.7.4-checkpoint.tar.gz -C \/content/model_checkpoints/
3. Install DeepSpeech Dependencies: There are multiple things we’ll need to install every time we train our model. The detailed steps are given in the notebook here.
4. Set default CUDA version: This part is going to be extremely frustrating if you don’t get it right the first time 😤. For some reason, whenever you try to set the default CUDA version as 10.0 (DeepSpeech requires CUDA 10.0 and CuDNN v7.6) in Colab, it always tries to revert to 10.1 🤷. I’ve found a sequence of steps that seem to work but I can’t guarantee it’ll work every time — you might need to dig a little to get it just right. The detailed steps are given in the notebook here. If you still can’t get it to work, I’d be more than willing to help :)
5. Install TensorFlow-GPU:
!pip3 uninstall tensorflow!pip3 install 'tensorflow-gpu==1.15.2'
6. Copy data to Colab from Google Bucket: You need to be careful here and copy the files to exactly the place they were when you generated the CSV files because the model uses the absolute path to the audio files as stated in the CSV file.
%cd /content/!gsutil cp gs://bucket-name/hindi_female_english.zip /content/hindi_female_english.zip!gsutil cp gs://bucket-name/hindi_male_english.zip /content/hindi_male_english.zip!unzip hindi_female_english.zip -d /content/hindi_female_english!unzip hindi_male_english.zip -d /content/hindi_male_english
7. Check for extra characters not in alphabet.txt: This file (present in /DeepSpeech/data) defines the alphabet of the language that DeepSpeech is being trained on. For our task since the language is the same, we don’t need to change it. If we want to train a new model in a completely different language, we need to redefine the alphabet and follow these steps. Upload the three CSV files to /content/ and run check_parameters.py on them.
!python3 /content/DeepSpeech/training/deepspeech_training/util/check_characters.py -csv /content/train.csv -alpha!python3 /content/DeepSpeech/training/deepspeech_training/util/check_characters.py -csv /content/dev.csv -alpha!python3 /content/DeepSpeech/training/deepspeech_training/util/check_characters.py -csv /content/test.csv -alpha
Note: You might need to comment out some lines in check_characters.py to get it to work. After commenting out some lines, this is how line 44 in that file should look like
if not args.disable_unicode_variants:  unicode_transcript = unicodedata.normalize("NFKC", row[2])  #if row[2] != unicode_transcript:    #print("Your input file", in_file, "contains at least one transript with unicode chars on more than one code-point: '{}'. Consider using NFKC normalization: unicodedata.normalize('NFKC', str).".format(row[2]))    #sys.exit(-1)all_text |= set(row[2])
8. Training the model (Finally 😌): This step is pretty straightforward. Try out different hyper-parameters and double-check the paths. The augmentation parameters can help your model generalize a bit better.
9. Exporting the model for inference: After the training process is done, a .pb model file is exported to export_dir. But this model is very big (~700MB) and not very efficient with batch predictions. Luckily, there’s an option to convert the exported model to a memory-mapped model, after which we get a .pbmm model file.
%cd /content/DeepSpeech/!python3 util/taskcluster.py --source tensorflow --artifact convert_graphdef_memmapped_format --branch r1.15 --target .!./convert_graphdef_memmapped_format --in_graph=/content/models/ft_model.pb --out_graph=/content/models/ft_model.pbmm
This .pbmm file is relatively smaller (~180MB) and efficient and can be used with the corresponding scorer file to improve inference results.
Phew! That was a long article 😵. If you’ve reached till here, thank you for sticking along :)
From my time working with DeepSpeech, I’ve found it to be one of the easiest libraries to work with for ASR. I hope this article helps anyone trying out things on their own.
You can reach me on: LinkedIn, GitHub