Generate Subtitles for any video file using Mozilla DeepSpeech | by Abhiroop Talasila | Towards Data Science

In the age of OTT platforms, there are still some who prefer to download movies/videos from YouTube/Facebook/Torrents (shush 🤫) over streaming. I am one of them and on one such occasion, I couldn’t find the subtitle file for a particular movie I had downloaded. Then, the idea for AutoSub struck me and since I had worked with DeepSpeech previously, I decided to use it to generate subtitles for my movie.
Given a video file as input, my goal was to generate an .srt file. The subtitles could then be imported into any modern video player. In this article, I’m going to walk you through some of the code. You can find the project on my GitHub here with instructions on how to install locally.
Prerequisites: Intermediate understanding of Python, some familiarity with Automatic Speech Recognition Engines, and a basic understanding of signal processing will be great.
Note: This is my first article on Medium. If you have any suggestions/doubts please comment them down. Happy reading :)
DeepSpeech is an open-source speech-to-text engine based on the original Deep Speech research paper by Baidu. It is one of the best speech recognition tools out there given its versatility and ease of use. It is built using Tensorflow, is trainable using custom datasets, trained on the huge Mozilla Common Voice dataset, and is licensed under the Mozilla Public License. The best advantage is that we can download the model files and perform inference locally just within a couple of minutes!
Although, DeepSpeech does have its issues. The model struggles with non-native English accent speech. There is a workaround to this — fine-tuning the model using a custom dataset in the language we want to predict. I’ll write another article on how to do that soon.
If you’re working with speech recognition tasks, I strongly recommend you have a look at DeepSpeech.
Let’s start off by installing some packages we’ll be needing. All commands have been tested on Ubuntu 18.04 in a pip virtual environment.
FFmpeg: FFmpeg is the leading multimedia framework, able to decode, encode, transcode, mux, demux, stream, filter, and play pretty much anything that humans and machines have created. We need it to extract audio from our input video file.
FFmpeg: FFmpeg is the leading multimedia framework, able to decode, encode, transcode, mux, demux, stream, filter, and play pretty much anything that humans and machines have created. We need it to extract audio from our input video file.
$ sudo apt-get install ffmpeg
2. DeepSpeech: Install the python package from PyPI and download the model file. The scorer file is optional but greatly increases accuracy.
$ pip install deepspeech==0.8.2# Model file (~190 MB) $ wget https://github.com/mozilla/DeepSpeech/releases/download/v0.8.2/deepspeech-0.8.2-models.pbmm# Scorer file (~900 MB)$ wget https://github.com/mozilla/DeepSpeech/releases/download/v0.8.2/deepspeech-0.8.2-models.scorer
Now that we’re all set up, let’s first extract the audio from our input video file using FFmpeg. We need to create a subprocess to run UNIX commands. DeepSpeech expects the input audio file to be sampled at 16kHz and hence the arguments to ffmpeg given below.
Now, suppose our input video file is 2 hours long. Running DeepSpeech inference on the whole file is not generally recommended. I’ve tried it and it didn’t work out too well. One workaround to this is to split the audio file on silent segments. After splitting, we have multiple small files containing speech which we need to infer. This is done using pyAudioAnalysis.
The following function uses the functions read_audio_file() and silenceRemoval() from pyAudioAnalysis and generates segment limits from where speech begins and ends. The arguments control the smoothing window size in seconds and weight factor in (0,1). Using the segment limits, smaller audio files are written to disk.
We now need to run DeepSpeech inference on these files individually and write the inferred text to a SRT file. Let’s start by creating an instance of the DeepSpeech Model and add the scorer file. We then read the audio file into a NumPy array and feed it into the speech-to-text function to produce inference. As mentioned above, pyAudioAnalysis saves files with the segment limits time in seconds. We need to extract those limits and convert it into a suitable form before writing to the SRT file. The write function is defined here.
The whole process shouldn’t take more than 60% of the original video file duration. Here’s a video showing a sample run on my laptop.
There’s one more area I hope to improve in the future. The inferred text is unformatted. We need to add proper punctuation, correct possible small mistakes in words (off by one letter), and split very long segments into smaller ones (although this will be difficult to automate).
That’s it! If you’ve reached till here, thank you for sticking along :)
You can find me here: LinkedIn, GitHub