Building Subtitle Text from Speech-to-Text’s Word Timestamps | by Ng Wai Foong | Towards Data Science

Speech-to-Text functionality has been gaining momentum recently as it offers a whole new user experience to users. It is being widely adopted by companies in the market especially in the customer services industry. In fact, big players such as Google and Microsoft provide their own Speech-to-Text API as part of their technologies.
For your information, most of the advanced Speech-to-Text APIs comes with word-level timestamps.
For example, you will get the following output when running Google’s Speech-to-Text API:
[    {        "startTime": "1.400s",        "endTime": "1.800s",        "word": "okay"    },    {        "startTime": "1.800s",        "endTime": "2.300s",        "word": "so"    },    ...]
On the other hand, Microsoft has its own Azure’s Speech-to-Text Service which returns the following result:
[    {        "Duration": 5500000,        "Offset": 7800000,        "Word": "seven"    },    {        "Duration": 2700000,        "Offset": 13400000,        "Word": "three"    },    ...]
Both APIs looks promising on the surface but there is a big issue if you intend to use the output for building subtitles, which is to combine them into sentences that is not too short or too long. If the end text is in English, you can still post-process and split it based on certain punctuation such as:
[',', '.', '!', '?']
In Azure’s case, this is not possible because the text in word-level timestamp is based on lexical (no capitalization and punctuation). Besides that, the display text after transcription is in dictation mode which differs from the text in word-level timestamp. For example, the display text:
007 James Bond
will have the following word-level timestamps:
[    {        "Duration": 2700000,        "Offset": 35600000,        "Word": "double"    },    {        "Duration": 700000,        "Offset": 38400000,        "Word": "oh"    },    {        "Duration": 4900000,        "Offset": 39200000,        "Word": "seven"    },    {         "Duration": 3900000,         "Offset": 44400000,         "Word": "james"    },    {         "Duration": 3300000,         "Offset": 48400000,         "Word": "bond"    }]
As a result, there is no way for you to map the result based on array since the length and text are different at some point. At the time of this writing, the developer has clarified that there is no support for dictation and punctuation mode in the word-level timestamps.
It will be a lot more complicated if you intend to support multiple languages. This is mainly because each language has its own punctuation symbols and certain languages are not tokenizable by space.
After a few rounds of experiment, I found a simple trick to split the output words into sentences. This method should works well for most languages as it is not dependent on the output text. The main gist is to go through the whole list and calculate the length of the empty gaps between them. If the length exceeds the given threshold, treat it as sentence break.
Have a look at the following image as reference:
In this case, you will end up with two sentences:
it is reported that
five people were injured
In addition to being language-independent, you can fine-tune the threshold based on your preferences in order to get the best segmentation.
Let’s proceed to the next section and start writing some code.
This tutorial will use the output from Azure’s Speech-to-Text as reference when doing the post-processing. Also, the code will be presented in Python for simplicity. Having said that, you can easily convert it to other programming languages since no external package is involved.
Create a new Python file in your working directory and initialize it with the following variables:
By now, you should have noticed that both the Offset and Duration are in nano seconds. Let’s create a function which converts input nano seconds to seconds in two decimal places:
def get_seconds(nanoseconds):    return round(nanoseconds / 10000000, 2)
We will need a function to join the words into a sentence. In order to support multiple languages, you can set a conditional statement to determine the input language and join the words using the appropriate delimiter. For example:
def join_words(words, lang):    if lang == 'ja-JP' or lang == 'th-TH':        return ''.join(words)    return ' '.join(words)
In this case, Thai and Japanese is build without space while the rest of the languages will be delimited by space. Modify it accordingly based on your own preferences.
The next step is to implement the post-processing functionality. Before that, let’s loop through the list based on its length and calculate the differences on the length of the gap between each word.
The loop will start from 0 and end at the total length of the list minus 1. This is mainly because the difference is calculated as follows:
Offset of next word - (Offset + Duration of current word)
You should see the following output on your console (without the arrow markers):
0.010.010.010.010.010.010.010.010.010.010.25  <--0.010.010.010.010.010.010.010.010.010.41  <--0.010.010.010.010.010.010.010.010.010.01
The are two instances in which the duration exceeds the given threshold of 0.1. Comment out the print statement and modify the loop into as follows:
The code works as follows:
calculate the differences
set start if tokens is empty which indicates start of sentence
build sentence if difference exceeds threshold, append the information as dictionary inside timestamp variable, clear tokens and set start to next Offset
if it reached the the second last word, add in the last word and break out of loop
Please be noted that the implementation above uses Offset + Duration as the end time. Modify it accordingly based on your own preference.
There can be cases in which it returned just a single item in the list. In this situation, the loop will exit and return an empty list. You can easily handle this by adding a conditional check statement as follows:
if len(data) == 1:    timestamp.append({'segment': data[0]['Word'], 'start': get_seconds(data[0]['Offset']), 'end': get_seconds(data[0]['Offset'] + data[0]['Duration'])})else:    # the rest of the code (loop)    ...
You can find the complete code at the following gist:
Run it as follows:
python script.py
You should get the following output:
[{'segment': 'average household income is up ten percent from four years ago', 'start': 5.11, 'end': 8.52}, {'segment': 'and our customers are spending twenty percent more per transaction', 'start': 8.77, 'end': 12.12}, {'segment': 'nearly everyone surveyed is employed in a professional or managerial occupation', 'start': 12.53, 'end': 16.8}]
You can use the output list to build your own subtitle file, be it SRT or VTT format. For example, the converted SRT file should be as follows:
100:00:05,110 --> 00:00:08,520average household income is up ten percent from four years ago200:00:08,770 --> 00:00:12,120and our customers are spending twenty percent more per transaction300:00:12,530 --> 00:00:16,800nearly everyone surveyed is employed in a professional or managerial occupation
As for VTT, the output should be as follows:
WEBVTT00:00:05.110 --> 00:00:08.520average household income is up ten percent from four years ago00:00:08.770 --> 00:00:12.120and our customers are spending twenty percent more per transaction00:00:12.530 --> 00:00:16.800nearly everyone surveyed is employed in a professional or managerial occupation
Let’s recap what you have learned today.
This article started with a brief explanation on Speech-to-Text and the problem that developers faced when using the word-level timestamps output from Speech-to-Text API.
Then, it proposed a working solution that is language-independent, which is to split based on the empty gaps between each word.
It continued and explained the implementation entire process which uses Azure’s Speech-to-Text API as reference output. At the end, you should get a list of dictionaries which can be used to create the corresponding subtitle file in SRT or VTT format.
Thanks for reading this piece. Feel free to check out my other articles. Have a great day ahead!
Google Speech-to-Text — Getting word timestampsAzure Documentation — Speech-to-Text quickstart
Google Speech-to-Text — Getting word timestamps
Azure Documentation — Speech-to-Text quickstart