Live Adaptive Video Speech Recognition
By: Michael Cunningham
Tags: Machine Learning, Live Streaming, Subtitles, Live Video

Speech recognition is an ever-evolving technology, to the point where we now have digital assistants in our homes and on our personal devices. Amazon’s Alexa, Apple’s Siri, Microsoft’s Cortana, Google’s Assistant—the list goes on and will continue to grow.

Being able to apply that same speech recognition technology to video content has the potential to vastly increase the prevalence of subtitled media content for audiences — assuming it’s easy (and cheap) enough to integrate into existing workflows.

Purpose

This post outlines two strategies for integrating an efficient and accurate automated subtitle generation solution for an adaptive video stream — either for Live or On-Demand content.

And by leveraging solutions that are readily available from big companies such as Google, Microsoft, Amazon (and more), we don’t have to deal with the large technical challenges that come with understanding Machine Learning.

Accuracy

In order for auto-generated subtitles to be viable, they need to be accurate.

Generally speaking, the industry currently hires human transcribers, which will typically result in exceptionally accurate subtitles but, at the same time, are quite expensive when you may be able to leverage an automated solution that’s accurate enough at a fraction of the human-sourced cost.

Captions vs Subtitles

In order to clear up any confusion, the two terms are often used interchangeably but there is a concrete difference. Subtitles are intended for use when the viewer can hear the content audio.[1]

Captions typically include cues for signalling even small environmental sounds in a scene or include cues when music is playing — sometimes even with lyrics.

Subtitles are translations of the spoken language, while captions are typically in the single language the content was originally recorded in.[1]

Requirements

The recognizer service (a speech-to-text API) is required to return an accurate timed transcript from the given audio. Without timing or with inaccurate timing, it’s not possible to map cues to the correct specific times in the content.

Approach

In order to make the integration of such a workflow as seamless as possible, the timed transcript provided by the recognizer service should ideally be transformed to conform to popular standards, like segmented WebVTT for example — something that many currently available video players support[2].

However, it is entirely acceptable to implement custom delivery solutions for transcripts to the client (video player) if the non-standard integration work isn’t a deal breaker for you.

The recognizer only cares about the format of the audio provided and any service-specific required settings. It is technically possible to implement recognition from any audio source.

At the time of writing this post, only Google Cloud Platform’s Speech-to-Text API has been tested as a recognizer in this application. The included web-based proof-of-concept (PoC) demonstrates this integration.

It is recommended for a recognizer to provide a “confidence score” alongside the timed transcript so that determinations can be made based on the result — such as whether or not to send the audio to a different recognizer to potentially get better transcription accuracy.

Strategies

The aforementioned PoC includes two strategies for integrating a speech recognition workflow for live adaptive video content.

Server-side focused (Process direct encoder output) <- Recommended Client-side focused (Process buffered client data)

Server-side focused is always the preferred approach as it enables the ability to start the transcription process as soon as data is available from the encoding pipeline. If you do not have access to the encoding pipeline, you could go with the Client-side focused approach.

Both of the strategies described above can leverage at least one of the of these readily-available Speech-to-Text APIs:

Pricing always matters. I recommend calculating your estimated cost for each of the above services. This will help determine which service has the best cost/accuracy ratio for your specific needs.

Server-side focused strategy

The server-side focused strategy involves extracting and processing audio content from stream segments as soon as they are output from the encoding pipeline. One approach to facilitate this could be watching for file-system changes — specifically when new segments are completely flushed to disk.

You may find that your encoder is faster at outputting segments than the transcriber can process, which could introduce rate limiting issues on the underlying recognizer service. To remedy this, a queuing system can be implemented to control the transcription request rate.

Flow Diagram: Server-side focused strategy

The order in which your integration transcribes stream segments is important depending on the type of content.

For live or event streams, transcribing the newest segments first is recommended, as most of the viewers will be watching from near the live edge of the sliding window. On-demand content is a special case, as transcription will likely occur during an initial content ingestion process. That said, starting from the first segment is logical.

Transcriptions can be delivered to the client (video player) in a variety of ways, some common methods are:

Segmented WebVTT

Custom (e.g. JSON payloads from a remote source)

ID3 (for metadata — could insert transcription confidence scores here)

You could also utilize the CEA-608/708 format and deliver directly via the picture user data of an MPEG-2 stream[3]. However it is worth noting that existing live content may already contain human-transcribed captions that are carried in that format.

Client-side focused strategy

The client-side focused strategy depends on a small server-side component simply for relaying audio data to the transcriber and returning the transcription (typically) in a non-standard way, such as simple JSON structures.

The need for the server-side component is primarily to protect your cloud service account information. Where your backend component will act as a proxy, optionally performing processing on the input / output. If security wasn’t a concern, the entire implementation could technically be applied in the web browser.

Flow Diagram: Client-side focused strategy

The included PoC for this strategy utilizes WebSocket connections between the client and server for transmitting audio bytes for transcription. Responses to these socket messages contain the timed audio transcript and corresponding confidence score. Client-side conversion to browser-consumable WebVTT cues is necessary.

Which strategy should you integrate?

Server-side focused — Encoder Output (Recommended)

Less bandwidth consumption; transcripts are available either within or alongside stream segments.

Lower processing requirement; only text parsing is required, from whichever delivery method of your choosing.

More accurate timing; easier to reference transcripts as they are associated with a specific segment from the encoder output.

Client-side focused — Buffered Client Data

Higher memory consumption due to the need to keep references to buffered playable audio content.

Higher bandwidth usage due to the need to transmit audio to the recognizer.

Potentially less accurate timing depending on the integration. Specifically for mapping audio to exact timeline values in the content stream.

Longer startup time between user-enablement and on-screen cue rendering.

Regardless of the strategy taken — the PoC is leveraging the open source adaptive playback library hls.js as it is far easier to utilize a pre-built solution and at the same time — verifying compatibility due to its popularity is valuable.

For the server side components, I opted to use Golang and the FFmpeg CLI to perform necessary audio pre-processing tasks.

Requirements

There should be some expectations of the PoC in order to consider it viable:

It must be quick. Less than 5 seconds from user-enablement to on-screen cue rendering is preferred.

It must be accurate enough. Using a corresponding confidence score, we can make determinations of the resulting transcript (for example, allowing multiple recognizers to compete based on who has the highest confidence score).

It must be consistent with non-auto-generated subtitle streams. The auto-generated subtitles should behave the same as other common subtitle streams so they appear in the player’s text track controls as ‘English (auto-generated)’ or similar. Cue rendering should not behave differently either.

For integrations that use segmented WebVTT or CEA-708 for auto-generated subtitle delivery, you will likely get parsing support out of the adaptive playback library you’re leveraging at a very little integration cost— if any.

For custom delivery methods, such as simple JSON payloads associated to a particular segment, the translation from that structure is necessary before populating text tracks in the browser.

Gotchas

Following the definitions of live latency from Mux, I’ve found that both of the described strategies here are unsuitable for ultra low-latency (ULL) streams and certain short-segment low-latency (LL) streams. Segment durations of ~8 seconds are ideal, or else you’re going to run into transcription issues where you have a higher chance of broken / incomplete words and sentences that are skipped by the transcriber. This could be remedied by performing stitching of given audio chunks and receiving a transcription stream so that there’s a continuous stream of input and output data.

Transcribing low latency streams is possible but it would likely require all processing and recognition to be performed on the client-side so network data transfer delay can be omitted from the equation.

What’s Next

The included PoC applications that implement both strategies outlined can be found here, follow the instructions in the README to try it out.

The PoC has plenty of room for optimizations or general improvements — some of the top ones I’ve noted are:

Integrating a two-phase transcription system that performs transcription, and then optionally performs text translation of the transcript to a target language. For example, with this, we could add additional subtitles in multiple languages for all of your VOD content.

Changing the delivery method of transcriptions when processing direct encoder output. Currently, it’s using a custom solution (JSON file per segment) but translating to segmented WebVTT is definitely possible.

Replacing file-system juggling for intermediary transcodes (e.g. AAC -> OGG @ 16kHz) with `stdin`/`stdout` piping with `ffmpeg`: https://ffmpeg.org/ffmpeg-protocols.html#pipe

Resiliency to errors — if a fatal error occurs during transcription, the process should be restarted (either globally or per-segment).

Providing encoder (ffmpeg) arguments / configuration via an external source.

Conclusion

If having exceptional subtitle accuracy is not mission critical, quality automated speech recognition for video content is achievable today and can drastically increase the prevalence of subtitled content for the people who need (or prefer) it.

Feel free to reach out to me directly on Slack #video-dev if you have questions, comments, or suggestions! Username: michaelcunningham

Notes

Cannot utilize Mozilla DeepSpeech currently:

Pending implementation of word offset timing in transcript output: https://github.com/mozilla/DeepSpeech/issues/1125 A branch has been made that integrates word offset timing, available for testing: https://github.com/mozilla/DeepSpeech/issues/1887

References

A short-list of playback libraries and/or players that support WebVTT and CEA-608/708 captions: