Speech Recognition with Tensorflow.js | by Benson Ruan | Towards Data Science

Voice Assistants like Amazon Alexa and Google Home have become widely popular, they allow users to quickly get things done by using speech recognition.
Thanks to an improvement in speech recognition technology, Tensorflow.js released a JavaScript module that enables the recognition of spoken commands.
In this article, we will use a pre-trained Tensorflow.js model for transfer learning. Let’s build an application which can recognize your speech command.
Try it yourself in the link below :
bensonruan.com
Isn’t it amazing that we can now use our voice to control computer contactlessly? Tensorflow.js really bringing Artificial Intelligence to the browser. Let’s dive into the code and show you step by step how to build this speech recognition application with tensorflow.js.
Simply include the scripts for tfjs and speech-commands models in the <head> section of the html file.
<script src="https://unpkg.com/@tensorflow/tfjs"></script><script src="https://unpkg.com/@tensorflow-models/speech-commands"></script>
HTML — index.html
Add a <div> place holder in the html body
<div id="candidate-words"></div>
Add a checkbox to control turn on/off the Microphone, I had customized the checkbox to look like a mobile switch using pure css.
<label class="form-switch"><input type="checkbox" id="audio-switch"><i></i> Microphone</label>
Javascript — speech_command.js
The tfjs speech command library support 20 item vocabulary, consisting of: ‘zero’, ‘one’, ‘two’, ‘three’, ‘four’, ‘five’, ‘six’, ‘seven’, ‘eight’, ‘nine’, ‘up’, ‘down’, ‘left’, ‘right’, ‘go’, ‘stop’, ‘yes’, and ‘no’, in addition to ‘background_noise’ and ‘unknown’. When the page is loaded, append the words into the list.
wordList = ["zero","one","two","three","four","five","six","seven","eight","nine", "yes", "no", "up", "down", "left", "right", "stop", "go"];$.each(wordList, function( index, word ) { if (!word.startsWith('_')){  $("#candidate-words").append('<span id='word-${word}'>${word}</span>'); }});
When Microphone switch (checkbox) turn on (checked), first call function loadModel() to load the pre-trained model, then call function startListening() to start voice command recognition.
When Microphone switch (checkbox) turn off (unchecked), call function stopListening() to disconnect the Mic.
if(this.checked){ if(!modelLoaded){  loadModel(); }else{  startListening(); }}else { stopListening();}
function loadModel() to load the pre-trained speech command model, calling the API of speechCommands.create and recognizer.ensureModelLoaded. When calling the create function, you must provide the type of the audio input. The two available options are ‘BROWSER_FFT’ and ‘SOFT_FFT’. — BROWSER_FFT uses the browser’s native Fourier transform. — SOFT_FFT uses JavaScript implementations of Fourier transform (not implemented yet).
function loadModel(){    recognizer = speechCommands.create("BROWSER_FFT");      Promise.all([        recognizer.ensureModelLoaded()      ]).then(function(){        words = recognizer.wordLabels();        modelLoaded = true;        startListening();      })}
function startListening() to call the recognizer.listen API to start listening to voice command. The listen() function takes two arguments: 1. A callback function that is invoked anytime a word is recognized. 2. A configuration object with adjustable fields such a — includeSpectrogram — probabilityThreshold (The callback function will be invoked if and only if the maximum probability score of all the words is greater than this threshold) — includeEmbedding
The output of scores contains the probability scores that correspond to recognizer.wordLabels(). Below code turning scores into a list of (score,word) pairs.
Please note that Microphone can only be accessed via the https protocol
function startListening(){    recognizer.listen(({scores}) => {        scores = Array.from(scores).map((s, i) => ({score: s, word: words[i]}));        scores.sort((s1, s2) => s2.score - s1.score);        $("#word-"+scores[0].word).addClass('candidate-word-active');    },     {        probabilityThreshold: 0.70    });}
Finally, to turn off the Microphone, call the recognizer.stopListening API
function stopListening(){    recognizer.stopListening();}
You can download the complete code of the above demo in the link below:
github.com
Tensorflow.js in 2020 has become the bread and butter for all Machine Learning JavaScript projects due to its comprehensive linear algebra core and deep learning layers. From this article, I hope you have fun and I encourage you to discover more about this library.
Thank you for reading. If you like this article, please share on Facebook or Twitter. Let me know in the comment if you have any questions. Follow me on GitHub and Linkedin.