OK Google, Tell Me How Trigger Word Detection works
By: Prateek Karkare
Tags: Recurrent Neural Network, Artificial Intelligence, Machine Learning, Neural Networks, Voice Assistant

Is google always listening to what I say? How does it wake up when I say OK Google? Are personal assistants like Google, Alexa and Siri always spying on me by listening to all my conversations? Questions like these have crossed everyone’s mind. Rather than answering these questions with a straight yes or no lets understand the technology behind these devices and understand how do they wake up when we call their name.

You now know a lot about sequence modelling and how it is done using Recurrent Neural Networks (RNN) and Long Short Term Memory (LSTM) and you might have also built an RNN. The systems which wake up on listening to a specific word are called trigger word detection systems and such problems within sequence modelling are known as trigger word detection problems. The literature on trigger word detection is still evolving so there isn’t a consensus yet on the best trigger word detection algorithm. We will try to understand one implementation of such a system using Keras. This implementation requires some familiarity with concepts of RNN, GRU and Neural Networks. OK Google, lets begin!

Overview

We will be using pre-recorded audio files for the purposes of illustration but you can record your own audio using microphone of your computer. The way we generate training examples are as follows. We have 3 types of audio recordings

Background Positive words Negative words

Background audio files contain recordings of sounds from environments like a cafe, home, office etc. which will form our backdrop. When we say Ok Google we do not say it in absolute silent environments there are always background noises coming from people around us, traffic and many other sources. We need to train our model to not wake up from these noises and to distinguish between the trigger word and the background sounds.

Positive words are the words on which we want to wake up our system which are called trigger words. You can choose and record any word as a trigger word but for this article we will use the word “Activate” as our trigger word. These words need to be spoken by different persons in different tones to train the model well enough to generalize across different scenarios.

Negative words are those words which should not wake up our system. All other words other than our trigger word (activate in this case) will act as negative words. We will insert few negative words around our trigger word so that our model learns to ignore them.

To generate lot of training examples we take the audio files with background noise and overlay positive and negative words on them at random time intervals. Which will sound like this eventually

Setting Up

If you are not familiar with Python, Anaconda, Jupyter notebooks etc. it is time you get your hands dirty with these. These are the most widely used tools for Data Science and Machine Learning. You can get a primer on setting up your system here and here. Once you are set up install a few packages that will help us to manipulate audio files in Python.

conda install -c auto pydub

Go back to the Jupyter notebook and import the required packages

import numpy as np

from pydub import AudioSegment

import random

import sys

import io

import os

import glob

import IPython

from scipy.io import wavfile

import matplotlib.pyplot as plt

%matplotlib inline

If you face a no module found error for any package. Go back to your conda environment and install that package. The last line in the above code helps us to plot the graphs inline inside Jupyter notebook. Below are some functions which will help you process those audio files to overlay positive and negative words on the background track.

# Load raw audio files for speech synthesis

def load_raw_audio():

activates = []

backgrounds = []

negatives = []

for filename in os.listdir("./raw_data/activates"):

if filename.endswith("wav"):

activate = AudioSegment.from_wav("./raw_data/activates/"+filename)

activates.append(activate)

for filename in os.listdir("./raw_data/backgrounds"):

if filename.endswith("wav"):

background = AudioSegment.from_wav("./raw_data/backgrounds/"+filename)

backgrounds.append(background)

for filename in os.listdir("./raw_data/negatives"):

if filename.endswith("wav"):

negative = AudioSegment.from_wav("./raw_data/negatives/"+filename)

negatives.append(negative)

return activates, negatives, backgrounds activates, negatives, backgrounds = load_raw_audio()

This function will load all the files corresponding to positive, negative and background sounds from folders. A spectrogram is used to analyze audio files visually. We can plot spectrograms of these audio files using —

# Calculate and plot spectrogram for a wav audio file

def graph_spectrogram(wav_file):

rate, data = wavfile.read(wav_file)

nfft = 200 # Length of each window segment

fs = 8000 # Sampling frequencies

noverlap = 120 # Overlap between windows

nchannels = data.ndim

if nchannels == 1:

pxx, freqs, bins, im = plt.specgram(data, nfft, fs, noverlap = noverlap)

elif nchannels == 2:

pxx, freqs, bins, im = plt.specgram(data[:,0], nfft, fs, noverlap = noverlap)

return pxx x = graph_spectrogram("audio_examples/example_train.wav")

You can find more details on what is a spectrogram here. The spectrogram is a frequency vs time plot which looks like —

OK, till now we are able to load our files and visualize them using a spectrogram. Lets do some manipulation on them

# Gets a random time segment from the audio clip

def get_random_time_segment(segment_ms): segment_start = np.random.randint(low=0, high=10000-segment_ms

segment_end = segment_start + segment_ms - 1



return (segment_start, segment_end) # Check if a segment is overlapping with any previous segments

def is_overlapping(segment_time, previous_segments):



segment_start, segment_end = segment_time



overlap = False



for previous_start, previous_end in previous_segments:

if segment_start <= previous_end and segment_end >= previous_start:

overlap = True return overlap # Insert an audio segment without overlapping

def insert_audio_clip(background, audio_clip, previous_segments):



segment_ms = len(audio_clip) segment_time = get_random_time_segment(segment_ms)



while is_overlapping(segment_time, previous_segments):

segment_time = get_random_time_segment(segment_ms) previous_segments.append(segment_time)



new_background = background.overlay(audio_clip, position = segment_time[0])



return new_background, segment_time

The above 3 functions accomplish the task of inserting audio segments in an audio file. We will use these to insert positive and negative words in our background file. The first function finds a random time segment in the background file where we wish to insert another positive or negative audio segment. The second function helps us to check whether the segment we have selected to insert audio doesn’t already contain a previously inserted segment. The third function inserts an audio clip on a background clip. The below function creates a label array which will be used to train our model. The labels of the 50 output steps strictly after the end of the segment should be set to 1. By strictly we mean that the label of segment_end_y should be 0 while, the

50 following labels should be ones. The label vector looks like —

[0 0 0 0 0 0 0 0 1 1 1 1 1 1 ….. 1 0 0 0 0 0 0 0 0 0 0] where ones start a step after the positive word is inserted. To visualize it better we see it in perspective with the spectrogram

As soon as the word activate is spoken we insert 1s in the label vector for 50 time steps.

def insert_ones(y, segment_end_ms):



segment_end_y = int(segment_end_ms * Ty / 10000.0)



for i in range(segment_end_y + 1, segment_end_y + 51):

if i < Ty:

y[0, i] = 1



return y

Building the Model

Most of our pre-processing steps are now done. We just need to build the model train it and perform predictions. Lets see how to load the datasets.

#Load training Examples

X = np.load("./XY_train/X.npy")

Y = np.load("./XY_train/Y.npy") #Load dev set examples

X_dev = np.load("./XY_dev/X_dev.npy")

Y_dev = np.load("./XY_dev/Y_dev.npy")

Lets see how our model will look. You will needs keras to be installed to be able to build your neural network.

Source: Coursera

Looks a bit intimidating but lets try to break it down and understand. The bottom most layer is a 1D CONV layer which is similar to a convolutional layer used in image classification networks. This converts our input spectrogram which is 5511 time steps for a 10 second audio, each time step is a 101 long vector. The convolution layer outputs a 1375 step output. It reduces the computation required by the subsequent layers. Conv layer is followed by batch normalization, activation and a drop-out layer. If you understand convolutional neural networks you will know the function of each of these layers. GRU (Gated Recurrent Unit) is very similar to a Recurrent Neural Network unit. The last layer is a sigmoid layer which predicts whether the trigger word was said or not. Lets load the required libraries for the model.

from keras.callbacks import ModelCheckpoint

from keras.models import Model, load_model, Sequential

from keras.layers import Dense, Activation, Dropout, Input, Masking, TimeDistributed, LSTM, Conv1D

from keras.layers import GRU, Bidirectional, BatchNormalization, Reshape

from keras.optimizers import Adam

We will now build the model depicted in the picture above using keras.

def model(input_shape):



X_input = Input(shape = input_shape)



X = Conv1D(196, kernel_size=15, strides=4)(X_input)

X = BatchNormalization()(X)

X = Activation('relu')(X)

X = Dropout(0.8)(X) X = GRU(units = 128, return_sequences = True)(X)

X = Dropout(0.8)(X)

X = BatchNormalization()(X)



X = GRU(units = 128, return_sequences = True)(X)

X = Dropout(0.8)(X)

X = BatchNormalization()(X)

X = Dropout(0.8)(X)



X = TimeDistributed(Dense(1, activation = "sigmoid"))(X)

model = Model(inputs = X_input, outputs = X)



return model

Keras allows us to see the summary of our model. It shows the layers their shapes and the number of parameters it has.

model.summary() _________________________________________________________________

Layer (type) Output Shape Param #

=================================================================

input_2 (InputLayer) (None, 5511, 101) 0

_________________________________________________________________

conv1d_2 (Conv1D) (None, 1375, 196) 297136

_________________________________________________________________

batch_normalization_4 (Batch (None, 1375, 196) 784

_________________________________________________________________

activation_2 (Activation) (None, 1375, 196) 0

_________________________________________________________________

dropout_5 (Dropout) (None, 1375, 196) 0

_________________________________________________________________

gru_3 (GRU) (None, 1375, 128) 124800

_________________________________________________________________

dropout_6 (Dropout) (None, 1375, 128) 0

_________________________________________________________________

batch_normalization_5 (Batch (None, 1375, 128) 512

_________________________________________________________________

gru_4 (GRU) (None, 1375, 128) 98688

_________________________________________________________________

dropout_7 (Dropout) (None, 1375, 128) 0

_________________________________________________________________

batch_normalization_6 (Batch (None, 1375, 128) 512

_________________________________________________________________

dropout_8 (Dropout) (None, 1375, 128) 0

_________________________________________________________________

time_distributed_2 (TimeDist (None, 1375, 1) 129

=================================================================

Total params: 522,561

Trainable params: 521,657

Non-trainable params: 904

_________________________________________________________________

As you can see that our model has about 500K trainable parameters. There are a few steps required to train the model.

opt = Adam(lr=0.0001, beta_1=0.9, beta_2=0.999, decay=0.01)

model.compile(loss='binary_crossentropy', optimizer=opt, metrics=["accuracy"]) model.fit(X, Y, batch_size = 5, epochs=3)

Now we have a trained model to detect our trigger word from an audio file. Lets see how to do the prediction.

Prediction

We build a function which takes in an audio file with positive and negative words overlaid on the background noise audio convert it into a spectrogram and feed it to our model for prediction.

def detect_triggerword(filename):

plt.subplot(2, 1, 1) x = graph_spectrogram(filename)

x = x.swapaxes(0,1)

x = np.expand_dims(x, axis=0)

predictions = model.predict(x)



plt.subplot(2, 1, 2)

plt.plot(predictions[0,:,0])

plt.ylabel('probability')

plt.show()

return predictions

The above function returns the predictions in a form of a vector of values between 0 and 1 depicting the probabilities of presence of a trigger word at a time step. If we plot the spectrogram and the output prediction values we will see a peak depicting that the model has detected a trigger word.

You can insert a audio cue, blink a light, print something on your screen, or maybe connect it to your lamp to light it just after the trigger word is detected.

Ending Remarks

The above flow outlines a basic implementation of a trigger word detection model. You can do a lot of customization to this implementation to make it real time detection so that you don’t have to record the audio beforehand. Now that you know how the trigger word detection works. With this new knowledge you can now revisit the question

Is Alexa and Google listening to all my conversations?

This implementation was inspired by the Deep Learning Specialization course on Coursera by Andrew Ng