Speech Recognition Analysis. From Siri to smart home devices, speech... | by Yihui April Chen | Towards Data Science

From Siri to smart home devices, speech recognition is widely used in our lives. This speech recognition project is to utilize Kaggle speech recognition challenge dataset to create Keras model on top of Tensorflow and make predictions on the voice files.
The link of the Kaggle speech recognition challenge dataset is listed below:
www.kaggle.com
Similar to image recognition, the most important part of speech recognition is to convert audio files into 2X2 arrays.
Sample rate of an audio file represents the number of samples of audio carried per second and is measured in Hz. The following image shows the relationship between the audio raw wave and sample rate of “bed” audio file:
train_audio_path = 'C:\\Users\\...\\train\\audio'filename = '\\bed\\00f0204f_nohash_0.wav'sample_rate, samples = wavfile.read(str(train_audio_path) + filename)ipd.Audio(samples, rate=sample_rate)print(sample_rate)print(samples)
# visualize this audio wave:fig = plt.figure(figsize=(14, 8))plt.plot(samples)plt.title('Raw wave of ' + filename)plt.ylabel('Wave')plt.show()
Next, I will introduce two important attributes of audio files:
The spectrogram is a spectro-temporal representation of the sound. The horizontal direction of the spectrogram represents time, the vertical direction represents frequency. (1) Spectrograms can be used as a way of visualizing the change of a nonstationary signal’s frequency content over time.
The formula of spectrogram (2) is listed below:
I utilized the code from the below sites to calculate and visualize the log_spectrogram:
https://www.tensorflow.org/api_guides/python/contrib.signal https://github.com/Tony607/tf_audio_signal/blob/master/tf_audio_signal.ipynb
def log_spectrogram(file, label):    sample_rate, samples = wavfile.read(str(train_audio_path) + '\\'+label+'\\' + file)    signals = tf.cast(tf.reshape(samples, [1,-1 ]),tf.float32)     spectrogram = signal.stft(signals, frame_length=1024, frame_step= 512)    magnitude_spectrograms = tf.abs(spectrogram)    log_offset = 1e-6    #When compressing with a logarithm, it's a good idea to use a stabilizing offset     #to avoid high dynamic ranges caused by the singularity at zero.    log_magnitude_spectrograms = tf.log(magnitude_spectrograms + log_offset)    return log_magnitude_spectrograms
Then plot the log_spectrogram of sample data: bed.
log_spe_bed = log_spectrogram(train.file[0],train.label[0]).numpy()array_bed = log_spe_bed.astype(np.float)[0]fig = plt.figure(figsize=(14,8))#plt.ylabel("Freqs in Hz")plt.xlabel("Log_Spectrogram")plt.imshow(np.swapaxes(array_bed,0,1).T)
Mel Frequency Cepstral Coefficent (MFCC) is the feature that is widely used in automatic speech and speaker recognition. The Mel scale relates perceived frequency, or pitch, of a pure tone to its actual measured frequency. Humans are much better at discerning small changes in pitch at low frequencies than they are at high frequencies. Incorporating this scale makes our features match more closely what humans hear. (3)
def mfcc(file=train['file'].tolist(), label=train['label'].tolist()):    sample_rate, samples = wavfile.read(str(train_audio_path) + '\\'+label+'\\' + file)    if len(samples) < 16000:        samples = np.pad(samples, (0,16000-len(samples)), 'linear_ramp')    else:        samples = samples[:16000]    signals = tf.cast(tf.reshape(samples, [1,-1 ]),tf.float32)     spectrogram = signal.stft(signals, frame_length=1024, frame_step= 512)    magnitude_spectrograms = tf.abs(spectrogram)    num_spectrogram_bins = magnitude_spectrograms.shape[-1].value    lower_edge_hertz, upper_edge_hertz, num_mel_bins = 80.0, 7600.0, 64    linear_to_mel_weight_matrix = tf.contrib.signal.linear_to_mel_weight_matrix(num_mel_bins, num_spectrogram_bins, sample_rate, lower_edge_hertz,upper_edge_hertz)    mel_spectrograms = tf.tensordot(magnitude_spectrograms, linear_to_mel_weight_matrix, 1)# Note: Shape inference for <a href="../../api_docs/python/tf/tensordot"><code>tf.tensordot</code></a> does not currently handle this case.mel_spectrograms.set_shape(magnitude_spectrograms.shape[:-1].concatenate(linear_to_mel_weight_matrix.shape[-1:]))    log_offset = 1e-6    log_mel_spectrograms = tf.log(mel_spectrograms + log_offset)    num_mfccs = 13# Keep the first `num_mfccs` MFCCs.    mfccs = tf.contrib.signal.mfccs_from_log_mel_spectrograms(log_mel_spectrograms)[..., :num_mfccs]    return mfccs.numpy()[0]
By using the “mfcc” function that is defined above, it is easy to calculate the mfcc of the audio file of “bed” and visualize its MFCC.
mfcc_bed = mfcc(train.file[0],train.label[0])fig = plt.figure(figsize=(14,8))plt.ylabel("MFCC (log) coefficient")plt.imshow(np.swapaxes(mfcc_bed,0,1))
I built a sequential neural network model, which is the simplest method to build a model in Keras —it tells keras to stack all layers sequentially. Then I added four dense layers, which are fully connected layers in the model.
After building the model, I used Adaptive Moment Estimation as optimizer, categorical crossentropy as loss, and accuracy as metrics to compile the model.
# Dense(64) is a fully-connected layer with 64 hidden units.# in the first layer, you must specify the expected input data shape:# here, 20-dimensional vectors.with tf.Session() as sess0:    assert not tf.executing_eagerly()    model = Sequential()    model.add(layers.Dense(32, input_shape=X_train_array.shape[1:], activation='tanh'))    model.add(Dense(64, activation='tanh'))    model.add(Dense(128, activation='tanh'))        model.add(Flatten())    #model.add(Dense(256, activation='relu'))    model.add(Dense(30))    model.add(Activation('sigmoid'))        model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])    #Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments.    model.summary()      #  history = model.fit(x=X_train_array, y=y_train_array, epochs=5, verbose=1, validation_split = 0.33, shuffle=True, class_weight=get_class_weights(pd.Series((list(set(labels))),dtype='category').cat.codes.values),batch_size=batch_size)     history = model.fit(x=X_train_array, y=y_train_array, epochs=25, verbose=1, validation_split = 0.1, shuffle=True, class_weight=get_class_weights(pd.Series(Y_train,dtype='category').cat.codes.values),batch_size=128)         model_evaluation = model.evaluate(x=X_test_array, y=y_test_array, batch_size=None, verbose=1)    prediction = model.predict(X_test_array, batch_size = 128, verbose = 1)        april_tst = model.predict(mfcc_april_test, batch_size = 128, verbose = 1)    sess0.close()
I chose 25 as the number of epochs, which is the number of times the model will cycle through the data. After running around 20 epochs, the validation accuracy of the model is improved to 61% — 62%.
As we can see from the above two pictures, the test and train accuracy is not close enough to each other, which means this model can be improved by overcoming over-fitting problems.
Audio file is usually converted to array to be the input of Keras model.Spectrogram and MFCC are the two features of audio files to be converted to arrays.We can modify the layers of Keras model to increase the model accuracy.Be aware of over-fitting problems by comparing training and testing accuracy.Sequential model is easier to modify, compared with Keras API model.
Audio file is usually converted to array to be the input of Keras model.
Spectrogram and MFCC are the two features of audio files to be converted to arrays.
We can modify the layers of Keras model to increase the model accuracy.
Be aware of over-fitting problems by comparing training and testing accuracy.
Sequential model is easier to modify, compared with Keras API model.
To accurately predict the voices with noises in the “test” audio file in Kaggle dataset, I need to process current training data by adding background noise.Since there are unknown voices in the Kaggle “test” audio file, I also need to add “unknown” as one of my labels.I can also normalize the mfccs of training data to see if I can improve the model accuracy.I can also use the combination of CNN and RNN to see if I can improve the model accuracy or not.
To accurately predict the voices with noises in the “test” audio file in Kaggle dataset, I need to process current training data by adding background noise.
Since there are unknown voices in the Kaggle “test” audio file, I also need to add “unknown” as one of my labels.
I can also normalize the mfccs of training data to see if I can improve the model accuracy.
I can also use the combination of CNN and RNN to see if I can improve the model accuracy or not.
Paul Boersma and David Weenink, Intro 3.1. Viewing a spectrogram, Institute of Phonetics Sciences of the University of AmsterdamWikipedia: https://en.wikipedia.org/wiki/Short-time_Fourier_transformCryptography: http://practicalcryptography.com/miscellaneous/machine-learning/guide-mel-frequency-cepstral-coefficients-mfccs/Chris Dinant, Kaggle Tensorflow Speech Recognition Challenge
Paul Boersma and David Weenink, Intro 3.1. Viewing a spectrogram, Institute of Phonetics Sciences of the University of Amsterdam
Wikipedia: https://en.wikipedia.org/wiki/Short-time_Fourier_transform
Cryptography: http://practicalcryptography.com/miscellaneous/machine-learning/guide-mel-frequency-cepstral-coefficients-mfccs/
Chris Dinant, Kaggle Tensorflow Speech Recognition Challenge