How to build a Neural Network for Voice Classification | by Jurgen Arias | Towards Data Science

Wouldn’t it be great to be able to have transcriptions of our Zoom meetings telling us in plain text who said what during the meeting? This project was born from the idea of transcribing meetings telling you what each person said. We have many transcribing tools for speech to text but not many for identifying each speaker. For this, we will build a python voice classifier from scratch with all the code included here.
First, as always, we need data. This data comes from OpenSLR. We will use 30 different speakers and 4 samples per speaker for our training data, giving us 120 samples. The samples average 15 seconds each thus about one minute per speaker for training. 30 speakers is a good amount of people for our problem statement; and as I have conducted this project already, I can assert that one minute is good enough to give us good results.
In addition to the 120 samples for training, we also need validation and testing data. We will validate on 3 samples and test on 3 samples. Since we have 30 speakers, we will need 90 samples for validation and 90 samples for testing. Therefore, we will need 10 voice clips from each speaker.
To summarize the samples, we will use 120 voice clips for the training data, 90 voice clips for validation data, and 90 voice clips for testing data for a total of 300 voice clips.
Once we have our voice clips in three different folders for training, validation and testing, we can create our pandas dataframes. We need to import os so that we can create a dataframe from the files in our folders (I am working on a mac, for other operating systems, the process might be a little different). My folder for training is called 30_speakers_train.
import os#list the filesfilelist = os.listdir('30_speakers_train') #read them into pandastrain_df = pd.DataFrame(filelist)
This will create a dataframe for my training data. It is very important to check the size of our dataframe, in order to make sure that it matches the number of files in our folder.
Let us name the file column as ‘file’.
# Renaming the column name to filetrain_df = train_df.rename(columns={0:'file'})
Sometimes a ‘.DS_Store’ file is created and we have to find and drop that specific file in order to continue. After that we would need to reset the index of our dataframe. Here is an example of that:
# Code in case we have to drop the '.DS_Store' and reset the indextrain_df[train_df['file']=='.DS_Store']train_df.drop(16, inplace=True)train_df = train_df.sample(frac=1).reset_index(drop=True)
Now, we need to identify each speaker. In my file names, the first set of numbers is the speaker ID thus I wrote a little code to create a column with that ID.
# We create an empty list where we will append all the speakers ids for each row of our dataframe by slicing the file name since we know the id is the first number before the hashspeaker = []for i in range(0, len(df)):    speaker.append(df['file'][i].split('-')[0])# We now assign the speaker to a new column train_df['speaker'] = speaker
Our dataframe should look like this:
We do the same for the validation and testing files and get three different dataframes.
Now that we have the dataframes, we need to write a function to extract the audio properties of each audio file. For this we use librosa which is a great python library for audio manipulation. We need to pip install librosa and import librosa. The following is the function that parses through every file in our folder and extracts 5 numerical features from each file, namely the mfccs, chroma, mel, contrast and tonnetz.
def extract_features(files):        # Sets the name to be the path to where the file is in my computer    file_name = os.path.join(os.path.abspath('30_speakers_train')+'/'+str(files.file))# Loads the audio file as a floating point time series and assigns the default sample rate    # Sample rate is set to 22050 by default    X, sample_rate = librosa.load(file_name, res_type='kaiser_fast')# Generate Mel-frequency cepstral coefficients (MFCCs) from a time series     mfccs = np.mean(librosa.feature.mfcc(y=X, sr=sample_rate, n_mfcc=40).T,axis=0)# Generates a Short-time Fourier transform (STFT) to use in the chroma_stft    stft = np.abs(librosa.stft(X))# Computes a chromagram from a waveform or power spectrogram.    chroma = np.mean(librosa.feature.chroma_stft(S=stft, sr=sample_rate).T,axis=0)# Computes a mel-scaled spectrogram.    mel = np.mean(librosa.feature.melspectrogram(X, sr=sample_rate).T,axis=0)# Computes spectral contrast    contrast = np.mean(librosa.feature.spectral_contrast(S=stft, sr=sample_rate).T,axis=0)# Computes the tonal centroid features (tonnetz)    tonnetz = np.mean(librosa.feature.tonnetz(y=librosa.effects.harmonic(X),    sr=sample_rate).T,axis=0)return mfccs, chroma, mel, contrast, tonnetz
Each mfcc is an array of length 40, chroma is 12, mel is 128, contrast is 7 and tonnetz is 6. Overall we have 193 numerical features for each voice clip from the audio features we have chosen. Feel free to try different ones if you would like from: librosa feature extraction. We apply the function to each single audio file and store that information. Note that this does take some minutes and it might even take hours depending on how much data you are using and your computing power.
train_features = train_df.apply(extract_features, axis=1)
Now we concatenate all those numerical features for each file so that we have a single array of 193 numbers for each file to feed into our neural network.
features_train = []for i in range(0, len(train_features)):    features_train.append(np.concatenate((        train_features[i][0],        train_features[i][1],         train_features[i][2],         train_features[i][3],        train_features[i][4]), axis=0))
Now we can set our X_train to be the numpy array of our features:
X_train = np.array(features_train)
Similarly, we do the same steps from above from the very beginning for our validation and test data. We should get an X_val and X_test.
Now, we are done with our X data. For Y, we need the speakers’ IDs. Remember we are doing supervised learning so we need the target data. We have that from our original dataframes, therefore we just get those values.
y_train = np.array(train_df['speaker'])y_val = np.array(val_df['speaker'])
We need to hot encode y in order to be able to use it for our neural network. You will need to import LabelEncoder from sklearn and to_categorical from keras which uses Tensorflow.
from sklearn.preprocessing import LabelEncoderfrom keras.utils.np_utils import to_categorical# Hot encoding ylb = LabelEncoder()y_train = to_categorical(lb.fit_transform(y_train))y_val = to_categorical(lb.fit_transform(y_val))
Now we need to scale our X:
from sklearn.preprocessing import StandardScalerss = StandardScaler()X_train = ss.fit_transform(X_train)X_val = ss.transform(X_val)X_test = ss.transform(X_test)
And finally, we are ready for the fun part: building the neural network! We will use a simple feed forward neural network. The input will be our 193 features. I played around with the dropout rates but you can change those values if you would like. Using relu is pretty standard. The output is softmax since we have 30 different classes and we use categorical crossentropy because of the different classes.
from keras.models import Sequentialfrom keras.layers import Dense, Dropout, Activation, Flattenfrom keras.callbacks import EarlyStopping# Build a simple dense model with early stopping and softmax for categorical classification, remember we have 30 classesmodel = Sequential()model.add(Dense(193, input_shape=(193,), activation = 'relu'))model.add(Dropout(0.1))model.add(Dense(128, activation = 'relu'))model.add(Dropout(0.25))model.add(Dense(128, activation = 'relu'))model.add(Dropout(0.5))model.add(Dense(30, activation = 'softmax'))model.compile(loss='categorical_crossentropy', metrics=['accuracy'], optimizer='adam')early_stop = EarlyStopping(monitor='val_loss', min_delta=0, patience=100, verbose=1, mode='auto')
Now we fit the model with the training and validation data:
history = model.fit(X_train, y_train, batch_size=256, epochs=100,                     validation_data=(X_val, y_val),                    callbacks=[early_stop])
This is some code to see a graph of the training and validation accuracy:
# Check out our train accuracy and validation accuracy over epochs.train_accuracy = history.history['accuracy']val_accuracy = history.history['val_accuracy']# Set figure size.plt.figure(figsize=(12, 8))# Generate line plot of training, testing loss over epochs.plt.plot(train_accuracy, label='Training Accuracy', color='#185fad')plt.plot(val_accuracy, label='Validation Accuracy', color='orange')# Set titleplt.title('Training and Validation Accuracy by Epoch', fontsize = 25)plt.xlabel('Epoch', fontsize = 18)plt.ylabel('Categorical Crossentropy', fontsize = 18)plt.xticks(range(0,100,5), range(0,100,5))plt.legend(fontsize = 18);
It should look something like this:
It looks like we got some good accuracy! Let’s check the values with our test data. We can generate predictions with the results of our neural network and compare them to the actual values.
# We get our predictions from the test datapredictions = model.predict_classes(X_test)# We transform back our predictions to the speakers idspredictions = lb.inverse_transform(predictions)# Finally, we can add those predictions to our original dataframetest_df['predictions'] = predictions
Now our test data dataframe should look like this:
# Code to see which values we got wrongtest_df[test_df['speaker'] != test_df['predictions']]# Code to see the numerical accuracy(1-round(len(test_df[test_df['speaker'] != test_df['predictions']])/len(test_df),3))*100
You should get something like this:
98.9% correct predictions! I had to run the neural network four times to finally get one of the predictions wrong. In the previous three runs, the neural network got all predictions right but I wanted to show how to find a wrong prediction. Thus with one minute of training audio, the neural network is near perfect for 30 speakers!
This proves how powerful a simple neural network is! I hope it was helpful!
Here is a link to my jupyter notebook with all the code. I also did this and similar projects using Convolutional Neural Networks (CNN), you can see my other post explaining that process here in case you are interested.
Datasets: http://www.openslr.org/12/
[1] David Kaspar, Alexander Bailey, Patrick Fuller, Librosa: A Python Audio Library (2019)
[2] Rami S. Alkhawaldeh, DGR: Gender Recognition of Human Speech Using One-Dimensional Conventional Neural Network (2019)
[3] Kory Becker, Identifying the Gender of a Voice using Machine Learning (2016)
[4] Jonathan Balaban, Deep Learning Tips and Tricks (2018)
[5] Youness Mansar, Audio Classification : A Convolutional Neural Network Approach (2018)
[6] Faizan Shaikh, Getting Started with Audio Data Analysis using Deep Learning (with case study) (2017)
[7] Mike Smales, Sound Classification using Deep Learning, (2019)
[8] Aaqib Saeed, Urban Sound Classification, Part 1, (2016)
[9] Marc Palet Gual, Voice gender identification using deep neural networks running on FPGA, (2016)
[10] Kamil Ciemniewski, Speech Recognition from scratch using Dilated Convolutions and CTC in TensorFlow, (2019)
[11] Admond Lee, How To Build A Speech Recognition Bot With Python (2019)
[12] Adrian Yijie Xu, Urban Sound Classification using Convolutional Neural Networks with Keras: Theory and Implementation, (2019)
[13] Sainath Adapa, Kaggle freesound audio tagging (2018)
[14] Librosa Feature Extraction