Classifying emotions using audio recordings and Python | by Tal Baram | Towards Data Science

Emotion recognition technologies have been widely used in a variety of different applications in the past few years and seem to be only growing more popular in today’s industry. from psychological studies to advanced analysis of customer needs, it seems like the potential of these technologies is inexhaustible. Yet, these methods are still perceived as incredibly complex and inaccessible for the average data scientist who do not hold the privilege of being an expert in analysis of audio frames and sound waves.
In this article we’ll aim at making this process as accessible and simplistic as we can by showing an example of an Emotion-Recognition classifier, using python and Librosa- a python package that makes the analysis of audio files incredibly easy and straight forward. We’ll go through this process by explaining each step both theoretically and technically.
First, in order to understand how to create an algorithm that processes audio sounds and understands their inherent meanings, we first need to understand how the human brain does the exact same thing.
The first term we’re going to talk about is the Sound Wave. Sound could be defined as a mechanical disturbance that propagates through an elastic material medium, such as air. It propagates in longitudinal waves (hence the term ‘Sound Waves’) and consists of alternating compressions and rarefactions, or regions of high pressure and low pressure, moving at a certain speed. The way our bodies able to create Sound Waves originates in our Glottal Pulse, which manipulates the folds of our vocal cords when we’re speaking. That is obviously very helpful for generating sounds but wouldn’t be enough to actually make sense of them, let alone communicate with each other. To accomplish that, we have our Vocal Tract.
The Vocal Tract is a term that describes a system of different organs in our mouth and throat- Nasal cavity, the tip of the tongue, teeth, soft palate and more. The goal of this system is to serve as a filter to our Glottal Pulse in a way that makes sense of the different sounds we’re generating. To make things easier, we can say that our speech is a result of the different movements and manipulations we’re applying on our Glottal Pulse using our Vocal Tract. We’ll talk more about the ways we can analyze these Sound Waves later, but for now let’s keep these details in mind as we dive into the more pragmatic part of this task.
To create this algorithm, we combined three different datasets which included voice recordings and their respective labels- Happy, Sad, Angry, Calm etc. The datasets we used are RAVDESS, TESS and SAVEE. Since the final dataset seemed to be pretty imbalanced toward some of the features (for example, we had much fewer male recordings then female recordings, and a relatively small number of ‘positive’ emotions compared to ‘negative’ ones) we decided to start with a simplified model first- classification of 3 different classes for both male and female (overall 6 different classes[1]):
mydf.head()
mydf.info()
Using Librosa, which we mentioned earlier, we managed to plot the raw display of one of our sound waves:
ind = np.random.randint(0,len(mydf))data, sampling_rate = librosa.load(mydf['path'][ind], sr=44100)emotion = mydf['emotion2'][ind]plt.title(f'Sound wave of- {emotion}')librosa.display.waveplot(data, sampling_rate)plt.show()
Then we displayed the same signal in a time-frequency domain, so that we could examine the different frequencies and amplitudes of our signal over time. This is done by performing the Fourier transformation on our data:
D = np.abs(librosa.stft(data))librosa.display.specshow(D, sr=sampling_rate, x_axis='time', y_axis='linear');plt.colorbar()plt.show()
The plot above, which shows a linear representation of our frequencies over time, doesn’t seem to give us too much valuable information. The reason for that is that human can only hear sounds which are concentrated in a very small frequency and amplitude ranges (unlike dogs, for example). To fix that, we applied log-scale on both the frequency and the amplitudes. Note that after applying log-scale on our data, we’re no longer measuring our signal in units of amplitudes, but rather in units of decibels.
DB = librosa.amplitude_to_db(D, ref=np.max)librosa.display.specshow(DB, sr=sampling_rate, x_axis='time', y_axis='log');plt.colorbar(format='%+2.0f db')plt.show()
This result is much more informative as we can see the decibels of the different frequencies over time. The formal term for this representation is called Spectrogram. The Spectrograms alone are also proven to be incredibly useful in classifying emotions thanks to their ability to approximate human-based perception of sounds.
Now that we learned about the Spectrogram, let’s consider the spectrum of our sound wave; that is, the representation of the Decibels against the Frequencies in our given time frame:
plt.magnitude_spectrum(data, scale='dB')plt.show()
The log-scale spectrum is achieved by applying Fourier transformation on our data, and then log-scale transformation on the result. The difference between Spectrograms and log-scale spectrums, which are both being achieved by similar mathematical operations, is that while the first displays the frequencies and decibels over time, the latter shows the relation between the decibels and the frequencies. A subtle difference but an important one.
Going back to our definition of speech, we can say that our log-scaled spectrum is a pretty accurate representation of the speech itself. As we mentioned earlier, speech could be described as a combination of the Vocal Tract and the Glottal Pulse. In order to analyze our speech efficiently we needed to extract the Vocal Tract, which resembles a filter and includes the integral information regarding the meaning of the sounds, from the speech without the additional noise. To simplify things- we needed to extract the Vocal Tract from the speech without the Glottal Pulse.
Now let’s consider a smoother version of our log-scaled spectrum- the Spectral envelope:
The plot above resembles the main and most important parts of our log-scaled spectrum- we can examine the main periodicity, the maximum points (marked with red), and the general trend as well. This Spectral envelope is the equivalent of what we considered as the Vocal Tract, and its maximum points are called ‘Formants’. These peaks contain the main ‘identity’ of the sound and are considered as the most important parts of our sound wave. Research shows that our brain can identify most of the meaning of the speech just by its Formants.
Now that we recognized our Vocal Tract, we need to find a way to extract it from our speech. For that we’re going to need to perform just a few more mathematical operations. First, we’ll need to transform our log-scaled spectrum to a Mel-scale[2] and then perform a Discrete Cosine transform on our data. The last transformation is relatively similar to the Fourier transform we used before, as it also creates a spectrum from our current state. So basically, what we’re doing here is creating a spectrum on our previous spectrum. Our new spectrum is now called Cepstrum, while our new frequencies are called Quefrencies. The reason for all that confusion is pretty simple- using our new Cepstrum and Quefrencies we can differentiate easily between our Spectral envelope (the Vocal Tract) and the noise (the Glottal Pulse). More then that, the Discrete Cosine transformation already results in the main and most important coefficients of this cepstrum- the Mel-frequency cepstrum coefficients, also known as the MFCC’s!
Lucky for us, Librosa makes the job incredibly easy, and we can extract the MFCC’s in one simple line of code, as we’ll soon see.
To use the MFCC’s as our features we first needed them to be in a homogenous shape. Since each one of our Signal Waves was of slightly different length, the procedure would have led to MFCC’s of different lengths as well, which won’t be very helpful in creating a model for our data. Therefore, we started by making all the Signal Waves the same length- signals shorter than the mean were padded (using the median) and signals longer than the mean were truncated (we also took down the ones with extreme lengths beforehand).
Afterwards, we extracted the MFCC’s easily from each Signal Wave. We extracted 20 different coefficients from each Signal Wave, and each coefficient included a vector of the given length. Note- The traditional number of MFCC’s is usually around 12–13, and the default in the Librosa function is 20. The first coefficients are usually the ones that carry most of the important information about the speech, therefore using a larger number of MFCC’s doesn’t improve the model in most cases.
mfccs = []for i in tqdm(X):    mfcc = librosa.feature.mfcc(y=i, sr=44000, n_mfcc=20)    mfcc = mfcc.T    mfccs.append(mfcc)
As we can see, each sample includes 20 MFCC’s of length 236 each[3]. We also dropped 3 outliers along the way. Now, we were finally ready for modeling.
RNN networks are known to work well for speech recognition tasks. however, there’s a strong body of research that proves that CNN networks can outperform RNN networks in a lot of cases. In this case, we decided to go for CNN networks using 1-dimensional convolution layers and 1-dimensional pooling layers (as our training data is made of 3 dimensions).
Our CNN network consisted of two blocks, each built of a 1-dimensional convolution layer, activation function (‘ReLu’), 1-dimensional pooling layer and dropout. The two blocks were followed by two fully connected dense layers and a ‘SoftMax’ activation function, as we are dealing with a multi-class problem:
model = Sequential()model.add(layers.Conv1D(256, 5,padding='same',                 input_shape=(236,40)))model.add(layers.Activation('relu'))model.add(layers.MaxPooling1D(pool_size=(8)))model.add(layers.Dropout(0.2))model.add(layers.Conv1D(128, 5,padding='same'))model.add(layers.Activation('relu'))model.add(layers.MaxPooling1D(pool_size=(4)))model.add(layers.Dropout(0.1))model.add(layers.Flatten())model.add(layers.Dense(64))model.add(layers.Dense(7))model.add(layers.Activation('softmax'))
We managed to reach relatively good results, reaching 90% accuracy. As expected, the scores of the male recordings were significantly lower than the female ones, as we had much fewer male samples in our dataset:
After that we aimed at a more complex model- classifying different emotions: ‘fear’, ‘surprise’, ‘sadness’, ‘disgust’, ‘happy’, ‘angry’ and ‘neutral’[4]. The distribution of the samples was more balanced than the one we had before, so we expected a more balanced classification this time.
We applied the same processing to the data as we have before and ran the exact same model. These are the results we reached:
As expected, this classification seems relatively balanced, and with pretty good results as well. We can also observe the Confusion matrix, which shows that the vast majority of our samples were classified correctly:
It’s interesting to see that quite a lot of samples that were predicted as ‘neutral’ turned out to be ‘sadness’ (1.5%). This could possibly be explained by the subtle characteristics of sadness when it is being expressed by voice, characteristics that could very well be mistaken as being associated with a neutral emotion instead.
We then wanted to examine the relations between these classes a bit further, so we used dimensionality reduction with PCA and KMeans clustering, where we classified the data into 7 clusters. Our hope was to find 7 unique groups that would correspond to our 7 emotions. We examined the relations between the cluster’s labels and the classes predictions by checking the correlations between them and looking at the distribution of the classes in relation to the different labels. Eventually we came out with this scatter plot:
We can clearly examine the 7 different clusters. The first thing that we came to notice was that ‘anger’ seems to be relatively detached from the other emotions. ‘Neutral’ seems to be located around the middle, which makes sense of course, and there is also some some proximity between it and between ‘sadness’, which could explain the assumption we made earlier regarding the connection between the two. ‘Disgust’ is also close to ‘sadness’, and ‘happiness’ is close to ‘surprise’- both seem pretty intuitive. It’s a little tougher to explain the location of ‘fear’, as it’s closest neighbour is ‘happiness’, which doens’t make much sense. I’t’s proximity to ‘surprise’ seems more intuitive in this context.
Overall, these results prove that even with a simple CNN network we can classify emotions from audio recordings with high levels of confidence, and even examine the relationships between them.
What makes audio classification so interesting is the way complex theory could be expressed by common practical means. The way mathematical procedures can be used to successfully implement these concepts is nothing less than remarkable. Lucky for us, most of the hard work is being done by pre-made tools. however, knowing the theoretical concepts which lie behind these tools remains the key aspect in building and using these models successfully.
We hope that this article helped by shedding some light on this fascinating subject and making it a little more apprehensible, and hopefully more usable as well.
For more information check out our project right here.
- https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/CNN_ASLPTrans2-14.pdf
- https://www.izotope.com/en/learn/understanding-spectrograms.html
- https://arxiv.org/pdf/2007.11154.pdf
- https://arxiv.org/pdf/2012.14781.pdf
- https://saurabhgarg1996.github.io/files/6.pdf
- https://www.youtube.com/watch?v=4_SH2nfbQZ8&t=1357s
http://www.socsci.uci.edu/~jsprouse/courses/s12/155/lectures/lecture4
https://librosa.org/doc/latest/index.html
[1] The column ‘emotion2’ in our data frame served as our first target and the column ‘emotion3’ served as our second target. Both are being described more thoroughly later in the article.
[2] a scale that resembles humans sound spectrum in a more realistic manner, given the fact that the human ear cannot identify changes in frequencies higher then 500Hz very easily.
[3] We transposed the shape of the MFCC’s for simplicity.
[4] Note that we used the emotions that had more samples in order to reach a balanced classification.