An attempt at Speaker Diarisation
By: Aakash Mallik
Tags: Coding, Audio, Speaker Diarization, Machine Learning

Understanding the Data

As easy as it may appear, feature extraction is not as simple as people think it to be. It is anything but calling some random library APIs on data and getting a set of ready to use feature vectors. Always remember:

“The power of an algorithm comes from the data it works on”

In our case, we are working with audio data and there are tons of features embedded in this simple looking wave plot.

To understand what a wave plot actually is I would suggest you to go through these slides: https://docs.google.com/presentation/d/1BSJzd6W5niJKA99Rf8uHhsN4MCjB_ux1djPRPP9XTKY/edit?usp=sharing

Now, let’s begin with something simple and move up the ladder to some advanced ones that we will actually make use of. I will attach relevant links for further reading.

1. Periodogram (for visualization only)

The wavy diagram that you usually see on your Spotify app is the wave plot of the song you are listening to. In a wave plot you see the amplitude changing over a period of time. Now if you apply a DFT ( Discrete Fourier Transform ) on the audio clip you’ll end up with the same audio clip but in the frequency domain. What this means is, you’ll lose all information related to time, but now you’ll see the individual frequencies of simple component sine waves that make up the whole complex wave.

The reason why we call it the frequency domain is because we have a plot that has frequency on the x-axis and amplitude on the y-axis. What you originally had was time on x-axis and amplitude on the y-axis thus making it the time domain.

The view from the frequency domain is what we call the periodogram

2. Spectrogram (for visualization only)

This visualization is generated by taking small regular sized windows of the whole audio timeline and applying DFT on each of those windows. Stacking them horizontally with a bit of post processing would lead to a Spectrogram.

This gives you a nice looking visualization of the whole audio clip with the frequency and time information intact. This infact is the key difference between Spectrogram and Periodogram.

These two may give you a lot of clues about the audio data visually but for an algorithm to work you need numbers. So now we will talk about the advanced features that I made use of in my code.

3. Spectral Roll-Off

This feature is basically the frequency bin (computers don’t see things as continuous values, instead they have discrete versions of them. In our case, the frequency axis is split into discrete bins) below which 85% of the spectral energy resides.

In the terms of priodogram — This is a measure of the amount of the right-skewedness of the power spectrum.

4. Spectral Centroid

Basically the weighted mean of frequency bins per frame in the spectrogram. The more dominant frequency in a frame would technically move the centroid towards itself.

5. Zero Crossing Rate

Zero cross rate tells us the rate with which the audio wave changes its sign from positive to negative and vise-a-versa. This are particularly useful for distinguishing between percussion instruments.

6. Mel Frequency Cepstral Coefficient

blah

This following video is an absolute gem…

For more text oriented description read the following two links…

Assuming that you have seen the video provided above let’s talk about a crucial decision you’d want to make here. By now you should know that MFCC is a set of 40 and more coefficients that basically provide a nice way of defining audio data as numbers. But for speech to text applications the first 20 coefficients are chosen which basically forms the spectral envelope. This is a crucial piece of information that should give you a hint as to why you should not use spectral envelope for speaker detection. The reason why spectral envelope is used for text to speech is because no matter who is speaking, the phonemes that two distinct individuals are uttering will form similar spectral envelops. Thus the first 20 coefficients are actually speaker independent. The rest 20 and above are called spectral details and they contain the information of pitch and overtones. Thus to actually make use of MFCC for speaker diarisation, we will chose MFCC — 11 to 40 (Even though using all 40 will do you no harm, but using 11–40 is a good feature selection that you can do from your end)

So far…

With all the feature extracted let’s take a peek at our feature vectors.

Each row is a feature vector of an audio frame ( Showing first 5 samples )

So we are all ready with the data, now we shall take a look at the algorithm that we shall apply on the data.