Audio signal feature extraction and clustering
By: Aakash Mallik
Tags: Machine Learning, Clustering, Unsupervised Learning, TensorFlow, Coding

Code

We firstly import all the necessary libraries in our jupyter-notebook. This is mine.

Follow along.

import tensorflow as tf

import numpy as np

import pandas as pd from pyAudioAnalysis import audioBasicIO #A

from pyAudioAnalysis import audioFeatureExtraction #B

import matplotlib.pyplot as plt import os #C

#A — This function is used to extract audio data like Frame rate and sample data of the audio signal.

#B — This function is responsible for extracting all the features from the audio signal that we talked about earlier.

#C — This is basically used to iterate through the music files in the file-system.

This is where we get our hands dirty. Let us start with the first point in our objective — Extraction. We start by defining the utility functions.

def preProcess( fileName ): [Fs, x] = audioBasicIO.readAudioFile(fileName) #A



#B

if( len( x.shape ) > 1 and x.shape[1] == 2 ):

x = np.mean( x, axis = 1, keepdims = True )

else:

x = x.reshape( x.shape[0], 1 )

#C

F, f_names = audioFeatureExtraction.stFeatureExtraction(

x[ :, 0 ],

Fs, 0.050*Fs,

0.025*Fs

)



return (f_names, F)

This function takes file name as an argument.

#A — We call the function provided by the library to read the audio data. This returns Fs and x. Fs is the framerate of the audio sample and x is a numpy array representing the sample data that you see in any music editing software.

A visualization of an audio file in Audacity

#B — Did you notice that the picture I provided above has two sample data( The yellow wave live thing ) for same song. This is because it has two channels, one for the right and the other for left (Stereo). But we can only deal with a single channel (Mono). So we basically check if the sample data in fact has data for two channels, and if it does, then we take the mean of the two channel data.

#C — This function extracts all the 11 features that I had listed earlier and returns the feature names and their values in a numpy matrix. The f_names (feature names) will not be really useful as we know which row of the F matrix holds which data. The first argument is the sample data itself. The second and third argument are the window size ( 50 msec ) and step amount ( 25 msec ) respectively.

Lastly we return the the data from this function.

def getChromagram( audioData ):



# A

temp_data = audioData[ 21 ].reshape(

1,

audioData[ 21 ].shape[0]

)

chronograph = temp_data



# B

for i in range( 22, 33 ):

temp_data = audioData[ i ].reshape(

1,

audioData[ i ].shape[0]

)just say you love me

chronograph = np.vstack( [ chronograph, temp_data ] )



return chronographjust say you love me

Like we discussed earlier, we are planning to use only the chronagram feature of the audio signals hence we will separate out that data from the rest of the function.

#A — We pick the “chroma_1” feature from the audioData numpy matrix and create a new numpy matrix.

#B — We loop through the rest 11 chroma and stack them vertically on top of each other.

This is how the final matrix looks like

def getNoteFrequency( chromagram ):



numberOfWindows = chromagram.shape[1] #A



freqVal = chromagram.argmax( axis = 0 ) #B



histogram, bin = np.histogram( freqVal, bins = 12 ) #C



normalized_hist = histogram.reshape( 1, 12 ).astype( float ) / numberOfWindows #D



return normalized_hist

Like we discussed earlier, now we need to find the most prominent note in each window, and then we wish to find the frequency with which each of the 12 notes are hit.

#A — Total number of time frame aka windows. This will be useful while normalization step.

#B — Find the index of the most prominent note in the vertical axis ( axis = 0 ). This gives us a number between 0–11. The output might look something like this:

[0, 0, 0, 1, 1, 3, 9, 9, 11, 0 …number of windows]

#C — Now we basically get the count of each note using the inbuilt function.

#D — Finally we normalize the data for the reason we mentioned above. This is the final feature vector which will define our data point. This will be a (1x12) vector — The relative frequency for each of the 12 notes.

Note: The plotting functions are trivial and can be directly copied from matplotlib examples page. You can play around with these functions and different files to see how the chromagram and frequency plot changes with different genre of music.

fileList = [] def getDataset( filePath ):

X = pd.DataFrame( )



columns=[ "G#", "G", "F#", "F", "E", "D#", "D", "C#", "C", "B", "A#", "A" ]



for root, dirs, filenames in os.walk( filePath ):

for file in filenames:

fileList.append( file )

feature_name, features = preProcess(filePath + file )

chromagram = getChromagram( features )

noteFrequency = getNoteFrequency( chromagram )

x_new = pd.Series(noteFrequency[ 0, : ])

X = pd.concat( [ X, x_new ], axis = 1 )



data = X.T.copy()

data.columns = columns

data.index = [ i for i in range( 0, data.shape[ 0 ] ) ]



return data

Nothing special in this, we are just iterating over all the files in the filepath directory. We have finally created a pandas Dataframe out of our feature vector. The reason I prefer pandas dataframe over numpy matrix is because it makes the data look beautiful with column names and indices. I believe in making my work look pleasing so it is totally fine if you disagree and stick to numpy. Here is my dataframe.