Building a Metallica-detecting neural network with TensorFlow: Part 1 - Cutting Code
By: Andy Macdonald
Tags: Machine Learning, Python, TensorFlow, Neural Networks, Artificial Neural Network

Neural networks have always fascinated me. Star Trek TNG’s Data had a positronic neural net which allowed Data to progress towards his goal of becoming more human over time. The Terminator’s neural net processor was capable of reacting and learning from the environment to better serve and/or terminate Sarah & John Connor.

There is something that is just inherently cool (and terrifying) about a machine which is capable of directing its own learning and becoming more than the sum of its parts. These capabilities are now no longer works of Sci-Fi fiction; neural networks are probably serving some of your needs right now in ways you wouldn’t necessarily expect.

As my first stab at entering the world of machine learning, neural networks and artificial intelligence, I decided to document the trials and tribulations of throwing a neural network at a very simple recognition / categorisation problem: is a particular piece of music by Metallica or R.E.M?

A neural network graph generated by TensorBoard

What is TensorFlow?

TensorFlow is an open-source framework released by Google for the purposes of ‘data-flow programming’ and is primarily used for machine learning tasks. It is at heart, an extremely powerful maths library driven by a combination of mathematical approachs:

graph theory : the study of relationships between mathematical objects and structures.

: the study of relationships between mathematical objects and structures. linear algebra : the mathematics of linear equations and their solutions.

: the mathematics of linear equations and their solutions. tensors: multi-dimensional vectors which are analagous to matrices / arrays but are more generalised and usually with higher dimensions.

What is a neural network?

A neural network is a digital analogue of the human brain. The human brain consists of billions of cells called neurons which are connected by weighted connections called ‘synapses’.

Neurons in the brain pass signals from ‘action potentials’ through to a receiving neuron

If enough inputs from a given set of neurons are received at a neuron’s synapse then that neuron will also fire. The firing of neurons against synaptic weights is called forward propagation. The process can also run backwards and is called ‘back propagation’. In this process the weights are recalculated and the neural network is in the process of ‘learning’.

A neural network complete with an ‘input layer’, ‘hidden layer’ and ‘output layer’.

Installing pre-requisites

TensorFlow has GPU support (for NVIDIA GPUs only currently) and CPU support. There are certain performance optimisations you can take advantage of by using the GPU to process on behalf of tensorflow. GPU’s are perfectly suited to churning out masses of vector calculations every single millisecond. Since my current machine doesn’t have an NVIDIA GPU, I will make do with the CPU distribution.

Installing TensorFlow is a simple case of using the pip package manager that usually comes bundled with python:

pip install tensorflow

Afterwards, you can start using tensorflow in python immediately:

import tensorflow as tf

Note: Of course you need python and pip installed as pre-requisites too.

Getting Data into the Neural Network

The first thing we need for any neural network is a set of training data so we can feed it into the base neural network and train a model.

For our specific machine learning problem, we need music samples; some examples of Metallica and examples of R.E.M. I’ve chosen the following because to me they are pretty classic examples of both groups:

Metallica - Fade To Black.mp3

Metallica - Master Of Puppets.mp3

R.E.M. - Everybody Hurts.mp3

R.E.M. - Losing My Religion.mp3

Our first problem to tackle is the audio format. MP3 is a compressed format which means most python libraries (and our neural network) will struggle to work with the format. As the files are compressed, they’re also conceptually only giving us a portion of the picture - we need the original waveform (a WAV file) so we have something that python and TensorFlow can work with.

Our next problem is the size of the individual audio files and the resultant size of the array representing the audio that we will feed to the network. For the python API of TensorFlow, our input training and test datasets need to be a list of numpy arrays.

If we passed 120 seconds of audio in 2-channel WAV form to our neural network at a sample rate of 44.1kHz, the resultant numpy array would have dimensionality of 5292000 x 2. This is pretty large and within that array it will also contain a lot of noise in the data (e.g. periods of silence and audio that isn’t particularly characteristic of a Metallica song).

To tackle both of these problems we will use the following code to chop our music into smaller chunks:

And the following to convert the audio into a lower quality sample rate after the chunks have been created:

Finally we need to take the audio we’ve chopped up into chunks and categorise it somehow. For this example I’ve chose to use the name of the file, some hardcoded categories and to append each entry into a list inside a container object:

The snippet takes an array of categories and generates a one-hot encoding array which represents whether a given sample matches one of the categories. This step is required because categorical data (such as whether a sample matches one label or another) needs to be converted into a form that fits better into a (logistic) regression model.

Composing the Neural Network

There are other parts of the code required to stitch some of the above together but for now lets continue onto creating the actual neural network, which is pretty simple! YAY!

For our particular task we’re pulling in tflearn — a higher level API for TensorFlow so we don’t have to concern ourselves with too much bootstrapping of our neural network.

network = tfl.input_data([None, AUDIO_SHAPE, AUDIO_CHANNELS])

On the first line in the method, we create the first layer of our network — the input layer. Here we define the shape of the numpy array which will be fed into the network. For our purposes, the variable AUDIO_SHAPE is the sample rate (Hz) multiplied by the length of the audio segment (in seconds). For a numpy array of 3000 x 2, you would declare your input layer like this:

network = tfl.input_data([None, 3000, 2])

Onto the other layers:

network = tfl.lstm(network, 30, dropout=0.8)

network = tfl.fully_connected(network, 2, activation='softmax')

network = tfl.regression(network, optimizer='adam', learning_rate=learning_rate, loss='categorical_crossentropy')

We next define another layer in our neural network, an LSTM cell which allows our neurons to hold some form of state — LSTM cells can ‘remember’ or forget information every iteration and use it in future iterations. This is a very useful property which we take advantage of in a RNN (recurrent neural network) for processing data in a sequence. Audio files happen to be data which are in a ‘sequence’ form so LSTM cells are well suited for this particular problem.

Next we define another layer, that we want our network to be ‘fully connected’, so that each neuron in our hidden layer will be connected to each other neuron. An alternative is a ‘convolutional’ network so that only specific neurons make connections to neighbours — but this is for specialised cases and we’re not ready to make that optimisation yet.

We also state our activation function, a function which calculates the weighted sum of the inputs to the neuron. There are several different types of activation function — softmax is quite commonly used in the problem domain so I’ll carry on using it for now. In this layer we also define our number of outputs: 2, Metallica or R.E.M.

Finally we define a regression layer which we use to define and minimise a loss function for training. A loss function is used to calculate a diff between the desired output and true output of the neural network. The regression layer’s purpose is backpropagation and optimisation — it uses this loss function and performs a stochastic gradient descent to optimise the coefficients that exist within the neural network:

Putting it all together

After some refactoring and tweaks we’re ready to run the model and train it against our data. I define a method called batch_reader to pool our audio data into a container:

And then write my main method to train our model. I also define a learning rate ‘hyper-parameter’ to determine how fast our model ‘learns’:

Summing up

That’s our progress so far. So far we’ve created a base model, a means of chopping up and categorising data and to feed it into the neural network for training.

Next up I will cover off training the model, the bumps in the road I experienced and the solutions I found. For now, here’s the whole code:

https://github.com/AndyMacDroo/metallica-tensorflow

Thanks for reading! :)