DIY AI: An old school matrix NN. How do NN work and how to create and... | by Andrew Tch | Towards Data Science

How do NN work and how to create and train NN like many years ago. Part of DIY AI series.
You will need Python and Python IDE. Consult DIY AI installation manuals if you need to install them (you will need Python and Console parts for this story).
You can download the full source code from Github here.
The draw.io diagram is accessible here.
As I have mentioned in previous article, since early 50s we tried to mimic the human brain and a smallest part of it — Neuron — itself.
The closest thing that acts like Neuron is a Perceptron — which is composed of multiple inputs, summing engine, single output and an activation function. Like so:
How does this thing work? It takes a lots of inputs (and input is actually a number, like 0.33, -13.7 an so on), multiples them by their weights (w1...wn in the drawing) and sums them up. The sum is then being fed to an activation function, and passed to output.
Why do we need an activation function? Mostly we require output values between 0...1, like “yes” or “no”. It is hard to adjust weights the way that they respond to any input with ones and zeroes, but, luckily, we have a logistic function for that that maps any numeric value to a value between 0 and 1 (feeding very big negative number, like -10000 to the function will return something close to zero, and feeding a very big positive number will result in something close to 1, like 0.99...).
Of course, we can join Perceptrons into layers, and form networks, like so:
Inputs are fed to every input on every Perceptron on the first layer, then the output of the first layer is fed to all the inputs of the next layer, repeating the same procedure for all following layers. The outputs of the last layer are actually the outputs of a Neural Network. This network is called feed-forward, because the data only flows in forward direction with no back-loops, and this network is fully-connected, as every Neuron receives outputs of all Neurons from previous layer.
To keep the widely used terminology, layers between input and output layers are called hidden layers.
If the input is a vector of features (like: does it have wings? does it live in a water? does it have fur? does it eat meat?) and the output is a vector, representing, for example a probable class of “it” (like: animal — 0.87, fish — 0.12), this type of network can be used for classification of inhabitants of the zoo.
So, what is the magic, you may ask? Weights are. The actual Holy Grail of NN is the method of adjusting and computing weights, called training a neural network. I will cover one of the simplest methods — Backpropagation in the article.
Every developer imagines Neural Network as a set of Neurons that can be coded like this:
class Neuron    array Neuron weights[]    float output    function compute_output()
...and so on, but this is over-engineering, since we can abstract a whole layer with with one simple thing: Matrix.
In an example above, Layer one takes 4 inputs and produces 3 outputs. The input is 4 numbers.
If we represent Input as 1x4 matrix, and Layer one by 4x3 matrix containing all Layer one weights, we can get Layer one out simply by multiplying input matrix (1x4) by Layer one matrix (4x3), creating 1x3 matrix. This matrix is multiplied by 3x6 Layer two matrix, and then 6x2 Layer three matrix, getting 1x2 Result matrix. Of course, we must apply an activation function to each member of the result matrix before passing it to the next layer.
Speaking in visual terms, NN from Fig 2. is processed like so:
So, basically, in NN, whole layers are abstracted as Matrixes, and most used function to simulate NNs is Matrix multiplication. That actually explains why NN systems are easily accelerated by GPUs — the 3D visualisation is heavily based on matrix algebra, too!
In multi-layer neural network we basically have only input and output matrixes. We can, however, compute difference between expected and actual results, but how do we adjust the weights on many layers with different Neuron count?
The idea is simple — we can go back, to the input layer, layer by layer comparing layer input with layer error. Each layer, no matter how deep it is, contributes to next layer error; therefore, we can adjust weights by value we can get by multiplying previous layer output by output error multiplied by current layer output. This is called Backpropagation, and it sound extremely bad and complicated when formalised, but is pretty simple to implement.
So, let’s take some classic example — like handwritten number recognition.
Behold, out first dataset, the famous Semeion dataset: http://archive.ics.uci.edu/ml/datasets/semeion+handwritten+digit
The dataset file contains 1593 digits, handwritten, scanned, and fit into black and white box of 256 pixels (16x16 pixel square).
The pixels (256 in a row) are put in separate lines of dataset file and appended with 10 bits representing actual result (e.g. 0 0 0 1 0 0 0 0 0 0 means 3).
We can take an example row of dataset which looks like so:
0.0000 0.0000 0.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 0.0000 0.0000 0.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 1.0000 1.0000 0.0000 0.0000 0.0000 1.0000 1.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 1.0000 1.0000 0.0000 1.0000 1.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 1.0000 1.0000 1.0000 0.0000 1.0000 1.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 1.0000 0.0000 1.0000 1.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 1.0000 1.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 1.0000 0.0000 1.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 1.0000 1.0000 0.0000 1.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 1.0000 1.0000 0.0000 0.0000 1.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 1.0000 1.0000 0.0000 0.0000 0.0000 1.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 1.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1 0 0 0 0 0 0 0 0 0
If we remove the .0000 part of the numbers and wipe extra spaces the example will look like so:
000111111111100001111100000111000111000000001110111000000000011111000000000000111101110000000011011100000000001111100000000000111100000000000110110000000000111011000000000111001100000000111000110000011111000001111111100000000111111100000000000111000000000 1 0 0 0 0 0 0 0 0 0
And, if we split it to 16 character lines we will get the actual drawn zero:
00011111111110000111110000011100011100000000111011100000000001111100000000000011110111000000001101110000000000111110000000000011110000000000011011000000000011101100000000011100110000000011100011000001111100000111111110000000011111110000000000011100000000001 0 0 0 0 0 0 0 0 0
Understanding your dataset is vital for successful training. Clean and precise dataset is kind of the greatest part of your model.
Ok, time for theory is over, let’s code something. First of all, we need to load the dataset into python code. I will use excellent numpy library for matrix processing. To store data, we need to define two numpy arrays (samples and results), and read the dataset line by line, adding first 256 numbers of each line to samples array and second ten to results array:
Fancy, eh? Now, lets build a neural network and make sure we can run out samples through it!
For this example I will use two layer neural network, where first layer is 256x256 matrix, and the second one is 256x10. The overall schema looks like so (note how the shapes of matrixes change as the data flows through the neural network):
The layers will be represented by numpy arrays, and pre-filled with relatively small random values (ranging from -0.01 to 0.01):
The code above actually shows how do you use trained neural network. Since out small net is not prepared yet, second_output will contain completely random data. Let’s fix it and train the network!
Training a NN is an iterative process — we take a certain part of dataset (I will start with 100 random records), pass all samples of dataset though NN, compute error and backpropagate the error. One pass of training dataset is called an Epoch.
NN is trained until it is good or it is stuck — this means that we need to limit a number of epochs (to prevent infinite training) and define some criteria that defines the NN as “good enough”.
I will use sum of absolute values of errors of a sample as a sample error, and maximum error of an epoch as Epoch error. I will consider NN as done as the Epoch error is less than 0.1 (for example, error of -1 0 1 0.5 when correct answer is 0 0 1 0 is 1.5).
Again, let’s remember how our NN will be processed:
The code supporting main training loop looks like so:
As you can see I also defined a rate that represents how fast NN reacts to changes; in most cases, the bigger is the network the smaller it should be — this makes training slower but more precise.
After getting output matrixes we need to compute output error (and add the value to current epoch errors):
As you remember, all outputs are processed via the sigmoid function, and we need to reverse it passing data back. Therefore, we create dsigmoid function which is a derivative of a sigmoid:
At the current time we have 1x10 error matrix, and we have 256x10 matrix to adjust. Therefore, we must find how much each of the weight on Layer two contributed to error by multiplying second Layer error with dsigmoid’ed second layer output:
Now we need to create 256x10 matrix to adjust Layer two weights. To do so, we take previous (Layer one) output (1x256), transpose it to 256x1 and multiply this to newly found Layer two delta. Adding the resulting 256x10 matrix (multiplied by rate) to Layer two weights will give us the new weights that hopefully will produce better results:
We can repeat the same procedure for Layer one, but what is Layer one error? It is actually easy to find — we pass second Layer delta through transposed second Layer matrix; speaking abstract terms we find how much each first layer output value contributed to second layer error. After the error matrix has been found, we can repeat the same procedure as above to adjust Layer one weights:
We also need a bit of control, so in the end of each epoch, let’s compute error and print results:
Please download and study the actual code before moving to the next chapter!
Ok, let’s run the model (again, you can download complete model from the Github). For convenience, I’ve added simple code piece that takes 12 random samples and tests the neural network against it:
So...let’s run the net against a small part of the dataset (100 records):
Success! After 40 Epochs the NN converged, and is to able detect numbers from the samples correctly!
Ok, let’s test it will full dataset (~1500 samples). Run it (if you downloaded the code from Github, it already contains full dataset), aaaaand...
And the NN will hang miserably and infinitely, with error value around ~1.
What happened? This is common case of overfitting — we were dragging the NN around the same dataset again and again, and the NN became good, but not good enough — not enough to minimise the error. It is also stuck no — no matter how many times we loop through the same dataset, the error will not decrease.
Overfitted models can still be usable — you can limit epoch count to, for example, 50 and get acceptable results:
...but this example illustrates one of the many reasons of first AI winter in mentioned in the first story — for example, the shocking discovery that you can’t just feed 10x more data to NN and expect it to become 10x more precise. You need to adapt, change your network structure, training modes and experiment a lot in overall for the technology to be usable. The actual approaches will be covered in future stories ).
Because it is not that simple. You could see how the behaviour of NN changes from the amounts of data, and it also depends of methods of training, number of epochs and network architecture.
In the future stories in the DIY AI series I will tell more about architectures, approaches, and — most important — frameworks — that dramatically reduces the amount of code you have to write and allows you to concentrate on actual data and architecture, and not matrixes.
Stay tuned!
Thanks to Trask for providing initial idea and UCI machine learning repository for datasets.