Creating a Neural Network from Scratch | by Joao Zsigmond | Towards Data Science

Neural networks have been around for many years. In fact, the idea behind the algorithm was first introduced over 60 years ago by a psychologist named Frank Rosenblatt.
It wasn’t until the start of the last decade, however, that these machine learning models started getting more attention, with the publication of this paper, proving the use and effectiveness of neural networks in machine learning.
Today, neural networks are at the core of deep learning. The most complex tasks in artificial intelligence are usually in the hands of Artificial Neural Networks and there are many libraries that abstract the creation of NNs in extremely few lines of code.
In this article we will walk through the fundamental concepts behind neural networks, and understand their inner workings. We will do this by creating a flexible NN from scratch.
The full source code for this article can be found at this link! 👇👇👇
github.com
The code we will be looking at is written in javascript, however, each concept and step will be deeply documented and explained, so you can follow along with any language you like!
Neural networks are a type of algorithm that belong inside deep learning. Deep learning, for that matter, is a subclass of machine learning, that is itself a subclass of artificial intelligence.
The core idea behind machine learning is that algorithms can be trained to classify and process data without being explicitly told the rules of classification.
What this means is that instead of having a model that makes decisions based on hard coded instructions, we can train the model on a great amount of input/output pairs (the training set), and after some time, have the model come up with it’s own rules of classification for a never-before-seen given input.
This idea is valid on all machine learning algorithms.
All right — now that the basics of machine learning are out of the way, let’s move on to neural nets!
Let’s talk about the fundamental structure of a neural network. The data processing works through layers. Any network will have an input layer, a subset of hidden layers (we’ll talk about them in just a sec), and an output layer.
Each layer of a network is composed by a number of neurons, and the neurons of each layer are connected to the neurons of the next through weights.
The input layer is where the data will enter the algorithm. Say we want to create a network to predict the probability of a patient having a certain disease parting from the symptoms he has shown.
In this case, our training data will be structured as an input/output pair. The input will be an array of 0's and 1's representing the symptoms we are analyzing, and the output will be a 0 or 1, representing the infectious status of the patient that presented the ‘input’ symptoms.
If we were to train a neural network on this data, we would have to have an input layer with five neurons, one for each of the symptoms. We can have an arbitrary amount of hidden layers, with an arbitrary amount of neurons in each one, but we must have an output layer of just one neuron.
The neuron in the output layer will have an activation close to 1 when the algorithm thinks that the input represents a sick patient and close to 0 when the algorithm thinks that the input represents a non-infected patient.
You can think of the activation of a neuron as being the value that it stores. The activations of the neurons in the first layer will be the same as the values of our input data. These values will be translated on to the neurons of the following layers through mathematical operations we will check out in a bit.
The activations of the neurons in our output layer will result in our network’s prediction. (It may be just one neuron, as in the case of the example above)
(and the sigmoid function!)
Alright, you might be wondering how the data that enters the input layer transverses the network and ends up in the output layer totally different than how it came in. This process is called feedforward and the full explanation involves some linear algebra and matrix multiplications that we will look at soon. For now we’ll stick with the idea behind the process.
As we saw earlier, each neuron is connected to a neuron of the previous layer through what we called weights. What this means is that the activation of a neuron in, say, the second layer will be relative to the sum of all the activations of the neurons in the layer before it multiplied by the weights connecting them.
If we represent the activation of a neuron with the letter a and the weight connecting two neurons as w, the activation of a single neuron in the second layer may be represented as in the below image.
The equation above is actually way oversimplified for this matter. In reality this is just the first step of calculating the activation of any given neuron. The image below will introduce us to the remaining operations we must execute to find the final activation.
The two new factors that we haven’t yet looked at are: the bias, represented by the letter b; and the activation function, represented by the function g(z).
Let’s start by tackling the bias.
The idea here is that each neuron will have a unique number that will be added to the weighted sum of the the previous neurons’s activations.
Adding the bias is useful so that neural networks can generalize better to any given data. You can think about it as being the constant in a linear function y=mx+c.
Adding a bias to each neuron isn’t fundamental for a network to work properly on simple data, however, when we are tackling very complex tasks it becomes indispensable.
Now that we got down the bias, our activation equation looks something like this:
The last thing we must understand to make sense of how data transverses the network are activation functions.
Lets represent with the letter z the weighted sum of a given neuron added up with it’s bias. The activation of the neuron in question will be the result of an activation function applied to the z factor.
The activation function is used to smoothen out small changes to a neurons activation and keep it between 0 and 1. One very common activation function, and the one that we’ll be using on our own network, is the sigmoid function.
The Sigmoid is useful as an activation function because inputs way greater than 0 will quickly result in a number very close to 1, once passed through the sigmoid function. By the same principle, the smaller the input, the closer the resulting number will be to 0. Input numbers very close to 0 will have a value that smoothly oscillates between 0 and 1.
To sum it all up, we can find the activation of any neuron by adding up the weighted sum of the activations of the neurons in the previous layer with the bias, and passing this result through the sigmoid function.
The process of finding the activation of the neurons in the output layer from any given input is called feedforward. It works by finding all the activations of the neurons in the second layer from the neurons in the input layer and the weights connecting them. The activations of the neurons in the second layer are used to find the activations of the following layer, and so on until the output layer.
The weights and biases are all initialized randomly in an untrained network. The process of training a network means finding the values of these weights and biases that make the resulting output layer of any given input extremely close to the expected output.
Let’s code all of this up, and then we’ll check out how we can teach our network by tweaking these weights and biases to have an optimal performance!
||→[{[{ DISCLAIMER }]}] ←||
We won’t be using any external JS libraries throughout this article except math.js to optimize matrix operations.
All right — Now let’s start coding.
Let’s start by initializing our Network class.
To allow our network to be more flexible, and generalize better to data, we will receive the number of layers, and the number of neurons in each layer in the constructor of our class.
The idea here is that the class will be instantiated with an array. Each elements of the array represents a new layer, and the value of each element is the number of neurons in the respective layer.
To do this, we’ll represent the weights connecting each layer to the previous as a matrix. Each column of this weight matrix will represent the weights connecting all neurons of a previous layer to a single neuron in the next.
For each layer in our network we’ll have a weight matrix, and the weights will be initialized randomly with values oscillating from 0 to 1.
To simplify the code, we’ll set the bias of our network’s neurons to zero.
To recap: The weights attribute of our network will be a list of matrices. Each one of the matrices represents all the weights connecting one layer to the next.
Now that we’ve got all the weights set up, let’s code the feedforward method.
The method will receive an array with the inputs as an argument. These inputs will be mapped to the input layer of our network.
To find the resulting activation of the next layer, we can use the result of a dot product between the input activations and the weight matrix connecting the input layer to the next.
This algebraic operation is analogous to multiplying each activation of the input layer by it’s connecting weight to a neuron in the second layer and summing up these products for each neuron. We can then apply the sigmoid function to each of the resulting weighted sums in the second layer to find all the resulting activations.
These resulting activations will be dot multiplied again by the next weight matrix in our weights array, and the dot product will undergo the sigmoid function, resulting in the activations of the neurons in the following layer. This happens until the resulting activations are the activations in our output layer, hence resulting our prediction.
To implement this in code, we can loop over our weight matrix array. The loop will start by dot multiplying the inputs with the first weight matrix and applying the sigmoid function to the results. The resulting activations will then be mapped to an activation variable, that will be dot multiplied with the second layer of weights and so on.
At every iteration, we’ll update the activation variable so that the dot product between the activation and the weight matrix results in the following layer’s activation (after undergoing the sigmoid, of course).
For the code above to work, we must first code the sigmoid function. The code is pretty simple:
The full code for our network, until now, looks like this:
Right now, our network can already make predictions!!!!
... Well, not exactly.
We can already feed in some input data and receive an output prediction through the feedForward method, but our weights and biases have random values, so our predictions won’t be at all precise. Our network will be guessing blindly the results.
How can we tweak these weights and biases so that our network stops guessing and starts making ‘informed’ decisions?
The complete answer involves partial derivatives and some multivariable calculus, but for now, let’s stick to the idea behind the calculations.
The first step in making our predictions better is finding out how bad they are.
The type of functions that fulfill this task are called cost functions. There are many cost functions available, but the one we will use for our network is the mean squared error function, or MSE.
The idea behind the MSE cost function is to compare the activation of each neuron in the output layer to the expected value of that output. We will subtract our expected output value from our predicted activations and square the result for each neuron. Summing up all these squared errors will give us the final value of our cost function.
The idea here is to tweak the weights and biases of each layer to minimize the cost function. The smaller the value of the MSE, the closer our predictions are to the actual results.
Here is where the partial derivatives come in. Let’s look at it this way: Each weight and bias is a variable that acts on the cost function in some way. Any change to any weight of any layer in our network will impact our cost in some way, large or small.
If we can find the partial derivative of each weight, that is, how much a tiny change to that single weight impacts the result of the cost function, we can figure out how to change that weight to minimize our cost function for the training example at hand.
For example: If, when we calculate the partial derivative of a single weight, we see that a tiny increase in that weight will increase the cost function, we know we must decrease this weight to minimize the cost. If, on the other hand, a tiny increase of the weight decreases the cost function, we’ll know to increase this weight in order to lessen our cost.
Besides telling us rather we should increase or decrease each weight, the partial derivative will also indicate how much the weight should change. If, by applying a tiny nudge to the value of the weight, we see a significant change to our cost function, we know this is an important weight, and it’s value influences heavily our network’s cost. Therefore, we must change it significantly in order to minimize our MSE. The reciprocal is also valid.
For each training example in our training set, we’ll find the cost for the example at hand and tweak all the weights accordingly. When we go over this process thousands (if not millions) of times — once for each training example — we’ll end up with an optimized network that can make excellent predictions for any given input.
The process for finding the optimal bias for each neuron is very similar, but since our simple network will have all the biases set to zero, we won’t dive deep into the bias calculation.
Let’s get coding!
Let’s start by adding a method to our Network class that calculates the cost of a single training example.
The cost method will receive an input, output pair of our training set.
It will begin by finding the networks prediction from the given input through the feedForward method. The activations of the output layer will be stored in the activations array. We’ll now create a new array with each element being the squared error of the output neuron.
We’ll return the sum of all of the squared errors in the array.
Now we’ve gotten to the most complex part of our network. The tweaking of the weights in order to find an optimal network. The process of calculating these new weights is called back-propagation.
There are complex algorithms that figure out all the partial derivatives of the weights in our network with respect to the cost value, speeding up the training process, but for the sake of simplicity, we’ll calculate these derivatives numerically and not analytically (for now).
What this means is that we’ll choose a very small number, say 0.01, and change a given weight by this value. Then we’ll recalculate the cost function for this tweaked network, to see if we made the prediction better or worse.
The difference between the original cost and the cost after we’ve changed the weight by 0.01, divided by 0.01, is a numerical approximation of the partial derivative of this weight in relation to the cost function.
Let’s add a new method to our Network class called backProp.
For each weight of our network, we’ll calculate it’s partial derivative with respect to the cost and store the value in an array. The structure of this array will be the same structure as the array with all of our network’s weights, but instead of having each weight value, it will have each partial derivative value.
To finish off, after we’ve calculated all of our partial derivatives, we’ll iterate through our weights array and subtract from each weight its corresponding partial derivative with respect to the cost function.
By doing this, we will magnify the value of the weights that diminish the cost, and diminish the value of the weights that maximize the cost.
The backProp method will look like this:
Phew... That was a lot of stuff! But now we have the fundamental parts of our network up and running!
The last thing we have to do is create a train method to receive a bunch of input/output pairs of training data and run the back-propagation algorithm for each entry.
After training, we’ll have a network with optimized weights, ready to predict the output for any new input.
The train method will also receive an epochs value.
The epochs value represents the number of times the network should be trained on the same training data. This may be interesting when we don’t have a very large training set, or other reasons that are very well explained in this article.
The full code for our network now looks something like this:
Notice we’ve also added a predict method which just implements the feedforward method and returns the result.
Now let’s see what we can do with our network!
To prove our networks functionality, let’s train a network to classify points on a coordinate plane.
In this example, let’s draw a diagonal line transversing the plane. This line would represent the function y = x.
We’ll try to train our network to classify points as being above the line or below the line.
To do this, we’ll feed into the network 1,000 examples of points, with the correct labels for them: 0 if the point is below the line and 1 if the point is above the line.
For now, we’ll only consider points with x and y values between 0 and 1.
I’ve created a simple function to automate the process of creating the training data. The function receives the number of input/output pairs to generate and then returns the input array and the corresponding output array.
Now we can train our network like so:
In the code above, first we initialize a new network with two neurons in the input layer, a hidden layer with 5 neurons and an output layer with 1 neuron.
In the second line, we create the x and y training data.
Next, we train our network for just one epoch, with the labeled data we generated.
Lastly, to see if we’ve been able to train a precise network, we’ll check out the networks prediction on a new value, in this case (0.5; 0.1).
We expect the network to classify the output as being 0, since this point on the coordinate plane is under the line we’re evaluating.
The output for the code above is
node index.js//returns [ 0.000019314711178699818 ]
Now let’s try to predict the output for the point (0.3, 0.7). The expected value should be close to 1.
When we run the predict method, the returned value is:
[ 0.9999203743668156 ]
This is exactly the output we expected. We’ve now created a network that can classify points on the coordinate grid after being trained with 10,000 examples.
We’ve already accomplished an intriguing task: understanding the nuts and bolts of a neural network and creating one ourselves, but there is still a lot of room for improvement.
The first improvement we could make to our network is to implement the back-propagation algorithm analytically instead of numerically.
The video above is really insightful for that matter, and I highly recommend you to watch it!
Implementing back-propagation through this algorithm can exponentially boost our training times, and allow us to train more complex nets in waaaaay less time.
Another adjustment we can make to our network is adding bias values to each neuron. In the references section you can find articles that explain and implement biases on other neural nets.
Lastly, another modification we could make to our network is to train it through batches. What this means is that instead of calculating the cost and tweaking the weights for every single training example, we could calculate the mean cost for, say, 100 training examples and tweak the weights based on this value. This technique can greatly speed up our training times!
There are definitely many other things we could tweak and tinker in our network to boost performance, but I hope that this simple example may have helped you to understand the key concepts behind the algorithm!
I hope you may have found this article inspiring, and that it may have helped you in any way! 😁😁😁
neuralnetworksanddeeplearning.com
towardsdatascience.com
playground.tensorflow.org
www.kdnuggets.com
Code snippets made with https://carbon.now.sh/