The Ultimate Beginner’s Guide To Implement A Neural Network From Scratch | by Ravsehaj Singh Puri | Towards Data Science

Neural networks have been with us for a long time but they have gathered importance in recent years due to advancement in computing machinery as well as growing needs of technology in various aspects of life.
From the perspective of a beginner in machine learning, a neural network is considered as a black box which can ingest some data and spit out predictions or classification categories depending upon the problem at hand. The beauty of neural networks is that they are based on simple calculus and linear algebra or a combination of both. These work together to come up with close to accurate results when provided with high quantity and quality data.
So, if you are motivated to learn the basics of how a neural network actually works, you have reached the right place. Follow me along this post and you’ll be able to build your own neural network from scratch.
A neural network’s architecture is derived from the structure of a human brain while from a mathematical point of view, it can be understood as a function which maps a set of inputs to desired outputs. The main idea of this post is to understand this function in detail and implementing it in python.
A neural network comprises of 7 Parts :
Input Layer (X) : This layer contains the values corresponding to the features in our dataset.A set of weights and biases (W1,W2,..etc); (b1,b2,..etc) : These weights and biases are represented in the form matrices and they decide the importance of each feature/column in our dataset.Hidden Layer : This layer acts as a brain of the neural network and also as an interface between the input and the output layer. There can be one or more than one hidden layers in a neural network.Output Layer (ŷ) : The values transmitted from the input layer will reach this layer via the hidden layer(s).A set of activation functions (A) : This is the component which adds a non-linearity flavour to an otherwise linear model. These functions are applied to the output of each layer except the input layer and activate/transform them.The Loss function (L) : This function calculates the measure as to how well our guess/prediction is, and it is used while back propagating through the network.Optimiser : This optimiser is a function that updates the model parameters according to gradients which are calculated during back propagation (You’ll learn shortly).
Input Layer (X) : This layer contains the values corresponding to the features in our dataset.
A set of weights and biases (W1,W2,..etc); (b1,b2,..etc) : These weights and biases are represented in the form matrices and they decide the importance of each feature/column in our dataset.
Hidden Layer : This layer acts as a brain of the neural network and also as an interface between the input and the output layer. There can be one or more than one hidden layers in a neural network.
Output Layer (ŷ) : The values transmitted from the input layer will reach this layer via the hidden layer(s).
A set of activation functions (A) : This is the component which adds a non-linearity flavour to an otherwise linear model. These functions are applied to the output of each layer except the input layer and activate/transform them.
The Loss function (L) : This function calculates the measure as to how well our guess/prediction is, and it is used while back propagating through the network.
Optimiser : This optimiser is a function that updates the model parameters according to gradients which are calculated during back propagation (You’ll learn shortly).
Neural networks learn/train from the training data and then their performance is tested using test data. There are 2 parts of the training process:
Feed forwardBack-propagation
Feed forward
Back-propagation
Feed forward is basically traversing the neural network from input layer to the output layer by predicting a value. On the other hand, back-propagation makes the network actually learn by computing gradients and pushing them back through the network and finally updating the model parameters. Let us first see how these 2 processes work on paper and then we will convert our equations into code.
Let us first design our neural network architecture. The architecture which we will be using for this post is shown below:
As far as our architecture is concerned, we just need 2 equations to carry out the feed forward process and for future purposes, you may remember:
number of layers of the neural network = number of hidden layers + 1(output layer)
number of weight matrices = number of layers
We will be using ReLu and Sigmoid activation functions in this post but other activation functions can alse be applied.
Here, X, W1, b1, W2 and b2 are matrices and matrix level computations are done in both the equations above. For better understanding, we can have a look at these matrices representing X, W1, b1, W2 and b2. In this post, we are ignoring the bias terms b1 and b2 but they can be dealt with in a similar manner.
Let us define a few values first :
n1= number of neurons in the input layer, n2= number of neurons in the hidden layer, n3= number of neurons in the output layer.
Now, you might be thinking of how to decide the number of neurons for Input layer, hidden layer and output layer. Here is the answer.
number of neurons in the input layer (n1)= number of features/columns in our dataset i.e. the number of independent variables.
number of neurons in the hidden layer (n2) is flexible.
number of neurons in the output layer (n3) =
Dimensions of X: [m, n1] ; Dimensions of W1: [n1, n2] ;
Dimensions of W2: [n2, n3]
Back propagation is an essential part in training a neural network. It is a method of learning on the basis of gradients which are calculated using the most beautiful part of calculus i.e. the Chain Rule. This chain rule is repetitively applied to calculate gradients across the network. To make it more clear, a gradient is the derivative of loss function w.r.t. to a model parameter (weight or bias).
As mentioned earlier, we won’t be using bias terms and will only consider weights as model parameters. So, we have 2 model parameters i.e. W1 and W2. The loss function which we will be using is MSE (Mean Square Error). Now let us see how back propagation actually works mathematically.
where m= number of training examples/rows in the dataset
y= ground truth/label in the dataset
Let us calculate the gradient of loss w.r.t. W2 using chain rule :
The derivation of ‘Sigmoid_derivative’ term is not in context of this post but it has been beautifully explained here.
Therefore, our first gradient term simply turns out to be :
Now, moving further, we calculate our second gradient term i.e. the derivative of loss w.r.t. W1 in a similar way :
You can also refer to this post to know more about these activation functions and their derivatives.
towardsdatascience.com
So, now that we have calculated the gradients w.r.t. W1 and W2, it is time to perform the optimisation steps. We will be using Gradient Descent Optimizer which is also called Vanilla Gradient Descent to update W1 and W2. You might be knowing that gradient descent algorithm has a hyper-parameter named as learning rate, which has a great importance in the training process. Tuning of this hyper-parameter is very essential and is done by using validation testing which is not in scope of this post.
Now that we have learnt the mathematical concepts behind a neural network, let us convert these concepts into code and train a neural network on a real-world machine learning problem. We will be using the famous IRIS dataset to train our network and then predict flower categories. You can download the dataset from here.
import numpy as npimport pandas as pdfrom sklearn import preprocessingfrom sklearn.preprocessing import StandardScalerfrom sklearn.preprocessing import LabelEncoderfrom sklearn.model_selection import train_test_split
data=pd.read_csv('IRIS.csv')X=data.iloc[:,:-1].valuesY=data.iloc[:,-1].valuesStandardScaler=StandardScaler()X=StandardScaler.fit_transform(X)label_encoder=LabelEncoder()Y=label_encoder.fit_transform(Y)Y=Y.reshape(-1,1)enc=preprocessing.OneHotEncoder()enc.fit(Y)onehotlabels=enc.transform(Y).toarray()Y=onehotlabels  
After doing aforesaid basic pre-processing on the input data, we move on to create our neural network class which contains the feed forward and back-propagation functions. But first, let us split our data into train and test.
X_train,X_test,Y_train,Y_test=train_test_split(X,Y,test_size=0.2,random_state=0)
I am also writing code to define functions which can calculate function values and function derivative values. I have written code for 3 activation functions and used only two (Relu and Sigmoid) out of them. However, you can try out various combinations of these activation functions and also incorporate other activation functions in your code.
def ReLU(x):   return (abs(x.astype(float))+x.astype(float))/2def ReLU_derivative(x):   y=x   np.piecewise(y,[ReLU(y)==0,ReLU(y)==y],[0,1])   return ydef tanh(x):   return np.tanh(x.astype(float))def tanh_derivative(x):   return 1-np.square(tanh(x))def sigmoid(x):   return 1/(1+np.exp(-x.astype(float)))def sigmoid_derivative(x):   return x*(1-x)
Defining the neural network class :
class Neural_Network:   def __init__(self,x,y,h):      self.input=x      self.weights1=np.random.randn(self.input.shape[1],h)      self.weights2=np.random.randn(h,3)      self.y=y      self.output=np.zeros(y.shape)
The class ‘Neural_Network’ accepts 3 parameters: x, y and h which are corresponding to X_train, Y_train and number of neurons in the hidden layer. As mentioned earlier, number of neurons in the hidden layer is flexible and therefore we will pass its value along with X_train and Y_train while creating an instance of our class.
‘weights1’ and ‘weights2’ matrices are initialized from a random distribution and by passing the dimensions required for the respective weight matrices. As previously W1 and W2 matrices have been shown along with their dimensions, we pass these dimensions to this random function to generate our initial weights.
   def FeedForward(self):      self.layer1=ReLU(np.dot(self.input,self.weights1))      self.output=sigmoid(np.dot(self.layer1,self.weights2))
Our first function in the class is ‘FeedForward’ which is the first step in the training process of a neural network. The code follows ‘Equation-1’ and ‘Equation-2’ described earlier.
Next, we move on to the most essential part, the back-propagation.
def BackPropogation(self):      m=len(self.input)      d_weights2=-(1/m)*np.dot(self.layer1.T,(self.y-self.output)       *sigmoid_derivative(self.output))d_weights1=-(1/m)*np.dot(self.input.T,(np.dot((self.y-    self.output)*sigmoid_derivative(self.output),self.weights2.T)*    ReLU_derivative(self.layer1)))self.weights2=self.weights2 - lr*d_weights2      self.weights1=self.weights1 - lr*d_weights1
The back-propagation step calculates gradients w.r.t. the model parameters while traversing back in the network from output layer to the input layer. This is governed by ‘Equation-3’ and ‘Equation-4’ which we obtained using chain rule. You will observe that both the terms ‘d_weights2’ and ‘d_weights1’ consist of 2 types of operations i.e. dot product and normal multiplication. In NumPy, np.dot(X,Y) represents matrix multiplication between X and Y whereas np.multiply(X,Y) or simply X*Y represents element wise multiplication between X and Y. To get more insights on how this piece of code is working, I will give you a small task which i myself did when i first implemented this from scratch.
If you look at the hidden layer, for each neuron (blue), there are 3 connections entering the neuron. Each such connection is emerging from a distinct neuron in the input layer. The basics of neural network suggest that there is a linear relation between the blue neuron of hidden layer and the set of neurons from which it is receiving information through the connections. This relation can be represented in the form of an equation. mentioned below :
Now, extending this equation to each and every neuron(blue) of the hidden layer, we obtain 4 such equations as we have 4 neurons in the hidden layer. The work that has been done till now, only represents 1 training example/row. In order to send our training data all at once into the feed forward function, we will have to use matrices to perform this operation. But the task really is to verify with full confidence that solving(on paper) the above piece of code which includes dot products and element-wise multiplications matches these 4 linear equations we discussed. But this time we will be applying these equations on all the rows of our training data simultaneously.
I hope you enjoyed getting your hands dirty in solving matrices and equations but trust me it is worth spending time on.
Moving further, we define a function which will predict the output class at run time after our model is trained.
def predict(self,X):   self.layert_1=ReLU(np.dot(X,self.weights1))   return sigmoid(np.dot(self.layert_1,self.weights2))
This is the last function in the ‘Neural_Network’ class and the ‘predict’ function does nothing but normal feed forward using the final optimized weights.
Now that we have defined our class, it is time to actually use it.
#----Defining parameters and instantiating Neural_Network class----#epochs=10000 #Number of training iterationslr=0.5 # learning rate used in Gradient Descentn=len(X_test)m=len(X)nn1=Neural_Network(X_train,Y_train)#creating an object of Neural_Network class#----------------------Training Starts-----------------------------#for i in range(epochs):   nn1.FeedForward()   y_predict_train=enc.inverse_transform(nn1.output.round())   y_predict_test=enc.inverse_transform(nn1.predict(X_test).round())   y_train=enc.inverse_transform(Y_train)   y_test=enc.inverse_transform(Y_test)   train_accuracy=(m-np.count_nonzero(y_train-y_predict_train))/m   test_accuracy=(n-np.count_nonzero(y_test-y_predict_test))/n   nn1.BackPropogation()   cost=(1/m)*np.sum(np.square(nn1.y-nn1.output))   print("Epoch {}/{}  ==============================================================:- ".format(i+1,epochs))#----------------Displaying Final Metrics--------------------------#print("MSE_Cost: {} , Train_Accuracy: {} , Test_Accuracy: {} ".format(cost,train_accuracy,test_accuracy))
After training for 10000 epochs, the following is the result:
Train accuracy : 0.97777
Test accuracy : 0.9
The results look amazing as compared to other machine learning models when applied to the same problem at hand. This network can be extended to a 3- layer network which will have 2 hidden layers using same concepts. You can also try different optimizers like AdaGrad, Adam etc.
You can access my additional work on the ‘Adult Income Prediction’ dataset which uses a similar neural network in my GitHub repository, the link to which is provided below:
github.com
In this post, we started learning the basic architecture of a neural network and went further into the components of a neural network architecture. We also witnessed the mathematics behind the working of the same and got a detailed explanation of application of chain rule and basic calculus in developing equations for back-propagation which is a major part of the training process. Finally, we converted our mathematical concepts and equations into code and implemented a full fledged neural network on the IRIS dataset.
Thanks a lot for reading through the article. I hope you understood each and every aspect of it. Feel free to ask any question in the responses section.