Postmortem of Artificial Neural Networks using PyTorch | by Asad Mahmood | Towards Data Science

My first interaction with Machine Learning was when I needed to complete an assignment, and I just searched the internet for ways of using Machine learning. I ended up using scikit-learn API for ANNs. I completed the assignment but had literally no idea about what was happening on the backend. So, then I started taking MOOCs and started exploring the mathematics behind this “Black Box”.
After all this exploration, when I came back to use ANNs (this time with Keras), I found it very unsettling that a line as simple as model.fit is doing everything itself on the backend. This is when I decided to shift to a framework that gives the programmer more power and whose code is more readable.
In this blog, I'll use PyTorch to explain Artifical Neural Networks (ANNs) both with code and mathematics. So without any further delay, let’s get cracking.
It is a deep learning framework gaining tremendous popularity among the researchers, mainly because of its very pythonic and understandable syntax.
As mentioned by the PyTorch tutorials [1], a typical model training follows the following six steps.
Defining Model (with all the learnable parameters)Iterating over a Dataset of inputsForward PassComputing LossBackPropagating GradientsUpdate weights.
Defining Model (with all the learnable parameters)
Iterating over a Dataset of inputs
Forward Pass
Computing Loss
BackPropagating Gradients
Update weights.
Let’s go over the code (in PyTorch) and mathematics behind these steps.
In a traditional ANN, there should be an input layer and an output layer along with optional hidden layers. The size of the input layer is the same as the size of each input in the dataset. The size of the output layer is the same as the number of classes and for hidden layers, the size could vary from problem to problem.
The model that we are going to use contains an input layer of size 2, one hidden layer of size 3 and one output layer of size 1. Following is the code for this in PyTorch. Note that the code also contains the function forward. We’ll look at this later in Forward Pass section.
import torch# defining model architectureclass ANN(torch.nn.Module):    def __init__(self):    super(NueralNetwork, self).__init__()    self.fc1 = torch.nn.Linear(2,3)    self.fc2 = torch.nn.Linear(3,1)def forward(self, x):    x = torch.sigmoid(self.fc1(x))    x = torch.sigmoid(self.fc2(x))    return x
This model here has 13 learnable parameters including 9 weight parameters(6 for the hidden layer and 3 for the output layer) and 4 bias parameters(3 for the hidden layer and 1 for the output layer). Following is the diagram representation of this model.
The second step involves preparing a dataset. In this blog, our dataset is the XOR gate which contains 4 rows each with 2 inputs and one output. We define this using the following piece of code.
# setting up dataX = torch.tensor(([0, 0],[0, 1],[1, 0], [1, 1]), dtype=torch.float)y = torch.tensor(([0], [1], [1], [0]), dtype=torch.float)
From here onwards, we stick to the first row in our dataset i.e., X[0] = [0,0] and y[0] = [0]. Mathematically forward pass is calculated using the following formula: y^ = w*input + b where ‘w’ and ‘b’ represent the weights and biases respectively.
We use the following piece of code to look at our initial weights and biases.
nueralNet = ANN()print(nueralNet)params = list(nueralNet.parameters())# print(params)# print(nueralNet.fc2.weight)print("Weight vector for layer 1")print(params[0])print("Bias vector for layer 1")print(params[1])print("Weight vector for layer 2")print(params[2])print("Bias vector for layer 2")print(params[3])print('==========================')out = nueralNet(X[0])
This prints out the following
NueralNetwork(  (fc1): Linear(in_features=2, out_features=3, bias=True)  (fc2): Linear(in_features=3, out_features=1, bias=True))Weight vector for layer 1Parameter containing:tensor([[-0.3122,  0.5551],        [-0.2819, -0.0666],        [ 0.6712, -0.2915]], requires_grad=True)Bias vector for layer 1Parameter containing:tensor([ 0.2816, -0.4757, -0.4125], requires_grad=True)Weight vector for layer 2Parameter containing:tensor([[-0.5761, -0.2721,  0.4413]], requires_grad=True)Bias vector for layer 2Parameter containing:tensor([-0.1019], requires_grad=True)
The line of code where we pass the first instance of our dataset to the model does the calculations for the forward pass. This runs the forward function defined earlier in the ANN class. Sigmoid in the forward pass is an activation function given by the following formula:
Following are the mathematical calculations behind the forward pass here.
This results in y^ = 0.4112
After the forward pass, we evaluate output created by our model against the original label. This is done by computing loss which essentially measures the distance between the label and the current output. In our implementation, we’ll use the popular mean squared error.
import torch.optim as optim# create your optimizeroptimizer = optim.SGD(nueralNet.parameters(), lr=0.01)criterion = torch.nn.MSELoss()# in your training loop:optimizer.zero_grad()   # zero the gradient buffersloss = criterion(out, y[0])print('Mean squared error or loss = ', loss)
Mathematically, it is given by
So, in our case, since we have only one input instance, the loss would be = 0.1691.
After calculating the loss, we need a way to minimize it in order to improve our method. For this, we first calculate the gradients and then propagate them back to update the learnable parameters. This is done by the following code in PyTorch
loss.backward()optimizer.step()    # Does the weight updateprint("Weight vector for layer 1")print(params[0])print("Bias vector for layer 1")print(params[1])print("Weight vector for layer 2")print(params[2])print("Bias vector for layer 2")print(params[3])print('==========================')
I encourage you to follow this link (https://www.youtube.com/watch?v=Ilg3gGewQ5U) to understand deeply the workings of backpropagation and weights update. As a fun exercise, try to calculate the gradients and parameters after update in this example.