Less than a simple introduction to Neural Networks (3/4)
By: Ahmed Selim
Tags: Loss Function, Backpropagation, Optimization, Gradient Descent, Machine Learning

Part 1: Introduction

Part 2: Weights and parameters

Part 3: Optimization and Backpropagation

Part 4: Hidden Layers and Nonlinearities

The image was taken from this post.

Optimization and Loss

I decided to explain optimization and loss first, before the hidden layers and non-linearities. If you came here before reading the first two parts, then don’t worry. If you are following the post and want the latter, I will try to write it up before next week. If you can leave me feedback in the comments that will be great. If you like the way I explain but interested in a different topic, leave me a comment and I will do my best to cover it.

I will follow the same simplicity for optimization and loss. It was one of the hardest things I had to understand since this is where the complicated calculus and what not takes place. Yet if you are just starting, all the new API’s or libraries like Tensorflow, Keras, Pytorch, etc.. made life so much easier. One line of code and you have the optimization set up.

In part 1, I have explained that you can think of the optimization method as trying combinations of values to learn the weights/parameters (part 2) of the function. However, that’s not the case as you may have guessed.

First of all, the weights are optimized according to how well it is performing. So how do we measure that? In the previous example, we were solving the equation y = wx and we know w=2 (our weight that we are trying to learn), also in my random initialisation of w before the training w = 0.5346. We provided the network with the training examples including the labels or the output that we already know beforehand. For example, we know that if x = 2 then y = 4. The first Iteration/epoch, the network performs an operation called feedforward and it just means calculate the function with the values we have. Let’s take a look at what it might look like in the first iteration:

Iteration 1:

x = 2.0 # One example of input

w = 0.5346 # The weight the network is learning

y = 4 # The real output of the function that we already given. Feedforward:

ŷ(predicted/calculated)= wx = 2 * 0.5346 = 1.0692 #FeedForward

Obviously the answer is not correct, notice that this y is predicted by the network not our labels and it is usually referred to as ŷ (y hat). This is what a feedforward is. Naturally, two questions come up: How does the network know that this is wrong? and How do we make it right?

The first question is easy, you calculate the error as the difference between the expected output and if they are not equal to zero, then it’s wrong. But of course, in real life, there are a number of ways to calculate the error which is out of the scope for this tutorial. For this level, we can think of the loss function as the function that calculated this error: error = y-ŷ.

So far we have:

Feedforward to calculate the ŷ. Loss function to calculate the error.

The second question is a little bit more complicated. This is where the scary backpropagation comes in. I will have to mention the role of derivatives briefly. That is how I think about differentiation, it is measuring the rate of change between let’s say two variables, answering the question of how much did x change when we changed y. To put it into our context, how much did the error change when we changed w? This is what it means by calculating the derivative of the loss function with respect to w (weights). In a larger network, as opposed to the one we used so far, the network will have many weights and layers and all of them contribute to the error. In other words, we have many weights and many layers and one error to be calculated. When the word backpropagation is used, in a nutshell, it means it is integrating the rate of change of error with the rest of the network BUT it starts from the end to the front (bottom to the top depends on how you want to visualise it). If the feedforward is a sequence that goes c = a +b then e = c + d, then backpropagation would go the opposite direction and calculate e = c+d then c = a + b. Of course, it is not doing summation, it is doing partial derivatives and starting with the end of the network to the beginning.

Finally, the magic happens when the weights are optimised by gradient descent. Fundamentally, it takes that rate that was calculated (the gradient/derivative and update (add/subtract ) the weights. That is how the weights are changed. Importantly, it keeps doing that over and over to find what is referred to as the minima of a function which in turn means, finding the values of the weights that have the lowest error. Another important parameter here is called the learning rate. It is a very important number that is multiplied by the calculated rate for updating the weights that tell the network to change the weights fast or slow.

To sum up, the main operations of a neural network are the following:

Feedforward to calculate the ŷ. Loss function to calculate the error. Backpropagation to calculate the rate of change of the loss when changing the values of the weights. Gradient descent to update and optimize the weights.

There are many great tutorials that go in details and explain the mathematics behind it. If you are not interested, at least get familiar with the idea of what it does because that is the core of neural networks. If you can leave some feedback if it needs more simplification or you got the idea. I will edit the post.

The next part, will do some coding and implement a two layers network!