Neural Networks Part 3: Understanding Back propagation & Learning Rates
By: Shweta Kadam
Tags: Backpropagation, Deep Learning, Neural Networks, TensorFlow, Gradient Descent

In this article , we will learn about how we can calculate the gradient and how the network learns through back propagation. Following from the previous article regarding our discussion of neural networks and gradient descent.We talked about gradient and how it works for reducing the loss till we converge to a global minimum.

Let’s have a look at a network having a single hidden neuron and a single output.

How does a small change in (ex.:w2) affect the final loss J(W)?

Computing our gradient of our loss with respect to weight W2(which is the second weight between the output and the hidden layer). Our output can tell us how much a small change in W2 will impact our loss.If we change W2 in a small way, how much does our loss change ? Does it go upon down?How much does it change and by how much?Lets look at the derivative of this loss with respect to W2 and to evaluate this we can apply the chain rule in calculus.

By applying chain rule

We can split this up into gradient of our loss with respect to our output y_hat multiplied by gradient of our output with respect to our weight (W2).Say now you want to repeat this process for a neural network say W1 and not W2 so we replace W1 on both sides & we also apply the chain rule.

But now you are going to notice that the gradient of y with respect to W1 is not directly computable.We have to apply the chain rule again to evaluate this.So after applying chain rule again we split this second term up gradient of y with respect to state z1 and gradient of state z1 with respect to W1.

Backpropgate all these gradients from output to input

So we kind off back propagate all of these gradients from the output all the way to the input that allows our error signal to propagate from output to input and allow these gradients to be computed.Now in practice ,it is not as crucial to understand the knitty –gritty math here as in a lot pf popular deep learning frameworks we have what’s called automatic differentiation which does all this backpropagation under the hood and we never even see it…which is incredible you don’t to have to implement backpropagation anymore from scratch but its still important to understand how these work at the ground level hence the purpose of this article.

Now you repeat the above process for every single layer in the network.Here we just showed for W1 and W2 which is a single layer but when you have more you can just repeat it again.Keep applying the chain rule from output and input and compute.And this process is backpropagation .This is how the network learns from this mistakes or loss and tries to correct those mistakes which aims in reducing the loss.

In practice, optimization of neural network is extremely tough,it is not as simple as the 1st diagram which is shown.If we try to visualize a landscape, a neural network with various parameters and try to collapse that down into a 2dimensionsal space so you can visualize it you can see that in reality the landscape is complex ,there can be many local minima where the gradient descent algorithm can get stuck into and applying gradient descent in these types of environments which is very standard in neural networks can be quite challenging.

Gradient descent algorithm can get stuck into local minima

Update Equation(3rd point) from Gradient descent algorithm

Now recall the update equation that we previously defined in gradient descent algo.That is the same equation we are going to update our weights in the opposite direction of our gradient.If we look at this η (eta) which is the learning rate which determines how much of a step we should take in the direction of that gradient and in practice setting this learning rate can have a huge impact on your performance.

If you set that learning rate too small ,that means you are not really trusting your gradient In each step so if your learning rate is too small you are only going to move a little bit towards in the opposite direction of your gradient or just in small increments.And what can happen is then you can get stuck in these local minima because you are not being as aggressive you should be to escape these local minima.

a.)Small learning rate converges slowly and get stuck in the false local minima. whereas b.)Large Learning rate overshoot ,become unstable and diverge.

If you set the learning rate too large,you can actually overshoot completely and diverge which is more undesirable.So setting the learning rate in practice can be quite challenging.

Stable learning rate converge smoothly and avoid local minima.

So the question you must be asking is how do we set the learning rate just right? Option 1 is we try a bunch of learning rates and see what works best.Second option is we can try to be a bit more clever and try to have an adaptive learning rate which changes with to our landscape or changes with respect to how fast the learning is happening or a range of other idea within our network optimization.

This means that our learning rate is no longer fixed but it can now increase or decrease throughout the training .So as your training progresses, your learning rate may speed up ,you can take more aggressive steps,you can take smaller steps and get closer to the local minima so you can really converge on that point.There are many options here as to how you want to design this adaptive algorithm and this has been a widely huge studied field in loss optimizations theory for ml & dl. And there have been different types of papers and implementations within tensorflow for these different types of adaptive learning rates.

So,SGD is the vanilla gradient descent which I showed before.All of the others are adaptive learning rates which means they change their learning rate during training itself.So they can increase or decrease depending on how optimization is going

In conclusion ,we can define our model in this sequential wrapper.Inside this sequential wrapper we have all of our layers which are composed of perceptrons or single neurons.Then we can use our optimizer stochastic gradient descent or it can also be any of those adaptive learning rates .

During our training loop,we forward pass all of our inputs thru that model ,we get our predictions .Using those predictions,we can evaluate them,compute our loss & our loss tells us how wrong our network was on that iteration and it also tells us how we can compute the gradients and how we can change all of the weights in our network to improve in the future .

Finally take those gradients and actually allows our optimizer to update the weights & the training variables such that on next iteration they do a little bit better.And overtime if we keep looping this will converge and hopefully should fit your data.

So this was all about back propagation and learning rates in neural networks .

Happy Learning.