We've been talking about the gradients. We've been assuming that the gradients are cool. You just might be going in the wrong direction. But what happens if they vanish flat? What happens if the gradients go to not a number? They explode. Fix, fix, fix. One of the problems is with vanishing gradients. If you know both the sigmoid and the tannage activation functions have zero derivatives, they have basically horizontal asymptotes. And in our output function, in our loss function, we have nested derivatives of the activation functions. If they are sigmoid and tannage, we could have a problem. We could also have a problem. We're going to look at that as well in choosing the weights too small. We'll see what happens with choosing the weights too small. And if the weights are near zero, what does that mean? It means we're not going anywhere. And we're going to stay on the same level of our funny mountainous loss surface because there's no motivation, nothing pushing us down towards the minimum. So we have to fix these. We'll look at initialization later. Now we're going to look at the first problem. And the first problem is fixed, assuming that you don't have initialization problems. And that's what we have as number one there. And now we're going to choose a new activation function called the rectified linear unit. So when z is equal to zero, it's less than zero. It's zero. And then when z is greater than zero, it's just the function itself. So there it is written. And the problem is that it has a jump. It's z is equal to zero because it takes off. The function is like that. So right at the knee, where the flat zero part reaches the z part for input z, you get z. So it's just at 45 degrees. And the f dz has a jump. Well, there are many ways to handle a jump. I still like Fourier's way. I don't think it will make a big difference how you handle the jump and z is equal to zero. Opposite problem, exploding gradient. And this is how we fix it. We use gradient clipping. So you have a check. If grad goes crazy and goes to not a number, you bring it back down to behave itself. So you set a gradient threshold, gamma, another hyperparameter. The norm of the gradient, usually it's L2, is too big. Then what you do is you divide by the magnitude and then replace, as you can see there, the gradient by gamma times the unit vector. Well, the question is, what happens if you don't compute things? We'll just put gradient of L over magnitude of L is a unit vector if it's L2, Euclidean length. So just multiply it by gamma and you know the direction. So that is an issue. You may not be able to do that division, in which case the gradient clipping will fail. But hopefully, it's not going to fail. And it will introduce a bias because you're going to push it off track, but it will keep things stable. Now, this often happens in recurrent networks and there are other techniques that we can use there.