Calculating Gradient Descent Manually
By: Chi-Feng Wang
Tags: Calculus, Math, Artificial Intelligence, Neural Networks, Gradient Descent

Gradient of A Neuron

We need to approach this problem step by step. Let’s first find the gradient of a single neuron with respect to the weights and biases.

The function of our neuron (complete with an activation) is:

Image 2: Our neuron function

Where it takes x as an input, multiplies it with weight w, and adds a bias b.

This function is really a composition of other functions. If we let f(x)=w∙x+b, and g(x)=max(0,x), then our function is neuron(x)=g(f(x)). We can use the vector chain rule to find the derivative of this composition of functions!

The derivative of our neuron is simply:

Image 3: Derivative of our neuron function by the vector chain rule

Where z=f(x)=w∙x+b.

There are two parts to this derivative: the partial of z with respect to w, and the partial of neuron(z) with respect to z.

What is the partial derivative of z with respect to w?

There are two parts to z: w∙x and +b. Let’s look at w∙x first.

w∙x, or the dot product, is really just a summation of the element-wise multiplication of every element in the vector. In other words:

Image 4: Expanded version of w∙x, or the dot product

This is once again a composition of functions, so we can write v = w⊗x and u=sum(v). We’re trying to find the derivative of u with respect to w. We’ve learned about both of these functions — element-wise multiplication and summation — before in Part 3. Their derivatives are:

Image 5: Derivative of u with respect to v and derivative of v with respect to w; where u=sum(w⊗x)

(Go back and review them if you don’t remember how they’re derived)

Therefore, by the vector chain rule:

Image 6: Derivative of u with respect to w

That’s it! Now, let’s find the derivative of z= u+b where u=w∙x with respect to both the weights w and the bias b. Remember that the derivative of a function with respect to a variable not in that function is zero, so:

Image 7: Derivative of z with respect to the weights and biases, where z=sum(w⊗x)+b

That’s it! Those two are the derivatives of u with respect to both the weights and biases.

What is the partial derivative of neuron(z) with respect to z?

Neuron(z)=max(0,z)=max(0, sum(w⊗x)+b).

The max(0,z) function simply treats all negative values as 0. The graph would thus look something like this:

Image 8: Max(0,z) // Source

Looking at that graph, we can immediately see that the derivative is a piecewise function: it’s 0 for all values of z less than or equal to 0, and 1 for all values of z greater than 0, or:

Image 9: Derivative of max(0,z)

Now that we have both parts, we can multiply them together to get the derivative of our neuron:

Image 10: Derivative with respect to the weights of our neuron: max(0, sum(w⊗x)+b)

And substitute z=w∙x+b back in:

Image 11: Substituting w∙x+b for z

Voila! We have our derivative for a neuron with respect to its weights! Similarly, we can use the same steps for the bias:

Image 12: Derivative of our neuron with respect to bias

And there you go! We now have the gradient of a neuron in our neural network!

Gradient of Loss Function

Our loss function, defined in Part 1, is:

Image 13: Loss Function

We can immediately identify this as a composition of functions, which require the chain rule. We’ll define our intermediate variables as:

Image 14: Intermediate variables for loss function

*Note, the u and v here are different from the u and v used in the previous section.

Let’s compute the gradient with respect to the weights w first.

Gradient With Respect to Weights

u is simply our neuron function, which we solved earlier. Therefore:

Image 15: Derivative of u=max(0, sum(w⊗x)+b) with respect to the weights

v(y,u) is simply y-u. Therefore, we can find its derivative (with respect to w) using the distributive property and substituting in the derivative of u:

Image 16: Derivative of v=y-u with respect to the weights

Finally, we need to find the derivative of the whole cost function with respect to w. Using the chain rule, we know that:

Image 17: Derivative of cost function

Let’s find the first part of that equation, the partial of C(v) with respect to v first:

Image 18: Derivative of cost function with respect to v

From above (Image 16), we know the derivative of v with respect to w. To find the partial of C(v), we multiply the two derivatives together:

Image 19: Derivative of cost function with respect to w

Now, substitute y-u for v, and max(0, w∙x +b) for u:

Image 20: Derivative of cost function with respect to w

Since the max function is on the second line of our piecewise function, where w∙x+b is greater than 0, the max function will always simply output the value of w∙x+b:

Image 21: Derivative of cost function with respect to w

Finally, we can move the summation inside our piecewise function and tidy it up a little:

Image 22: Derivative of cost function with respect to w

That’s it! We have our derivative with respect to the weights! However, what does this mean?

w∙x+b-y can be interpreted as an error term — the different between the predicted output of the neural network and the actual output. If we call this error term ei, our final derivative is:

Image 23: Derivative of cost function with respect to w represented with an error term

Here, the greater the error, the higher the derivative. In other words, the derivative represents the slope, or how much we have to move our weights by in order to minimize our error. If our neural network has just begun training, and has a very low accuracy, the error will be high and thus the derivative will be large as well. Therefore, we will have to take a big step in order to minimize our error.

You might notice that this gradient is pointing in the direction of higher cost, meaning we cannot add the gradient to our current weights — that will only increase the error and take us a step away from the local minimum. Therefore, we must subtract our current weights with the derivative in order to get one step closer to minimizing our loss function:

Image 24: Gradient descent function

Here, η represents the learning rate, which we as programmers can set. The larger the learning rate, the bigger the step. However, setting a too-large learning rate may result in taking too big a step and spiraling out of the local minimum For more information, check out this article on gradient descent and this article on setting learning rates.

Gradient With Respect to Bias

Once again, we have our intermediate variables:

Image 25: Intermediate variables for loss function

We also have the value of the derivative of u with respect to the bias that we calculated previously:

Image 26: Derivative of u with respect to the bias.

Similarly, we can find the derivative of v with respect to b using the distributive property and substituting in the derivative of u:

Image 27: Derivative of v with respect to the bias

Again, we can use the vector chain rule to find the derivative of C:

Image 28: Derivative of cost function with respect to the bias

The derivative of C with respect to v is identical to the one we calculated for the weights:

Image 29: Derivative of cost function with respect to v

Multiplying the two together to find the derivative of C with respect to b, and substituting in y-u for v, and max(0, w∙x +b) for u, we get:

Image 30: Derivative of cost function with respect to the bias

Once again, because the second line explicitly states that w∙x+b>0, the max function will always simply be the value of w∙x+b.

Image 31: Derivative of cost function with respect to the bias

Just like before, we can substitute in an error term, e = w∙x+b-y:

image 32: Derivative of cost function with respect to the bias, represented with an error term

Just like the derivative with respect to the weights, the magnitude of this gradient is also proportional to the error: the bigger the error, the larger step towards the local minimum we have to take. It is also pointing towards the direction of higher cost, meaning that we have to subtract the gradient from our current value to get one step closer to the local minimum: