Deeplearning Backprop chain rule | Towards Data Science

Have you ever used a neural network an wondered how the math behind it works? In this blogpost, we will derive forward- and back-propagation from scratch, write a neural network python code from it and learn some concepts of linear algebra and multivariate calculus along the way.
I will start off by explaining some linear algebra fundamentals. If you are proficient enough in this, you can skip the next part.
Simple numbers (scalars) are written in small letters.
Vectors are denoted by bold letters and they are column vectors by default.
Names of row vectors are also written in bold but with a capital T for transposed.
Matrices are denoted in bold capital letters.
When transposing a matrix, we switch rows and columns.
The dimension of a Matrix or vector is:
( number of rows, number of columns)
When multiplying two vectors or matrices with the dot product, the number of columns of the left must match the number of rows of the right vector or matrix in the multiplcation.
The dimension of the result can be calculated as follows:
The “inner” dimension in this multiplication must match and disappear in the output dimension.
When multiplying two vectors or matrices using the elementwise product, they have to have the same dimensions.
We also need calculus with vectors and matrices. We use the concept of a Jacobian. A Jacobian is simply a vector or matrix of derivatives.
The derivative of a scalar valued fuction with respect to a vector x will be defined as follows:
We define the derivative of a function with an ouput vector f with respect to a single variable x as follows:
The derivative of a function with an output vector and a vector of input variables is defined the following way:
The vector chain rule looks very similar to the scalar chain rule. In this exmaple,vector f is a function of vector g and vector g itself is a function of vector x. So the derivative of f(g(x)) with respect to x is calculated the following way:
x: Is the input sample into the neuron. In the above image, it has 3 features. This could be for example the height(1.75 m), weight(80kg) and age(30) of one human. Note that we only input one sample (one human in our example) into the neuron at once.
a: Is called the activation and it is the output of the neuron. It is a prediction the neuron makes according to one input sample. We can choose ourselfs what our neuron should predict. A useful thing to predict in our human example could be the BMI.
w: Weights of the neuron. It has seperate weights for each input feature of vector x.
b: bias of the neuron. One neuron has only one bias.
y: Is the true or target value of the output of the network. We want a to be as close to y as possible. We therefore change the weights w and the bias b in the training phase.
L: Is called the Loss. The closer a is to y, the smaller the loss is and therefore the better. Hence we want to minimize the loss. The most common way to minimize the loss in neural network is gradient descent. I will use the “mean squared error” loss over the course of this blogpost.
During training w is updated in the following way.
Where w ́ is the new updated weight vector and λ is the learning rate. λ is a hyperparameter and can be chosen freely. Normally it is around 0.05.
∇L(w) is the gradient of the loss with respect to the weights. It is simply the Transpose of the derivative of the loss L with respect to the weights. In general the gradient of a function with respect to some varaibles is always the transpose of the according derivative.
b ́ is the new updated bias. ∇L(b) is the gradient of the loss with respect to the weights.
In this case it does not matter, that the gradient is the transpose of the derivative, because it is a single number anyway and transposing a single number does not change it.
Applying vector chain rule:
Derivative with respect to the weights:
x does not depend on w, therefore it be treated as a constant and the derivative must only be taken with respect to w.
Derivative with respect to the bias:
Code to test our neuron:
We then test our data with a dummy dataset to see, if it works. Each sample has 3 features. The desired output is 1, if the first feature is 1. The other 2 features do not matter for the outcome. So we expect the first weight to become 1, while the other weights and the bias will go to 0.
We test the trained network on a sample it has never seen during training to see how well it generalizes to unseen data.
Test: predicted y: 0.9992918862001037, true y: 1
The ouput from our test sample shows, that we got very close to the true value.
each data point has size 3:
we have datapoints x1, x2, ...
The dots denote, that the number of datapoints (m) we can input in the neuron is variable.
To get the input matrix, we stack the input vectors vertically:
We have to adjust the loss as following:
where m is the number of datapoints we use.
a, the activation vector is then calculated the following way:
Applying vector chain rule:
Calculating the derivatives.
With respect to the weights:
With respect to the bias:
Stochastic gradient descent means, that we train on all training samples at the same time.
We can see the same evolution of weights, bias and loss as in the neuron with a single input.
We can see below, that the predicted output is very close to the true y:
Test: predicted y: 0.9999991060923578, true y: 1
Minibatch gradient descent means, that we train the neuron on a subset of the training set at the same time.
We have to write a function to generate random batches.
predicted y: 0.9956078874633946, true y: 1
Activation functions are nonlinear functions.
3 very important activation functions and their derivatives:
tanh:
simoid:
rectified linear unit:
Applying vector chain rule:
Calculating the derivatives:
The np.diag function constructs a matrix from a vector where the diagonal of the matrix is equal to the vector and it is 0 off-diagonal.
With respect to the weights:
With respect to the bias:
Activation functions:
Neuron:
Minibatch gradient descent
The values, the weights and the bias converge to are different to the previous cases.
prediceted y: 0.9394357707018176, true y: 1
Forward pass
The superscripts in parentheses denote the layer.
Layer 1 (= input layer):
Layer 2 (= output layer):
Loss:
Layer 1 (= input layer):
Layer 2 (= output layer):
Calculating the derivatives of layer 2 poses a problem since we would have to take the partial derivatives of vectors with respect to matrices and matrices with respect to matrices. This would result in tensors, which would make it unnecessary complicated.
We do not use the vector chain rule but calculate the final derivatives of the loss with respect to weights and biases directly. To do this, we:
Rewrite the forward pass in index notation. In this notation we can see what each element of the final derivative of the loss with respect to the weights and biases looks like.Calculate the derivatives.Work back to a vectorized form in order to write efficient code.
Rewrite the forward pass in index notation. In this notation we can see what each element of the final derivative of the loss with respect to the weights and biases looks like.
Calculate the derivatives.
Work back to a vectorized form in order to write efficient code.
Layer 1 (= input layer):
Layer 2 (= output layer):
Loss:
Layer 2 (= output layer):
To derive the formulas in index notation we can look at the computation graph. For example to derive the formula for the weights of layer 2, we construct a computation graph, where all variables that depend on each other are connected.
We rewrite the same coputation graph in index notation.
We now want to want to know derivative of the loss with respect to each weight. For each weight we have to sum over all branches of the graph, where this weight occurs. As an example for weight 1 of layer 2:
The derivatives of the loss with respect to all weights is then:
Written back in vectorized form:
Note that by looking at the left hand side of the equation we can see the dimension our vectorized form must have in the end. This helps us construct the dot product the right way around and gives us informations where we must transpose.
Written back in vectorized form:
Layer 1 (= input layer):
Written back in vectorized form:
Regrading the bias:
Written back in vectorized form:
To make the summation more clear:
“axis = 1” means we are summing over the columns.
To generalize the formulas we derived above we introduce the concept of input Loss into the neurons into a layer. The input error δ(L) into the last layer L is:
In vectorized form:
For all other layers l of our network:
In vectorized form:
f denotes the nonlinear activation function. This could be for example sigmoid or relu. And f′ denotes the derivative of f.
derivative with respect to weight:
In vectorized form:
derivative with respect to bias:
In vectorized form:
This Code can be used to create and train arbitrary deep neural networks. The list “layers” passed to the __init__ method can be changed and this changes the network. The lenght of the list is equal to the number of layers. The numbers are equal to the number of neurons in each layer. The first must be equal to the number of features.
Testing our code:
We once again look at the evoultion of the weights and the bias of the output layer and the evolution of the loss during training.
predited y: 0.9514167066315814, true y: 1
If you are still with me, I congratulate you, this was a though ride. And don’t be desperate if you did not get it the first try. It took me more than a month to figure this all out.
Today you have learned how to derive the gradients for weights in arbitrary deep neural networks. You should now also be able to apply this knowledge to other proplems like linear regression, Recurrent neural networks and reinforcement learning to name a few examples.
But be aware of the fact that there exist very good autograd framework nowadays, which are capable of numerically calculating the gradients for you. I am nevertheless of the mind that deriving these formulas by hand once helps our general understanding of how neural networks operate and some problems that arise like vanishing and exploding gradients in vanilla RNN.
towardsdatascience.com
medium.com
Linkedinhttps://www.linkedin.com/in/vincent-m%C3%BCller-6b3542214/Facebookhttps://www.facebook.com/profile.php?id=100072095823739Twitterhttps://twitter.com/Vincent02770108Mediumhttps://medium.com/@Vincent.MuellerBecome medium member and support me (part of your membership fees go directly to me)https://medium.com/@Vincent.Mueller/membership