Backpropagation from Scratch: How Neural Networks Really Work | by Florin Andrei | Towards Data Science

How do neural networks really work? I will show you a complete example, written from scratch in Python, with all the math you need to completely understand the process.
I will explain everything in plain English as well. You could just follow along, read just the text and still get the general idea. But to re-implement everything from scratch on your own, you will have to understand the math and the code.
I’ve seen several articles such as this, but in most cases they were incomplete. I wanted to give this topic a complete treatment: all the math + working code, batteries included.
You must grasp linear algebra, calculus, and statistics to fully understand the math part: matrix multiplication, dot product, partial derivatives, the chain rule, normal distribution, probabilities, standard deviation, etc.
We have a large set of images, each 28 x 28 pixels, containing hand-drawn numbers. We need to write an app that can look at the images and recognize the numbers drawn in there. In other words, if the app sees this image...
...then it should print out this...
[0, 0, 0, 0, 1, 0, 0, 0, 0, 0]
...which means “I see the number 4”: there are 10 slots in there, the first corresponds to number 0, then next to number 1, the next is number 2, and so on; all have the value 0 except for the slot corresponding to number 4 which has the value 1. That means the app has “recognized” number 4.
Or if it sees this image...
...then it should print out this...
[0, 0, 1, 0, 0, 0, 0, 0, 0, 0]
...which means “I see number 2”.
This is called image recognition because we’re recognizing images. It is also called classification because we have a bunch of inputs (images of digits) that need to be classified (categorized) into 10 different categories (the kinds of digits they represent, 0 through 9).
The images are literally just matrices full of numbers. Each pixel is a different number in the matrix. We could use square 28 x 28 pixel matrices, or we could simply unroll each matrix into a string of 784 numbers — the software actually doesn’t care about the shape, as long as we’re consistent.
A neural network should work pretty well for image classification. Let’s try to build one from scratch.
A neuron is the basic building block of a neural network. It has several inputs Ii and an output O.
The internal sum Z is simply the weighted sum of all inputs I (multiplied by their weights W):
Usually, there is a constant bias term B added at the end of the formula for Z (so Z typically ends with “... + B”). In this article we will ignore bias; the network will perform somewhat worse, but the math will be simpler.
Z could have values in a wide range, but for O we want to keep the values in a sane interval, such as [0, 1], which are similar to probabilities, and just easier to deal with. You also want some non-linearity in there, or else the output will always be a linear aggregate of all inputs, which is definitely not what you want.
So we need to transform Z into O using a function such as the sigmoid, whose output is always in that range. We say the sigmoid is the activation function of the neuron. There are many different activation functions A(Z) that we could use instead, but the sigmoid is good for a basic example, so we’ll use it here.
Later on, we will need the derivative of the sigmoid as well, so let’s stick it in here:
Python code for the sigmoid:
def sigmoid(x, derivative = False):    if derivative:        return np.exp(-x) / ((np.exp(-x) + 1) ** 2)    else:        return 1 / (1 + np.exp(-x))
Neural networks are typically made of several layers, each layer containing multiple neurons:
Writing all equations for all connections, to calculate the final output based on some input can be pretty boring. A network can have a lot of connections, and that means a giant pile of equations.
However, matrix notation can replace a system of equations with a much simpler matrix equation.
For each layer l, the internal sums Z of all neurons are the weighted sums of the outputs of the previous layer. The outputs of each layer are simply the activation function applied to all internal sums of that layer. In matrix form for layer l:
Let’s extend these equations to the whole network.
We will use the X symbol to denote the input (the hand-written images we’re recognizing). So, let’s write all of the above in matrix notation, all layers, all values.
The final equation for On is a bit different (the final layer gets special treatment), we will get back to it in a few moments when we talk about softmax below.
The matrices are multiplied using the dot product.
The shape of each Wl matrix is rectangular; the size of one side is the number of neurons in the current layer; the size of the other side is the number of neurons in the previous layer; the total number of weights is the product of the number of neurons in the previous layer times the number of neurons in the current layer. This is the “magic” that makes the dot product work.
When we create the network, we don’t know what weights we will need. We could simply assign random values to the weights. For example, we could pull random values from the standard normal distribution.
To reduce the amount of very large values that the weights may get, we will narrow down the standard deviation of the distribution, proportional to the number of weights. More weights = more narrow distribution (closer to 0). This will avoid having to deal with very large numbers when computing Z.
The number of neurons is a bit tricky to figure out. The input and output are determined already: if you have 784 pixels in the image, you need 784 inputs. If you have 10 categories to recognize, you need 10 outputs. For the other layers, it gets more complicated; in general, it’s a good idea for the number of neurons to gradually “diminish” from input to output, layer by layer.
See the code below. The whole network (the “model”) is in the variable called “model”. It has 4 layers. The size of the input layer is the same as the size of one input sample data (784). The size of the output is the same as the size of one output sample data (10). The other two layers have intermediate numbers of neurons (128 and 64). Remember, we only store the weights W in the model.
def nn_init(lsize):    # human-readable names for clarity    input_layer  = lsize[0]    hidden_1     = lsize[1]    hidden_2     = lsize[2]    output_layer = lsize[3]        # narrowing down the standard deviation by layer size, with np.sqrt()    # large layers have tighter initial values    nnet = {        'w0': np.random.randn(hidden_1, input_layer) * np.sqrt(1. / hidden_1),        'w1': np.random.randn(hidden_2, hidden_1) * np.sqrt(1. / hidden_2),        'w2': np.random.randn(output_layer, hidden_2) * np.sqrt(1. / output_layer)    }    return nnetlayer_sizes = [x_train[0].shape[0], 128, 64, y_train[0].shape[0]]model = nn_init(layer_sizes)
We’re doing classification, so the network has K outputs, one for each category of object we’re recognizing. In this case, K = 10, since the objects are digits and there are 10 categories (0, 1, 2, ..., 9).
We could use the sigmoid function on each output as we do on all other neurons in the network. Then each output will have values between 0 and 1, and the highest value (closest to 1) wins. But we could go one step further: to make the outputs look like actual probabilities, let’s make sure their sum is always 1. Each output is between 0 and 1, and their sum needs to be exactly 1, because the total probability that the object is in any class out of the total of K classes is always 1 (we never show “foreign” objects to the network, at least not during training, which is the part that matters).
That could be done with the softmax function. For each output Oi we calculate its value like this:
The Python function below calculates softmax, but the exponentials are shifted to avoid computing very large values. The result is the same, the function looks a bit more complex, but behaves better in practice.
def softmax(x, derivative = False):    # for stability, we shift values down so max = 0    # https://cs231n.github.io/linear-classify/#softmax    exp_shifted = np.exp(x - x.max())    if derivative:        return exp_shifted / np.sum(exp_shifted, axis = 0) * (1 - exp_shifted / np.sum(exp_shifted, axis = 0))    else:        return exp_shifted / np.sum(exp_shifted, axis = 0)
Okay, we have the network, we apply a sample value (an image of a digit) at the input, how do we calculate its output? We go layer by layer, multiplying inputs / previous outputs by weights, we apply activation functions (sigmoid), and so on until we reach the output, where we apply softmax. Let’s show the equations again:
That’s all. That’s how a fully trained network works. It’s a series of linear algebra operations, layer by layer, that end up computing the output based on the input and the network weights.
The Python function below does exactly that. It assumes the model (all W matrices) is a global variable (for code simplicity — this is school stuff, not production code), it takes one input sample X as an argument, and returns a dictionary with a bunch of matrices, each one containing the O and Z values for each layer in the network.
The last O matrix in that dict is the network output. That’s the result we’re looking for.
def forward_pass(x):    # the model is a global var, used here read-only        # NN state: internal sums, neuron outputs    nn_state = {}        # "output zero" is the output from receptors = input to first layer in the NN    # these are activations for the input layer    nn_state['o0'] = x        # from input layer to hidden layer 1    # weighted sum of all activations, then sigmoid    nn_state['z1'] = np.dot(model['w0'], nn_state['o0'])    nn_state['o1'] = sigmoid(nn_state['z1'])        # from hidden 1 to hidden 2    nn_state['z2'] = np.dot(model['w1'], nn_state['o1'])    nn_state['o2'] = sigmoid(nn_state['z2'])        # from hidden 2 to output    nn_state['z3'] = np.dot(model['w2'], nn_state['o2'])    nn_state['o3'] = softmax(nn_state['z3'])        return nn_state
But how to train a network? If we start with random weights and apply an input X (an image), the output On is going to be nonsense, it’s not going to be anywhere near the “true” output Y.
The central idea of training is that we must adjust the weights W until the output of the network On is close to Y (the “ideal” value). The training algorithm at a high level may look like this:
initialize a network with random weights
apply an input sample X (an image)
do forward propagation and compute the output On
compare the output On with the real value Y that you want to get
adjust the weights of the last layer so the output gets a little “better” (the error is reduced)
go back layer by layer and adjust weights so the error is reduced (this is the hard part)
apply the next input sample X (another image)
repeat all of the above thousands of times, using many different pairs of input X and output Y samples, until the network performs well enough
The hard part is figuring out how to adjust the weights, going backward from output to input, so the error is diminished. That is called backpropagation. But to get there, we need to create a few tools first.
Let’s say we have the network output On and we compare it with the “perfect” output Y from the training data. How close is On to Y, how do we define “closeness”? If we had a function with a value that would increase as On becomes more and more “different” from Y, we could call that a cost function (because mistakes are expensive), and we could use it to reduce the cost.
For this classification problem, cross-entropy could play the role of cost (other functions are used in other cases). Assuming we have K outputs, it is the sum over all outputs of:
where Oi are the various outputs on the network, and Yi are the corresponding training values for each output.
The reason why this formula works is Yi can be either 0 or 1. If you have 10 values in a Y sample, then nine of them will be 0 (the object X does not belong to any of those categories) and one of them will be 1 (the object X belongs to that category). Then either one or the other terms in the sum becomes 0, and the other term depends only on Oi.
If both O and Y are 1-dimensional vectors (which they are), then in matrix notation using the dot product the equation above becomes:
The logarithm is applied element-wise, and so is the difference 1 — Y.
The Numpy library can do matrix operations, so the Python function for cost is a direct, literal translation of the last formula:
def part_cost(o, y):    c = np.dot(y, np.log(o)) + np.dot((1 - y), np.log(1 - o))    return -c
Let’s keep this function in mind, we will use it later to evaluate the performance of the network.
This is the keystone of the argument: given a cost C calculated from the output O and the training value Y, we want to change the weights W so that C is reduced (by making O more similar to Y).
If we could calculate the derivative of C with regards to W (let’s call it Δ), and then adjust W in the down direction of the derivative plane, then C would decrease. If the derivative is positive then we decrease W (because the minimum is towards smaller Ws), and vice-versa. Another name for this technique is gradient descent.
In other words: think of the cost function C in the W space. Forget about X or anything else. Think only of C(W). You need to walk C through the W space until you stumble upon some minimum in the W space. That’s what “training” means: find the best set of W values that minimizes C.
What follows is not strict proof. It’s more of a mathematical intuition. If you follow the equations carefully, you will notice a few places where we’re abusing the math (e.g. confusing differentials with actual finite variations). In practice, it will work out — neural networks are robust against error.
Let’s define δ as the partial derivative of the cost C w.r.t. the internal sums Z. For the layer l this is written:
For the previous layer l-1 we obtain by applying the chain rule:
But we know this to be true from the definition of Z:
And by plugging this last formula into the previous one, we get:
This formula is recursive. All we need is the δl for the last layer, which we can calculate from C and Zl which we know from forward propagation. Once we have δl we can go back layer by layer and calculate the δ values for all layers, one by one.
More importantly, now we can also go back layer by layer and compute the derivative Δl of the cost function C with regards to the weights Wl of each layer l. This is what we really need to make backpropagation work. We’re getting close now.
Let’s recalculate Δ for each layer using the chain rule:
In other words:
But we have all the Ol values from forward propagation (they are the outputs of each layer), and we’ve computed all the δl values above. That means we can compute Δl for all layers, starting at the output and moving back through the network.
Once we have all the big deltas, we can use them to update the weights. In practice, the updates will be weighted down by a learning rate factor λ which is typically a small value (0.001), so we don’t make sudden large changes to the network.
Again, if Δ (the derivative of C w.r.t. W) is positive, that means there will be smaller C values in the direction of the smaller W values. So we need to decrease W to make C decrease. And vice-versa.
λ can be tricky to think about. Do not get stuck on “units” or anything like that. Just pick a λ value so the changes to W are small enough — baby steps. Whatever happens to Δ, after multiplying it by λ the result ought to be small compared to W. And that’s all there is to it. That’s enough to make the algorithm work.
For each training sample, modifying the network weights W to minimize the cost via backpropagation works like this:
apply the training sample X (the image) to the input
do forward propagation, calculating all the Z and O (output) values for all layers
calculate all the δ matrices recursively (backward) for all layers
calculate all Δ matrices from the δ matrices
update the weights
That’s a complete single backpropagation step. See the Python code below:
def backward_pass(x, y):    # do the forward pass, register the state of the network    nn_state = forward_pass(x)        # small deltas: derivatives of the error w.r.t. z    nn_state['d3'] = nn_state['o3'] - y    nn_state['d2'] = np.dot(nn_state['d3'], model['w2']) * softmax(nn_state['z2'], derivative = True)    nn_state['d1'] = np.dot(nn_state['d2'], model['w1']) * sigmoid(nn_state['z1'], derivative = True)        # large deltas: adjustments to weights    nn_state['D2'] = np.outer(nn_state['d3'], nn_state['o2'])    nn_state['D1'] = np.outer(nn_state['d2'], nn_state['o1'])    nn_state['D0'] = np.outer(nn_state['d1'], nn_state['o0'])        return nn_state
The backward_pass() function takes an image X and a training response Y as arguments. The model is a global variable. It returns the nn_state dictionary containing all the Z, O, d, and D matrices. d stands for δ here. The D matrices are the weight adjustment terms Δ calculated during backprop.
See the full training code below. “epochs” are the number of times the code needs to go over the whole training data; multiple passes (epochs) are needed for good training; we do 5 epochs. A training rate t_rate is defined, to be used for weight updates — it’s really the λ in our equations (sorry for the notation change).
You could update the weights after each input sample X (stochastic gradient descent), or you could accumulate W changes (accumulate the Δ matrices) for a whole epoch and apply everything at the end (batch gradient descent), or you could do something in between. Here we use stochastic descent.
The code also evaluates cost (divergence from perfect responses) and accuracy (the number of times the classification was correct).
At the end, the model (the W matrices) is saved to disk.
epochs = 5t_rate = 0.001# trainprint('################### training ####################')for e in range(epochs):    print('epoch:', e)        samples = x_train.shape[0]    cost = 0    hit_count = 0    for i in tqdm(range(samples)):        m_state = backward_pass(x_train[i], y_train[i])        # add partial cost        cost += part_cost(m_state['o3'], y_train[i])                # stochastic gradient descent        # update weights        model['w0'] -= t_rate * m_state['D0']        model['w1'] -= t_rate * m_state['D1']        model['w2'] -= t_rate * m_state['D2']                if np.argmax(m_state['o3']) == np.argmax(y_train[i]):            # successful detection            hit_count += 1# performance evaluation    cost = cost / samples    accuracy = hit_count / samples    print('cost:', cost, 'accuracy:', accuracy)# save the modelwith open('model.pickle', 'wb') as f:    pickle.dump(model, f)
So how well does the network do? We’ve trained it with the training data x_train and y_train. But we’ve set aside data for testing: x_test and y_test.
It’s always a good idea to leave some data alone just for testing. That’s data that the network has never seen in training. Therefore we’re guaranteed that the performance estimate will be fair. Something like 20% is a good size for the test data to set aside (assuming you have plenty of data — thousands of samples or more).
The evaluation only goes through the test data and calculates cost and accuracy, and prints out the results.
# testprint('################### testing ####################')# load the modelif os.path.isfile('model.pickle'):    with open('model.pickle', 'rb') as f:        model = pickle.load(f)# run the whole test datasamples = x_test.shape[0]cost = 0hit_count = 0for i in tqdm(range(samples)):    m_state = forward_pass(x_test[i])    cost += part_cost(m_state['o3'], y_test[i])    if np.argmax(m_state['o3']) == np.argmax(y_test[i]):        hit_count += 1# evaluate performancecost = cost / samplesaccuracy = hit_count / samplesprint('cost:', cost, 'accuracy:', accuracy)
Because we don’t use bias for our neurons, the performance of the network will be rather modest: about 60% accuracy. It could definitely do better. But this is a learning example, not production code.
This is the network performance in training, notice how it gets better with every epoch:
See below the performance in testing. It’s actually a bit better than in training, which is not typical, but the behavior of the training algorithm is a bit random to some extent.
Here’s a Jupyter notebook with the full code used in this article:
https://github.com/FlorinAndrei/misc/blob/master/nn_backprop.ipynb
It downloads the MNIST_784 dataset, which is a bunch of images with hand-written numbers, about 70,000 images total. It uses functions from the Tensorflow libraries to prepare the training/testing split (which is totally overkill, but very convenient).
It trains and tests the network. As a bonus, it generates a few images used in this article.
All images and all code were created by the author of this article.
I took the machine learning class by Andrew Ng on Coursera several years ago, and it’s very useful if you want to understand the mathematical intricacies of this algorithm.
www.coursera.org
Thanks for reading!