Back Propagation, the Easy Way (Part 1)
By: Ziad Salloum
Tags: Machine Learning, Neural Networks, Deep Learning, Gradient Descent, Backpropagation

Following the article on Gradient Descent, the Back Propagation makes good use of this technique in order to compute the “right” values of weights in a neural network.

Important: This article contains a considerable number of equations, however they are all logically connected. The reader should have some patience going through them, and understand how they are built. It is essential to the good comprehension of the Back Propagation technique.

As with the Gradient Descent approach, we borrow the radio metaphor again to help understand the intuition of the Back Propagation. Whereas in the other article there was only one button to handle in order to get an acceptable output, in here we have many buttons that need adjusting.

The image above shows that there is a fine tuning button on each line connecting two nodes (even though they are not all represented).

As seen before, our job is to adjust those buttons so that the error at the output is minimum, which means that the effective output is as close as possible to the expected results.

In the Gradient Descent we talked about rather a simple relation between one fine tuning parameter 𝜃 and the error or loss 𝓛, in this case we have a more complex relation.

However we will start slowly by considering only one neuron.

Case of One Neuron

As a reminder a single neuron receives one or more inputs x and multiplies each of the input by a certain weight w, sums them and adds a bias b, then applies an activation function g() to produce an output ŷ.

To be able to assess the output of the neuron, a loss function 𝓛 is used. One such loss function could be 𝓛(y, ŷ) = ½(y - ŷ)²

The aim, of course, is to determine values of w and b in order to minimize 𝓛.

By looking at the neuron architecture we can easily notice the following chain of events:

Variation of wᵢ causes z to vary,

variation of z causes g(z) to vary,

variation of g(z) causes 𝓛(y, ŷ) to vary.

For simplicity of the notation let’s call a = g(z), it follows that ŷ = a.

The above chain reaction can be derived using the chain rule which states that the variation of the output relative to the wᵢ is the multiplication of the partial derivative of the phases in between:

∂𝓛/∂wᵢ =∂𝓛/∂a * ∂a/∂z * ∂z/∂wᵢ

Let’s consider each term alone:

∂𝓛/∂a = ∂(½(y - ŷ)²)/∂a = ∂(½(y - a)²)/∂a = -(y - a) = a - y

∂a/∂z = ∂g(z)/∂z = g’(z) (where g’(z) is the derivative of g(z). Note that g(z) can be any derivable function)

∂z/∂wᵢ = ∂(x1*w1 + x2 * w2 + … + xᵢ * wᵢ +…xn* wn)/∂wᵢ

= ∂(xᵢ * wᵢ)/∂wᵢ = xᵢ

Replacing each term by its value will give us the ∂𝓛/∂wᵢ

∂𝓛/∂wᵢ = (a - y) * g’(z) * xᵢ

Let’s define 𝛿 such as 𝛿 = (a - y) * g’(z), the final form of ∂𝓛/∂wᵢ will be:

∂𝓛/∂wᵢ = 𝛿 * xᵢ

If you recall from the Gradient Descent article, in order to find the 𝜃 that minimizes 𝓛 we iterate the following equation until ∆𝓛 becomes too small or zero:

𝜃𝑛₊₁ = 𝜃𝑛 -𝛂 . ∆𝓛

The same logic will be applied in the case of the neural network.

For each i, wᵢ⁺ = wᵢ - 𝛂*∂𝓛/∂wᵢ = wᵢ - 𝛂*𝛿 * xᵢ

where wᵢ⁺ is the new value of wᵢ.

Important: for each i we iterate wᵢ enough number of time until ∂𝓛/∂wᵢ becomes small enough

Using the same logic we can also find the effect of the variation of b:

∂𝓛/∂b =∂𝓛/∂a * ∂a/∂z * ∂z/∂b

Since ∂z/∂b = 1 and all the other terms have been already computed

∂𝓛/∂b = 𝛿

it follows that b⁺ = b - 𝛂*∂𝓛/∂b = b - 𝛂*𝛿

where b⁺ is the new value of b.

Again we keep iterating b enough number of time until ∂𝓛/∂b becomes small enough.

Case of Two Neurons

Now let’s take the case of two neurons put in sequence (they form two layers).

Before starting it is worth noting that the notation has changed to accommodate the new architecture. The superscript number denotes the layer. For example z², b², a², g² belong to layer #2, while z³, b³, a³, g³ belong to layer #3.

It is also worth mentioning that the output of layer #2 is a single value a², which is multiplied by the weight v in the next layer.

Layer #3

We will start with the last layer (layer #3) first, because its result will be used in the previous layer.

The chain of events is nothing unusual.

Variation of v causes z³ to vary,

variation of z³ causes g³(z³) to vary,

variation of g³(z³) causes 𝓛(y, ŷ) to vary.

Using the chain rule we compute the derivative ∂𝓛/∂v:

∂𝓛/∂v =∂𝓛/∂a³ * ∂a³/∂z³ * ∂z³/∂v

As previously done we compute each term alone:

∂𝓛/∂a³= a³ - y

∂a³/∂z³ = g³’(z³) (where g³’(z³) is the derivative of g³(z³))

∂z³/∂v = a²

So ∂𝓛/∂v becomes ∂𝓛/∂v = (a³ - y) * g³’(z³) * a²

Let’s define 𝛿³ = (a³ - y) * g³’(z³)

It follows that ∂𝓛/∂v = 𝛿³ * a²

Similarly we can find ∂𝓛/∂b³ = 𝛿 ³

Layer #2

Now let’s move to layer #2.

The chain reaction starts by applying variation to wᵢ which affects affects z²,

variation of z² affects g²(z²)

variation of g²(z²) affects z³ (note that at this point v is considered fixed)

variation of z³ affects g³(z³)

variation of g³(z³) affects 𝓛(y, ŷ)

The derivative equation is

∂𝓛/∂wᵢ =(∂𝓛/∂a³ * ∂a³/∂z³ * ∂z³/∂a²) * ∂a²/∂z² * ∂z²/∂wᵢ

We have already calculated ∂𝓛/∂a³ and ∂a³/∂z³, as for the rest they can be easily derived.

∂z³/∂a² = ∂(a² * v))/∂a² = v

∂a²/∂z² = ∂g²(z²)/∂z² = g²’(z²)

∂z²/∂wᵢ = xᵢ

After replacing each term but its value

∂𝓛/∂wᵢ = ((a³ - y) * g³’(z³) * v) * g²’(z²) * xᵢ

We already defined 𝛿³ as 𝛿³= (a³ - y) * g³’(z³)

∂𝓛/∂wᵢ = 𝛿³ * v * g²’(z²) * xᵢ

Lets define 𝛿² as 𝛿² = 𝛿³ * v * g²’(z²)

This will give the final form of ∂𝓛/∂wᵢ: ∂𝓛/∂wᵢ = 𝛿² * xᵢ

Similarly we can find ∂𝓛/∂b² = 𝛿 ²

Sequence of L Layers

Now consider a network with L layers each layer containing only one neuron.

Such network will look like the image below.

By using the same logic as above we can find the different components needed for the Back Propagation.

So for layer L

𝛿ᴸ = (aᴸ - y) * gᴸ’(zᴸ)

For any other layer 𝒍 < L

𝛿ˡ = 𝛿 ˡ⁺¹ * w ˡ⁺¹ * gˡ’(zˡ)

For any layer 𝒍 ≤ L

∂𝓛/∂wˡ = 𝛿ˡ * aˡ⁻¹

∂𝓛/∂bˡ = 𝛿ˡ

where aˡ⁻¹ is the output of the layer 𝒍-1, or if we are at layer 1 it will be the input x.

The formula for the gradient descent to find the wᵢˡ (weight at layer 𝒍, neuron i) that

wᵢˡ⁺ = wᵢˡ - 𝛂*∂𝓛/∂wᵢˡ = wᵢˡ - 𝛂*𝛿 ˡ* aˡ⁻¹

where wᵢˡ⁺ is the new value of wᵢˡ.

Important: for each wᵢˡ we iterate enough number of times until ∂𝓛/∂wᵢˡ becomes small enough

General Case

Multiple Layers with Multiple Neurons

In this section we have multilayer neural network with multi neurons in each layer instead of one as we have used so far.

Recall that in the previous cases we had one neuron per layer and we got the 𝛿ᴸ = (aᴸ - y) * gᴸ’(zᴸ) for layer L

and

𝛿ˡ = 𝛿 ˡ⁺¹ * w ˡ⁺¹ * gˡ’(zˡ) for layer 𝒍 < L

This is only for one neuron in each layer.

When we have multiple neurons per layer, we have to compute 𝛿 for each neuron of each layer.

So now variable with only superscript letter such as 𝛿ᴸ, w ˡ are vectors and matrices, whereas variables with both superscript and subscript letters such as 𝛿ᵢˡ and wˡᵢᵣ are single values.

Since we have multiple output layer, the definition of loss function 𝓛(y, ŷ) = ½(y - ŷ)² is no more sufficient. We have to account to all output neurons. So we define the cost function as the square sum of all output neurons

C=½∑ ᵢ(yᵢ−ŷᵢ)².

So the formulas that we have computed so far will have a vector form:

Where ∇aC is a vector of variation of the cost C relative to the output of the network aᵢᴸ, which is ∂C/∂aᵢᴸ .

The ⊙ operator is a member-wise vector/matrix multiplication.

The Back Propagation Algorithm

Input x: Set the input in layer 1. Forward: For each layer l = 2,3,…,L we compute

zˡ = wˡ aˡ⁻¹+bˡ

and

aˡ=gˡ(zˡ). Output error 𝛿ᴸ: At the output layer we compute the vector

𝛿ᴸ=∇aC⊙gᴸ′(zᴸ).

This will be the start of the back propagation. Back propagation: We move backward, for each layer l=L-1,L-2,L-3,…,2 we compute the error of each layer

𝛿ˡ=((w ˡ⁺¹)ᵀ * 𝛿 ˡ⁺¹) ⊙ gˡ’(zˡ).

Then we update the weights of each layer using the Gradient Descent Formula:

wˡ⁺ᵢᵣ = wˡᵢᵣ -𝛂 * 𝛿ˡᵢ * aᵣˡ⁻¹

and

bˡ⁺ᵢ = bˡᵢ -𝛂 * 𝛿ˡᵢ

where wˡ⁺ and bˡ⁺ are the updated values of wˡ and bˡ at layer l after each iteration. Output: At the end we will have the weights w and biases b at each layer that have been computed to minimize the cost function C.

Conclusion

Back propagation might be tricky to understand and trickier to implement in code, where it is easy to get entangled with matrices and vector and their dimensions. However, it is important for beginners to put enough efforts to gain enough intuition about this technique as it will help them acquire in depth knowledge of Neural Networks.

Related Articles

Part 2: Practical implementation of Back Propagation

Part 3: How to handle dimensions of matrices