Tutorial: Linear Regression with Stochastic Gradient Descent
By: Raimi Karim
Tags: Backpropagation, Linear Regression, Gradient Descent, Machine Learning

2. Implementation

The workflow for training our model is simple: forward propagation (or feed-forward or forward pass) and backpropagation.

Definition: training

Training just means regularly updating the values of your weights, put simply.

Below is the workflow. Click to jump to the section.

— — — — — — — — — — — — —

2.1 Forward propagation

2.1.1 Initialise weights (one-time)

2.1.2 Feed data

2.1.3 Compute ŷ

2.1.4 Compute loss

2.2 Backpropagation

2.2.1 Compute partial differentials

2.2.2 Update weights

— — — — — — — — — — — — —

Let’s get started.

To keep track of all the values, we first build a ‘computation graph’ that comprises nodes colour-coded in

orange — the placeholders (x₁, x₂ and y), dark green — the weights and bias (w₁, w₂ and b), light green — the model (ŷ) connecting w₁, w₂, b, x₁ and x₂, and yellow — the loss function (L) connecting the ŷ and y.

Fig. 2.0: Computation graph for linear regression model with stochastic gradient descent.

For forward propagation, you should read this graph from top to bottom and for backpropagation bottom to top.

Note

I have adopted the term ‘placeholder’, a nomenclature used in TensorFlow to refer to these ‘data variables’.

I will also use the term ‘weights’ to refer to w and b collectively.

2.1 Forward Propagation

2.1.1 Initialise weights (one-time)

Since gradient descent is all about updating the weights, we need them to start with some values, known as initialising weights.

Here we initialised the weights and bias as follows:

These are reflected in the dark green nodes in Fig. 2.1.1 below:

Fig. 2.1.1: Weights initialised (dark green nodes)

There are many ways to initialise weights (zeros, ones, uniform distribution, normal distribution, truncated normal distribution, etc.) but we won’t cover them in this post. In this tutorial, we initialised the weights by using truncated normal distribution and the bias with 0.

2.1.2 Feed data

Next, we set the batch size to be 1 and we feed in this first batch of data.

Batch and batch size We can divide our dataset into smaller groups of equal size. Each group is called a batch and consists of a specified number of examples, called batch size. If we multiply these two numbers, we should get back the number of observations in our data. Here, our dataset consists of 6 examples and since we defined the batch size to be 1 in this training, we have 6 batches altogether.

Current batch of data used to feed in the model is bolded below:

x1 x2 y

1) 4 1 2

2) 2 8 -14

3) 1 0 1

4) 3 2 -1

5) 1 4 -7

6) 6 7 -8

Eqn. 2.1.2: First batch of data fed into model

In Fig. 2.1.2, the orange nodes are where we feed in the current batch of data.

Fig. 2.1.2: Feeding data to model with first batch (orange nodes)

2.1.3 Compute ŷ

Now that we have the values of x₁, x₂, w₁, w₂ and b ready, let’s compute ŷ.

Eqn. 2.1.3: Compute ŷ

The value of ŷ (=-0.1) is reflected in the light green node below:

Fig. 2.1.3: ŷ computed (light green node)

2.1.4 Compute loss

How far is our predicted ŷ from the given y data? We compare them by calculating the loss function L as defined earlier.

Eqn. 2.1.4: Compute the loss

You can see this value in the yellow node in the computation graph.

Fig. 2.1.4A: L computed (yellow node)

It is a common practice to log the loss during training, together with other information like the epoch, batch and time taken. In my demo, you can see this under the Training progress panel.

Fig. 2.1.4B: Logging loss and other information

2.2 Backpropagation

2.2.1 Compute partial differentials

Before we start adjusting the values of the weights and bias w₁, w₂ and b, let’s first compute all the partial differentials. These are needed later when we do the weight update.

Fig. 2.2.1: Indicated partial differentials to the relevant edges on the graph

Namely, we compute all possible paths leading to every w and b only, because these are the only variables which we are interested in updating. From Fig. 2.2.1 above, we see that there are 4 edges that we labeled with the partial differentials.

Recall the equations for the model and loss function:

Model

Loss function

The partial differentials are as follows:

L (yellow) — ŷ (light green):

Eqn. 2.2.1A: Partial differential of L w.r.t. ŷ

ŷ (light green) — b (dark green):

Eqn. 2.2.1B: Partial differential of ŷ w.r.t. b

ŷ (light green) — w₁ (dark green):

Eqn. 2.2.1C: Partial differential of ŷ w.r.t. w1

ŷ (light green) — w₂ (dark green):

Eqn. 2.2.1D: Partial differential of ŷ w.r.t. w2

Note that the values of the partial differentials follow the values from the current batch. For example, in Eqn. 2.2.1C, x₁ = 4.

2.2.2 Update weights

Observe the dark green nodes in Fig. 2.2.2 below. We see three things:

i) b changes from 0.000 to 0.212

ii) w₁ changes from -0.017 to 0.829

iii) w₂ changes from -0.048 to 0.164

Fig. 2.2.2: Updating the weights and bias (dark green nodes)

Also pay attention to the ‘direction’ of the pathway from the yellow node to the green node. They go from bottom to top.

This is stochastic gradient descent — updating the weights using backpropagation, making use of the respective gradient values.

Let’s first focus on updating b. The formula for updating b is

Eqn. 2.2.2A: Stochastic gradient descent update for b

where

b — current value

— current value b’ — value after update

— value after update η —learning rate, set to 0.05

∂L/∂b — gradient i.e. partial differential of L w.r.t. b

To get the gradient, we need to multiply the paths from L leading to b using chain rule:

Eqn. 2.2.2B: Chain rule for partial differential of L w.r.t. b

We would require the current batch values of x, y, ŷ and the partial differentials so let’s just place them below for easy reference:

Eqn. 2.2.2C: Partial differentials

Eqn. 2.2.2D: Values from current batch and the predicted ŷ

Using the stochastic gradient descent equation in Eqn. 2.2.2A and plugging in all the values from Eqn. 2.2.2B-D gives us:

That’s it for updating b! Phew! We are left with updating w₁ and w₂, which we update in a similar fashion.

End of batch iteration

Congrats! That’s it for dealing with the first batch!

x1 x2 y

1) 4 1 2 ✔

2) 2 8 -14

3) 1 0 1

4) 3 2 -1

5) 1 4 -7

6) 6 7 -8

Now we need to iterate the above-mentioned steps to the other 5 batches, namely examples 2 to 6.

Iterating through batch 1 to 6 (apologies for the poor GIF quality! )

End of epoch

We complete 1 epoch when the model has iterated through all the batches once. In practice, we extend the epoch to more than 1.

One epoch is when our setup has seen all the observations in our dataset once. But one epoch is almost always never enough for the loss to converge. In practice, this number is manually tuned.

At the end of it all, you should get a final model, ready for inference, say:

Let’s have a review of the entire workflow in a pseudo-code:

initialise_weights() for i in epochs: for j in batches: #forward propagation

feed_batch_data()

compute_ŷ()

compute_loss() #backpropagation

compute_partial_differentials()

update_weights()

Improve training

One epoch is never enough for a stochastic gradient descent optimisation problems. Remember that in Fig. 4.1, our loss is at 4.48. If we increase the number of epochs, which means just increasing the number of times we update the weights and biases, we can converge it to a satisfactory low.

Below are the things you can improve the training: