Machine Learning Concept behind Linear Regression
By: Vitou Phy
Tags: Machine Learning, Artificial Intelligence, Gradient Descent, Predictions, Linear Regression

Machine Learning Concept behind Linear Regression

Applying Simple ML model on CO2 Prediction

Photo by Alexander Tsang on Unsplash

Introduction

In the past recent years, there have been a lot of hypes on Artificial Intelligence (AI). You can find it almost anywhere from turning on the light with your voice to a fully autonomous self-driving car.

Most of modern AI requires a lot of data. The more you give, the better it learns. For instance, to train AI to understand an image of a cat, you need to give a lot of images of cats and non-cats so that it can distinguish between the two.

But how exactly does AI learn from the data?

In this post, we will look at a very simple model to get an idea of how AI learns. We will focus on the amount of global CO2 emission for the past 55 years and attempt to predict its amount in 2030.

CO2 Emission Data

The data we will be using is from WorldBank. Unfortunately, the data is not up to the current date of 2019 (at the time of writing this). It’s from 1960 to 2014, but this will do just fine for this experiment.

Fig 1: Amount of CO2 Emission annually from 1960 to 2014. Source: WorldBank Data

On the x-axis corresponds to the year (assuming year 0 is 1960) and on the y-axis corresponds to the amount of CO2 emission. Based on the chart, we might be able to make a rough estimation of what the value of 2030 should be.

Suppose we just draw a straight line that best fits this data, we could predict the future if we extend this line further and further.

Fig 2: Fit a line and use it to make a rough estimation at year 70 (2030)

Keep in mind that we can not accurately predict the future. Things might change for the better, but for simplicity, we just assume that the rate of changing is constant.

Linear Regression

Let’s bring in a little bit of math. The blue line above might be familiar to you. It’s a simple straight line that is represented by the equation below.

Uh huh. Now you remember. Again, x is the year and y is the amount of emission. To get the line shown in Figure 2, m=446,334, and b=9,297,274. Don’t worry about m and b. I will explain them in detail later.

In case we want to know info on the year 2030 (year 70 if we count from 1960), we can now use our equation above.

Now as promised, let’s take a close look at what m and b are.

Fig 3: Behavior of m on the line

In Figure 3, as we change the value of m, the line rotates around. Hence, variable m controls the direction of the line. Whereas in Figure 4, the variable b controls the position of the line by moving it up or down.

Fig 4: Behavior of b on the line

Learning Process

With m and b, we can control the line and adjust it to best fit our data.

The question now is how do we find the value of variables m and b? The idea is the following:

Randomize the value of m and b. Give the variables to a loss function to determine how bad the line is compared to the data, also known as error rate. Adjust the value of m and b based on the error rate. Go back to Step 2. Repeat until the variables stop changing.

Loss Function

If our line is very bad, this loss function will give a very big error. At the same time, if the line fits the data well, the error will be small.

Fig 5: The difference between predicted line and actual data

The line has its predicted value y’ every year. Then, we can compare the predicted value y’ to actual value y to find the difference. We compute this value on every year and take its average. This is also known as Mean Squared Error (MSE).

Fig 6: Closed form of MSE

Now we are almost ready to update our variables. There is one small catch. The error rate we found earlier is always positive. Pretty much we do not know which direction should update our line. Should it rotate clockwise or counter-clockwise? That is when Gradient Descent comes in.

Fig 7: Formula to update each variable (in this case, variable m)

In a nutshell, this tells the direction and how much each variable affects the error. The more effect it has, the more value it should change. Then, we can use this information to update our variables. We won’t dive deep down into the derivative, but if you are interested, you can check out this video on Coursera. I find the explanation there quite clear.

Note the alpha α, also known as, learning rate, is to control how much we should update our variables. Usually, we set it to a small value, like 0.001, so that we can slowly update our variables toward optimal value.

Good news: in practice, we don’t manually do this derivation.

Popular frameworks like Tensorflow or PyTorch will compute this automatically. But I put it here to give a brief idea of how it can know which direction to change the value.

Implementation

So, do you need to code everything as described above to get linear regression working? Fortunately, many existing libraries simplify everything for you. In this example, we will explore scikit-learn, a machine learning library built for Python, to predict CO2 emission.

With just a few lines of codes, scikit-learn makes linear regression very accessible. Line 9 alone does all the necessary steps needed to train the model. Phew, you were worry for nothing, weren’t you? Not only for linear regression, but also this applies to implementing many other ML algorithms with just a few lines of codes.

Conclusion

That’s it. Not so complicated right (except the derivation part)? Linear regression is a simple method yet quite effective even though it is just a straight line. In this post, we only looked at the case where there is only one variable (year). Actually, this can be extended to handle multiple variables, but that is the topic for later post.