Simple Linear Regression Using Python — An entry into Data Science World
Tags: Machine Learning, Linear Regression, Python Programming, Data Science, Regression

Understanding Linear Regression is the first step towards mastering more intricate topics of Machine Learning. I have picked up a simple problem to help you understand how to write a linear regression algorithm using Python. From gathering data, to cleaning and plotting it, I have tried to explain each step in the simplest way possible. In this article, I have assumed that you already know the math behind linear regression, therefore, I have not explained the theory of regression instead just focussed on how to deal with the data.

Great! Let’s start.

The problem?

Imagine you are hired as a Data Scientist by a newly launched movie production house and your company has put a lot of money ($175,000,000) in a new movie. Being their first production, they are worried how the movie would perform at the box office. You take up the task at hand and try to predict its earnings. Now the performance of the movie depends on a lot of factors, some prominent ones being the cast in the movie, the director of the movie, genre, and movie budget. You pick the factor, movie budget and try to find if it has any sort of correlation with its earnings.

You start digging up the internet and find data of 50000+ movies which should help you to find some sort of relationship between the budget and movie earnings. Once you find the relationship, you can predict the earnings of your own movie.

Photo by @jakobowens1 on Unsplash.

Now that we have a problem at hand. Let’s use some Python to create our prediction model.

X = Budget

y = Revenue

Data downloaded from: https://www.kaggle.com/rounakbanik/the-movies-dataset?select=movies_metadata.csv

IDE Used: Jupyter

Importing required modules:

import pandas as pd

import sklearn.linear_model as sk

Importing my data:

df = pd.read_csv("movies_metadata.csv")

Viewing the first 2 rows of my dataset to understand the type of data present in the set.

Since there are 24 columns and they are not all visible on the screen, I would like to know which columns are present and then decide which one is relevant for my research. So I am going to use the below line of code to fetch the list of columns.

Now from this analysis I want to find out if I can predict the earnings through the budget so I am going to delete all the irrelevant columns and keep only the relevant ones.

I have deleted all irrelevant columns and kept only 3 columns in the dataset.

I did not delete status column just yet because I sense that there might be some statuses which could be irrelevant and might disrupt the result of my analysis, if not removed. Hence, I would, first, like to go through their types and find out which ones are relevant. I would use the below line of code to check the type of statuses.

For this research, I am only concerned with the movies that are released, hence I would be deleting all rows except those against Released.

In the above code, I have created a list of statuses which I want to remove from my dataframe. Once that list is created, I have passed it in isin method and dropped it using drop method and executed the block of code. Now let me check if I have only released statuses.

Awesome! Now let me drop this column as well and keep only two columns viz. Budget and Revenue.

Super. Now my dataset looks a little clean but is it? Let me check the datatypes of my two columns.

If you notice, the datatype of the budget column is object and this wouldn’t work in my regression analysis. I need to change that to float. This is little tricky as people would mostly use the below code to change it from datatype object to datatype float.

df['budget'].astype(str).astype(int)

But this would throw an error since there might be NaN values. Let me try a different trick to change it from datatype object to datatype float.

Convert_dtypes has changed the data to datatype string. Now I would change it to datatype float using the below code.

Wonderful. Now both the columns are datatype float and wouldn’t throw any error in the analysis. Next, I suspect that there would be a lot of junk values in budget and revenue columns. Let me check if there are any.

I want to find out if there are any rows where the value of budget is less than $10,000. A movie with a budget less than $10,000 is not really useful for my analysis, as it wouldn’t really help me in my prediction.

My fear proved to be true, there are more than 35000 movies with a budget less than $10,000. For the sake of this analysis, I will remove all such movies.

Noice! After removing movies which had their budget less than $10,000, my dataset now contains 8556 movies. Now let’s check the revenue column. For the sake of this research, I do not want any values where revenue is less than $1000.

I have repeated the same steps for revenue column and deleted 3253 movies from my dataset. My dataset now contains 5303 movies.

Finally, I will check for any null values in my dataset.

With the above line of code, I have checked for null values and it showed that my dataset indeed has some null values. I will remove all those rows where there are null values.

I have deleted all the rows which had even one null value. If you notice, my dataset earlier had 5303 rows and after removing null values the dataset contains 5300 rows which means there were 3 rows with null values.

Awesome! Now my data looks cleaned. Let me plot it and see how it looks.

I have used matplotlib module to plot the graph between Budget and Revenue. On the first view of this graph, I can see there is some sort of relationship between Budget and Revenue.

Finally, let’s write a regression algorithm to predict revenues against their budgets.

Great! Our Linear Regression model is trained by the training data and it is ready to predict values. Let us find the coefficient and intercept.

Super. Our regression line is:

y = 3.02206903*x + -3691165.1759039164

Let us draw this line on our data.

I have added the below line of code to draw the regression line.

plt.plot(df[['budget']],reg.predict(df[['budget']]),color="blue",linewidth=3)

We are doing great! Now it is time to predict how much revenue our movie will generate at the box office. Our company has put $175,000,000 into making this movie. Let us predict its earnings.

We could have also used the equation of line viz.

y = mx + c

m = 3.02206903

c = -3691165.1759039164

x = 175000000 y = 3.02206903*175000000 + (-3691165.1759039164)

y = 525170915

The predicted value of the revenue is $525,170,915. That’s quite a decent number given our first movie, right?

Let us also check the R-square value.

It is quite a good number given that we have used just one variable viz. budget and it explains almost 53% of the variations in the revenue generated by the movie. Of course, this model is extremely simple and we have just considered a single variable. There can be many variables that might affect the movie earnings but that is left for the next article.