Linear Regression with example | by Sandeep Khurana | Towards Data Science

Machine Learning — Fictional story. Once there was a doctor. He would look at person and predict if s/he has lack of Haemoglobin (red blood cells) or not. And almost all the time his prediction will come true upon testing the blood of the patient.
How would he predict? Lets assume pale color of face, tiredness in body are symptoms of low Hb (heamoglobin). If the weight of the person is less then more chances such symptoms are because of low Hb. The doctor “based upon his past experience” will “predict” after looking at the person.
What if we can make computer also learn things like this and make such prediction. Assuming we have numeric quantified variables like paleness and tiredness.
Linear Regression
We have seen equation like below in maths classes. y is the output we want. x is the input variable. c = constant and a is the slope of the line.
y = c + axc = constanta = slope
The output varies linearly based upon the input.
y is the output which is determined by input x. How much value of x has impact on y is determined by “a”. In the two dimensional graph having axis ‘x’ and ‘y’ , ‘a’ is the slope of the line.
‘c’ is the constant (value of y when x is zero).
Historical Data
Lets say we have good amount of historical data wherein values of input ‘x’ and output/result ‘y’ are provided.
We also know that change in x impacts y somewhat linearly. It means if we take the values of these ‘x’s and ‘y’s and plot of graph then we get a graph like this. Not exactly same but somewhat like this wherein there seems to be clutter of points going in one linear direction. Image courtsey — https://en.wikipedia.org/wiki/Linear_regression
What do we want to do
We want to look at the historical data and find the values of ‘c’ and ‘a’. Lets call ‘c’ as the constant and ‘a’ as the weight. If we are able to figure out the values of c and a then we can predict the value of y for any new x.
How to do
As you can see from the graph, although the dots form a linear pattern but they are not in same line. So we cant know the 100% perfect value of c and a. But what we can do it to find the best fit line, shown in red above.
This means our output will be approximate. There might be slight deviation from actual value. But it works fine for most business applications.
How in details
Look at the dots in the graph. What if we figure out a line whose distances from each dot in the graph is optimal/minimal. This would mean ‘best fit line’ as shown in red in the pic above. Our objective is to draw the red line in the graph above.
So, we draw a random line on the graph for some random value of c and a. Lets say we keep c and a both 1 (c=1, a=1) and draw the line on the graph for each (well at least 2 ) x. Based upon values of x this line might end up in one of the following positions
On leftish side of the dots towards y axis. More vertical.
On rightish side of dots towards x axis. More horizontal.
Somewhere between dots but still not best fit line.
Since this was a random line, we need a mechamism to move this line iteratively and slowly towards the place where it best fits the sample data (dots in the graph).
So, effectively we need
To find if its best fit line or not
If its not best fit line then move it towards the best fit line. It means we will have to change the value of c and a.
How much values of c and a we need to change and in which direction? We will use combination of gradient descent with least square method to achive these objectives. These are explained below.
Mathematics
For each item in the sample data (called training set too), get the value of y from our estimated line (c=1, a=1). Lets call it h(y).
Also, we have y which is real value for each sample data.
We get the difference between approximated h(y) and y as h(y) — y. Square this difference up. So for one sample we have (h(y) — y) ^ 2
Do it for all samples. Lets say we have sample size as ‘m’, we get the squares of differences for each sample size, sum it up (summmation from 1 to m), get the average of it (i.e. divide by m), make it half (as half of squared difference provides better results). We now have our cost function. We need to minimize this cost function. It means we need to minimize the distance of our line with sample data (dots) to get best fit line.
More explanation of cost function is at — https://www.coursera.org/learn/machine-learning/lecture/rkTp3/cost-function
But we have 2 variables c and a which we need to keep changing to get at best fit line.
For initial combination of c and a, we need to find how much to move and then move it. Once we have new line for new values of c and a then do the distance calculation with each point again and keep doing it till we find that we are not moving much e.g. moving quite less.
It bascially means our line is stil far from the dots in the graph.
Since we need to change both c and a independently, we will use partial differentiation. So we will get the derivative of above cost function wrt c and then wrt a. Remember, we have h(y) for one sample = c + ax
So result of partial derivates of cost function wrt ‘c’ will be
And result of partial derivates of cost function wrt ‘a’ will be
Now we decide the step size i.e. how much big or small of step we want to take in each iteration. Lets call it alpha and we set this value to 0.01. It can be any suitable value based upon the data.
So the distance to be moved by c will be
And the distance to be moved by a will be
Above farmulae will not only give the distance we moved but also the direction e.g. if the random like we initially drew was more verticlish and had to move down to be the best fit then new values will decrease e.g. a will change from 1 to 0.99. But if our initial random line was horizontalish and had to move up to be best fit like then new values will be more e.g. a will change from 1 to 1.01 .
And as mentioned earlier we stop when we realize we are almost not moving or after certain number of iterations. It means we have reached our best fit line i.e. the red line.
What is learning without an example exercise.
Lets take the data of passenger vehicle sales in India. To keep things simple, lets say only one variable which is GDP of the country has impact on the sales. In reality there are more factors like auto interest loan rates etc but thats for next article. For this lets focus on linear equation of number of vehicle sold in India wrt GDP.
Sample Data
I looked at passenger vehicles sales in India year wise for last few years. Also checked GDP for each year. While looking at both the data it dawned upon me that impact of GDP in ‘current year’ will have effect on vehicle sales ‘next year’ . So whichever year GDP was less, the coming year sales was lower and when GDP increased the next year vehicle sales also increased. Hence they say preparing the data for ML analytics is more important and thats where most time needs to be spent.
Lets have equation as y = c + ax .
y = number of vehicles sold in the year
x = GDP of prior year.
We need to find c and a.
Below is table of data. I saved this data in a file called ‘vehicle_sale_data’. Please note that number of vehicles sold figure is in lakhs (1 lakh = 0.1 million).
year,GDP,4wheeler_passengar_vehicle_sale(in lakhs)2011,6.2,26.32012,6.5,26.652013,5.48,25.032014,6.54,26.012015,7.18,27.92016,7.93,30.47
First column = year, which is of not much use in the code below
Second column — GDP for the ‘previous’ year. This is x in the equation.
Third column — Number of vehicles sold. This is what we want to predict maybe for nexy year if we know the GDP of current year.
We will use python to create the model. Below are the steps.
Read the file. ‘gdp_sale’ dictionary will have key as GDP and value is sales.
def read_data() :    data = open("vehicle_sale_data" , "r")    gdp_sale = collections.OrderedDict()    for line in data.readlines()[1:] :        record = line.split(",")        gdp_sale[float(record[1])] = float(record[2].replace('\n', ""))    return gdp_sale
Calulate the step and get to new ‘c’ and ‘a’. For first time, we will pass the initial value ‘c’ and ‘a’. This function will calculate the values of new c and new a after moving one step. This function need to be called iteratively till it stablizes.
def step_cost_function_for(gdp_sale, constant, slope) :    global stepSize    diff_sum_constant = 0 # diff of sum for constant 'c' in "c + ax" equation    diff_sum_slope = 0  # diff of sum for 'a' in "c + ax" equation    gdp_for_years = list(gdp_sale.keys())    for year_gdp in gdp_for_years: # for each year's gdp in the sample data        # get the sale for given 'c' and 'a'by giving the GDP for this sample record        trg_data_sale = sale_for_data(constant, slope, year_gdp) # calculated sale for current 'c' and 'a'        a_year_sale = gdp_sale.get(year_gdp) # real sale for this record        diff_sum_slope = diff_sum_slope + ((trg_data_sale - a_year_sale) * year_gdp) # slope's (h(y) - y) * x        diff_sum_constant = diff_sum_constant + (trg_data_sale - a_year_sale) # consant's (h(y) - y)    step_for_constant = (stepSize / len(gdp_sale)) * diff_sum_constant # distance to be moved by c    step_for_slope = (stepSize / len(gdp_sale)) * diff_sum_slope # distance to be moved by a    new_constant = constant - step_for_constant # new c    new_slope = slope - step_for_slope # new a    return new_constant, new_slope
Function to get the sales of vehicles provided the values of c, a and x. Used by above function for each sample data (gdp).
def sale_for_data(constant, slope, data):    return constant + slope * data   # y = c + ax format
Iteration to get optimum weights ie optimum values of c and a. It will stop if c and a both are not moving more than 0.01 in next iteration.
def get_weights(gdp_sale) :    constant = 1    slope = 1    accepted_diff = 0.01    while 1 == 1:  # continue till we reach local minimum        new_constant, new_slope = step_cost_function_for(gdp_sale, constant, slope)        # if the diff is too less then lets break        if (abs(constant - new_constant) <= accepted_diff) and (abs(slope - new_slope) <= accepted_diff):            print "done. Diff is less than " + str(accepted_diff)            return new_constant, new_slope        else:            constant = new_constant            slope = new_slope            print "new values for constant and slope are " + str(new_constant) + ", " + \                  str(new_slope)
And of course the main function
def main() :    contant, slope = get_weights(read_data())    print "constant :" + contant + ", slope:" + slopeif __name__ == '__main__':    main()
I got the equation as
y (vehicles sales) = 1.43 + 3.84 * x
x is value of GDP
So if we have GDP as 7.5 this year then we will have passenger vehicles sales next year as — 1.43 7.5*3.84 = 30.23
The complete program is below. Its also on github at https://github.com/skhurana333/ml/blob/master/linearRegSingleVariant.py
# sales of vehicle as a function of GDP (for India)import collectionsstepSize = 0.01def read_data() :    data = open("vehicle_sale_data" , "r")    gdp_sale = collections.OrderedDict()    for line in data.readlines()[1:] :        record = line.split(",")        gdp_sale[float(record[1])] = float(record[2].replace('\n', ""))    return gdp_saledef sale_for_data(constant, slope, data):    return constant + slope * data   # y = c + ax formatdef step_cost_function_for(gdp_sale, constant, slope) :    global stepSize    diff_sum_constant = 0 # diff of sum for constant 'c' in "c + ax" equation    diff_sum_slope = 0  # diff of sum for 'a' in "c + ax" equation    gdp_for_years = list(gdp_sale.keys())    for year_gdp in gdp_for_years: # for each year's gdp in the sample data        # get the sale for given 'c' and 'a'by giving the GDP for this sample record        trg_data_sale = sale_for_data(constant, slope, year_gdp) # calculated sale for current 'c' and 'a'        a_year_sale = gdp_sale.get(year_gdp) # real sale for this record        diff_sum_slope = diff_sum_slope + ((trg_data_sale - a_year_sale) * year_gdp) # slope's (h(y) - y) * x        diff_sum_constant = diff_sum_constant + (trg_data_sale - a_year_sale) # consant's (h(y) - y)    step_for_constant = (stepSize / len(gdp_sale)) * diff_sum_constant # distance to be moved by c    step_for_slope = (stepSize / len(gdp_sale)) * diff_sum_slope # distance to be moved by a    new_constant = constant - step_for_constant # new c    new_slope = slope - step_for_slope # new a    return new_constant, new_slopedef get_weights(gdp_sale) :    constant = 1    slope = 1    accepted_diff = 0.01    while 1 == 1:  # continue till we reach local minimum        new_constant, new_slope = step_cost_function_for(gdp_sale, constant, slope)        # if the diff is too less then lets break        if (abs(constant - new_constant) <= accepted_diff) and (abs(slope - new_slope) <= accepted_diff):            print "done. Diff is less than " + str(accepted_diff)            return new_constant, new_slope        else:            constant = new_constant            slope = new_slope            print "new values for constant and slope are " + str(new_constant) + ", " + \                  str(new_slope)def main() :    contant, slope = get_weights(read_data())    print "constant :" + contant + ", slope:" + slopeif __name__ == '__main__':    main()