Univariate Linear Regression-Theory and Practice | by Gokul S Kumar | Towards Data Science

Introduction: This article explains the math and execution of univariate linear regression. This will include the math behind cost function, gradient descent, and the convergence of cost function.
Machine Learning is majorly divided into 3 types
Supervised Machine LearningUnsupervised Machine LearningReinforcement learning
Supervised Machine Learning
Unsupervised Machine Learning
Reinforcement learning
In supervised machine learning, a set of training examples with the expected output are used to train the model. The model then after training on these examples, tries to predict the output values of another set of examples. There are two types of supervised machine learning:
Regression- predicts continuous value output. All the types of regression come under this section.Classification- predict discrete-valued outputs. SVM, KNN, and random forests come under this section.
Regression- predicts continuous value output. All the types of regression come under this section.
Classification- predict discrete-valued outputs. SVM, KNN, and random forests come under this section.
There are several types of regression algorithms such as Linear regression, Logistic regression, Polynomial regression, Stepwise regression, Ridge regression, Lasso regression, Elastic-net regression.
Let’s consider the case of Linear regression with one variable. We will be dealing with math and executions side-by-side. The link to the dataset used is given below:
https://www.kaggle.com/c/home-data-for-ml-course/data
It is the dataset having the info about the sale values of different houses in Boston and its dependency on other features.
Let’s import the required libraries:
import pandas as pdimport matplotlib.pyplot as pltfrom matplotlib import style
And load the dataset into a Pandas dataframe
read_df = pd.read_csv(‘train.csv’)df = read_df.copy()df.head()df.info()
The detailed description of the features is provided along with the data-set. Brief info can be obtained:
<class 'pandas.core.frame.DataFrame'>RangeIndex: 1460 entries, 0 to 1459Data columns (total 81 columns): #   Column         Non-Null Count  Dtype  ---  ------         --------------  -----   0   Id             1460 non-null   int64   1   MSSubClass     1460 non-null   int64   2   MSZoning       1460 non-null   object  3   LotFrontage    1201 non-null   float64 4   LotArea        1460 non-null   int64   5   Street         1460 non-null   object  6   Alley          91 non-null     object  7   LotShape       1460 non-null   object  8   LandContour    1460 non-null   object  9   Utilities      1460 non-null   object  10  LotConfig      1460 non-null   object  11  LandSlope      1460 non-null   object  12  Neighborhood   1460 non-null   object  13  Condition1     1460 non-null   object  14  Condition2     1460 non-null   object  15  BldgType       1460 non-null   object  16  HouseStyle     1460 non-null   object  17  OverallQual    1460 non-null   int64   18  OverallCond    1460 non-null   int64   19  YearBuilt      1460 non-null   int64   20  YearRemodAdd   1460 non-null   int64   21  RoofStyle      1460 non-null   object  22  RoofMatl       1460 non-null   object  23  Exterior1st    1460 non-null   object  24  Exterior2nd    1460 non-null   object  25  MasVnrType     1452 non-null   object  26  MasVnrArea     1452 non-null   float64 27  ExterQual      1460 non-null   object  28  ExterCond      1460 non-null   object  29  Foundation     1460 non-null   object  30  BsmtQual       1423 non-null   object  31  BsmtCond       1423 non-null   object  32  BsmtExposure   1422 non-null   object  33  BsmtFinType1   1423 non-null   object  34  BsmtFinSF1     1460 non-null   int64   35  BsmtFinType2   1422 non-null   object  36  BsmtFinSF2     1460 non-null   int64   37  BsmtUnfSF      1460 non-null   int64   38  TotalBsmtSF    1460 non-null   int64   39  Heating        1460 non-null   object  40  HeatingQC      1460 non-null   object  41  CentralAir     1460 non-null   object  42  Electrical     1459 non-null   object  43  1stFlrSF       1460 non-null   int64   44  2ndFlrSF       1460 non-null   int64   45  LowQualFinSF   1460 non-null   int64   46  GrLivArea      1460 non-null   int64   47  BsmtFullBath   1460 non-null   int64   48  BsmtHalfBath   1460 non-null   int64   49  FullBath       1460 non-null   int64   50  HalfBath       1460 non-null   int64   51  BedroomAbvGr   1460 non-null   int64   52  KitchenAbvGr   1460 non-null   int64   53  KitchenQual    1460 non-null   object  54  TotRmsAbvGrd   1460 non-null   int64   55  Functional     1460 non-null   object  56  Fireplaces     1460 non-null   int64   57  FireplaceQu    770 non-null    object  58  GarageType     1379 non-null   object  59  GarageYrBlt    1379 non-null   float64 60  GarageFinish   1379 non-null   object  61  GarageCars     1460 non-null   int64   62  GarageArea     1460 non-null   int64   63  GarageQual     1379 non-null   object  64  GarageCond     1379 non-null   object  65  PavedDrive     1460 non-null   object  66  WoodDeckSF     1460 non-null   int64   67  OpenPorchSF    1460 non-null   int64   68  EnclosedPorch  1460 non-null   int64   69  3SsnPorch      1460 non-null   int64   70  ScreenPorch    1460 non-null   int64   71  PoolArea       1460 non-null   int64   72  PoolQC         7 non-null      object  73  Fence          281 non-null    object  74  MiscFeature    54 non-null     object  75  MiscVal        1460 non-null   int64   76  MoSold         1460 non-null   int64   77  YrSold         1460 non-null   int64   78  SaleType       1460 non-null   object  79  SaleCondition  1460 non-null   object  80  SalePrice      1460 non-null   int64  dtypes: float64(3), int64(35), object(43)memory usage: 924.0+ KB
There are a total of 81 features including ID. The total no. of entries is 1460. We won’t be doing any data cleaning as of now as the purpose is to get to know about Linear regression alone. Lets us consider the feature LotArea as the input feature and SalePrice as the output. Plotting both of them:
fig = plt.figure(figsize = (10,10))style.use(‘ggplot’)plt.scatter(df.LotArea, df.SalePrice, s = 5, c = ‘k’)
As we can see most of the houses are having a plot area less than 50000 sqft and having a sale price less than 500000 $. All the other cases may be considered as exceptional ones or outliers(depending upon other features).
The basic idea of Linear regression is to find a straight line that fits the set of data points. I will be using the following notations in this article:
m = No. of training examples => 1460 in our case
x’s = ‘Input’ variable/features => LotArea
y’s = ‘Output’ variable/ labels => SalePrice
(x,y) comprises of one training example.
represents the i’th training example.
The process of regression, in general, is explained by the following flow chart:
The training set is seeded into the learning algorithm to produce a hypothesis. A hypothesis is a function that predicts the output values based on the input values. In our case, the LotArea is given as input to the hypothesis to get the estimated SalePrice. h is a function that maps from x’s to y’s.
In the case of Linear regression the hypothesis is represented by the following equation:
are the parameters of the hypothesis. I’ll be using ‘theta_0’ and ‘theta_1’ while referring to the parameters.
The above figure shows a pictorial example of the hypothesis. It is the regression line that fits the given data-points. Here the hypothesis is similar to the equation of a straight line as we are using Linear Regression. We can also use any type of function as the hypothesis to fit the data accordingly. The hypothesis used here is that of linear regression with one variable or also Uni-variate Linear Regression.
Let’s see how to choose the parameters of the hypothesis. The idea here is to choose the parameters so that it best fits the y’s i.e., choose theta_0 and theta_1 such that h(x) is close to the values of y for each x. This condition can be mathematically represented as follows:
where m is the no. of training examples.
The above equation is equivalent to:
Here h(x) is the hypothesis mentioned above and y is the output values for corresponding x values. The above equation reduces to finding the value of theta_0 and theta_1 which minimizes the difference between h(x) which is the estimated output and y which is the actual output.
J(theta_0, theta_1) is called the cost function. There are many types of cost functions available. The mean square error (MSE) cost function is the one usually used in regression problems, which is shown above.
Lets further analyze the cost function. For simplification consider:
The hypothesis then becomes:
and the cost function
This means that all the hypothesis functions pass through origin i.e., (0,0).
Consider the following data points:
Consider:
The following regression line will be obtained:
The value of the cost function is:
Consider
The regression line obtained will be:
The lines dropping from the data-points to the regression line are the errors between the actual and the estimated values. Calculating cost J
Consider
Plotting the cost for different values of theta_0, we get:
From the plot, we can see that the value of cost function is minimum at
which corresponds to the regression line passing through all the three data-points. From this, we can imply that the value of parameters giving the least value of cost function corresponds to the best-fitting regression line.
Let’s come back to our original cost function. As we had considered theta_0 =0 in the previous section, we were able to draw a 2-D plot for the cost function. But in reality, as cost-function depends on both theta_0 and theta_1, we will be getting a bow-shaped 3-D figure(the shape depends upon the training set). Consider the LotArea and SalePrice features from the housing prices dataset. Let’s try plotting the cost function for those features.
from mpl_toolkits.mplot3d.axes3d import Axes3Dimport numpy as npm = len(df.LotArea)theta_0 = np.linspace(10000, 100000, num = 1000)theta_1 = np.linspace(5, 10, num = 1000)Theta_0, Theta_1 = np.meshgrid(theta_0, theta_1)def cost(theta_0, theta_1, LotArea, SalePrice):    h = 0    for i in range(len(LotArea)):        h += (((theta_0 + (theta_1 * LotArea[i])) - SalePrice[i]) **            2)    J = (1/2*len(LotArea)) * h    return Jfig = plt.figure(figsize = (12,12))style.use('ggplot')ax = fig.add_subplot(111, projection = '3d')ax.plot_surface(Theta_0, Theta_1, cost(Theta_0, Theta_1, df.LotArea, df.SalePrice), cmap = 'viridis')ax.set_xlabel('θ0')ax.set_ylabel('θ1')ax.set_zlabel('J')
The following plot is obtained:
The XY plane represents the different values of theta_0 and theta_1 and the height of the surface plot from any point on the XY plane gives the value of J for the values of theta_0 and theta_1 corresponding to the point. We can also draw a contour plot to represent J(theta_0, theta_1).
fig = plt.figure(figsize = (12,12))style.use(‘ggplot’)ax = fig.add_subplot(111)cs = ax.contourf(Theta_0, Theta_1, cost(Theta_0, Theta_1, df.LotArea, df.SalePrice))cbar = fig.colorbar(cs)ax.set_xlabel('θ0')ax.set_ylabel('θ1')plt.show()
The following contour plot is obtained:
The side-bar shows the changing values of J with the variation in the colors of the plot. All the points in a particular ring in the contour plot have the same value of J for different values of theta_0 and theta_1.
Lets plot the regression lines for the values of theta_0 and theta_1 that we have considered above:
minj = np.min(cost(Theta_0, Theta_1, df.LotArea, df.SalePrice))point = np.array(cost(Theta_0, Theta_1, df.LotArea, df.SalePrice)) == minjposition = np.where(point)positionoutput>>(array([9]), array([999]))
The output shows that the position of values of theta_0 and theta_1 that gives the minimum cost value.
theta_1_min = Theta_1[9][999]theta_0_min = Theta_0[9][999]def fitline(theta_0, theta_1, LotArea):    x = []    for i in range(m):        x.append(theta_0 + (theta_1 * LotArea[i]))    return xfig = plt.figure(figsize = (10,10))style.use('ggplot')count = 0for i in range(len(theta_0)):    plt.plot(df.LotArea,fitline(theta_0[i], theta_1[i], df.LotArea), color = 'k', alpha = 0.1, linewidth = 0.1)    count += 1print(count)plt.scatter(df.LotArea, df.SalePrice, s = 5, c = 'r')plt.plot(df.LotArea, (theta_0_min + (theta_1_min * df.LotArea)), color = 'k')plt.show()
The shaded area consists of 1000 lines corresponding to the theta_0 and theta_1 values and the highlighted line is the line corresponding to the values of parameters for which we get the minimum cost. This is not the actual minimum cost but only in the minimum in the considered range of parameters.
The algorithm used to find the values of the parameters which gives the actual minimum value of cost is Gradient Descent. It is a general algorithm which is used to minimize not only cost function J but also other function as well. The general outline of this algorithm will be:
Start with some value of theta_0 and theta_1(generally both are set to zero.)Keep changing the values until we reach the minimum value of J(theta_0, theta_1).
Start with some value of theta_0 and theta_1(generally both are set to zero.)
Keep changing the values until we reach the minimum value of J(theta_0, theta_1).
The algorithm is as follows:
where
Alpha or learning rate decides the size of steps that the algorithm takes to reach the minimum cost value(i.e., the values of parameters which give the minimum cost value). The very important detail that we have to notice is that we have to update theta_1 and theta_0 simultaneously. We shouldn’t be updating theta_0, then upgrade the cost function and then theta_1, that doesn’t work it the way we want.
By looking into the partial derivative part of the algorithm, consider the 2-D cost function plot that we plotted earlier. Assume we are updating the value of theta_1 as we neglected theta_0 in that case.
The partial derivative term gives us the slope of the tangent line at each theta_1 point on the plot. The algorithm will be:
Consider the point and its tangent line in the above plot. As we can see the tangent line has a positive slope, which puts the equation as:
Therefore we can see that the value of theta_1 is reducing, which is what we have to do reach the minimum point step by step. If we consider a value of theta_1 on the left side of the plot, the slope will be a negative number thus increasing theta_1 and reaching the minimum point eventually.
The learning rate(alpha) can also affect the path of convergence. If alpha is too small then the algorithm takes a long time to reach the minimum value thereby delaying convergence.
If alpha is too large then the algorithm can overshoot the minimum thus not converging or maybe even diverging, as shown in the above plot.
Considering applying the algorithm when we have reached the local minimum value. The partial derivative part of the algorithm will be zero as the tangent line will be horizontal at that point, thereby the value of theta_0 won't be changed.
As we approach a local minimum value, the slope of the tangent lines keeps on decreasing to zero. So the value by which the parameters are decreased or increased also reduces as we near the minimum value. By this, we can see that the algorithm converges even if we maintain a constant value of alpha.
Applying this Gradient Descent algorithm to the problem of Univariate Linear regression.
For j = 0
For j = 1
which gives us the gradient descent algorithm as
Consider the housing prices dataset again. Let's perform Gradient Descent on the feature LotArea and the output SalePrice.
from sklearn import preprocessingx = df.LotAreay = df.SalePricex = preprocessing.scale(x)theta_0_gd = 0theta_1_gd = 0alpha = 0.01h_theta_0_gd = 1h_theta_1_gd = 1epoch = 0fig = plt.figure(figsize = (10,10))style.use('ggplot')plt.scatter(x, y, s = 5, c = 'r')while h_theta_0_gd != 0 or h_theta_0_gd != 0:    if epoch > 1000:        break    h_theta_0_gd = 0    h_theta_1_gd = 0    for i in range(m):        h_theta_0_gd += (theta_0_gd + (theta_1_gd * x[i]) - y[i])        h_theta_1_gd += ((theta_0_gd + (theta_1_gd * x[i]) - y[i]) * x[i])    h_theta_0_gd = (1/m) * h_theta_0_gd    h_theta_1_gd = (1/m) * h_theta_1_gd    theta_0_gd -= (alpha * h_theta_0_gd)    theta_1_gd -= (alpha * h_theta_1_gd)    plt.plot(x,(theta_0_gd + (theta_1_gd * x)), color = 'k', alpha = 0.1, linewidth = 1)    epoch += 1plt.plot(x,(theta_0_gd + (theta_1_gd * x)), color = 'r', linewidth = 3)plt.show()
The initial values of parameters are set to be zero. The learning rate is set as 0.01. A maximum of 1000 repetitions or epochs is allowed. All the shaded part in the plot consists of regression lines plotted after obtaining the parameters in each epoch. The highlighted red-line is the final regression line after 1000 epochs. As we can see the density of the lines increases as it nears to the red-line. This happens as the steps taken when the algorithm nears t the minimum value are small thus reducing the space between the corresponding regression lines in each epoch. To confirm if we have correctly applied the algorithm, we can cross-verify it with the LinearRegression model from sklearn.
from sklearn import model_selectionfrom sklearn.linear_model import LinearRegressionx_model = np.array(df.LotArea).reshape(-1,1)y_model = np.array(df.SalePrice).reshape(-1,1)x_model = preprocessing.scale(x_model)x_train, x_test, y_train, y_test = model_selection.train_test_split(x_model, y_model, test_size = 0.33)clf = LinearRegression()clf.fit(x_train, y_train)theta_1_model = clf.coef_theta_0_model = clf.intercept_fig = plt.figure(figsize = (10,10))style.use('ggplot')plt.scatter(x, y, s = 5, c = 'r')plt.plot(x_model,(theta_0_model + (theta_1_model * x_model)), color = 'k')
As we can see the regression model from sklearn also produces the same graph.
Conclusion: We came to know about the different types of machine learning and also the types of supervised learning. We saw the concepts behind hypothesis, cost function, and gradient descent of univariate linear regression and built a regression line on the housing prices dataset’s features using the above concepts from scratch. It was then compared with the model built using sklearn’s LinearRegression model.
References:
I am learning most of the concepts about Machine learning from this Youtube playlist. It is quite helpful and easy to understand too.