A Practical Approach to Linear Regression in Machine Learning | by Ashwin Raj | Towards Data Science

In the previous blog post, I tried to give you some intuition about the basics of machine learning. In this article, we will be getting started with our first Machine Learning algorithm, that is Linear Regression.
First, we will be going through the mathematical aspects of Linear Regression and then I will try to throw some light on important regression terms like hypothesis and cost function and finally we will be implementing what we have learned by building our very own regression model.
Regression models are supervised learning models that are generally used when the value to be predicted is of discrete or quantitative nature. One of the most common example where regression models are used is predicting the price of a house by training the data of sale of houses of that region.
The idea behind Linear Regression model is to obtain a line that best fits the data. By best fit, what is meant is that the total distance of all points from our regression line should be minimal. Often this distance of the points from our regression line is referred to as an Error though it is technically not one. We know that the straight line equation is of the form:
where y is the Dependent Variable, x is the Independent Variable, m is the Slope of the line and c is the Coefficient (or the y-intercept). Herein, y is regarded as the dependent variable as its value depends on the values of the independent variable and the other parameters.
This equation is the basis for any Linear Regression problem and is referred to as the Hypothesis function for Linear Regression. The goal of most machine learning algorithms is to construct a model i.e. a hypothesis to estimate the dependent variable based on our independent variable(s).
This hypothesis, maps our inputs to the output. The hypothesis for linear regression is usually presented as:
In the above mentioned expression, hθ(x) is our hypothesis, θ0 is the intercept and θ1 is the coefficient of the model.
Cost functions are used to calculate how the model is performing. In layman’s words, cost function is the sum of all the errors. While building our ML model, our aim is to minimize the cost function.
One common function that is often used in regression problems is the Mean Squared Error or MSE, which measure the difference between the known value and the predicted value.
It turns out that taking the root of the above equation is far better option as the values would be less complicated and thus Root Mean Squared Error or RMSE is generally used. We can also use other parameters such as Mean Absolute Error for evaluating a regression model.
RMSE tells us how close the data points are to the regression line. Now we will be implementing what we have learned till now by building our very own linear regression model to predict the price of a house.
Here I will be using Google Colab for building this model. You can also use other IDEs like Jupyter notebooks for playing around with the model.
The code used for this linear regression project can be found here.
Our first step is to import the libraries that might be required to build our model. It is not necessary to import all the libraries at just one place. To get started we will be importing Pandas, Numpy, Matplotlib etc.
#Import Required Librariesimport pandas as pdimport numpy as npimport matplotlib.pyplot as pltimport seaborn as sns#Read the Datasetdf=pd.read_csv('kc_house_data.csv')
Once these libraries have been imported our next step will be fetching the dataset and loading our data. While loading the data, it is important to note that the file name shall have its format (.csv/.xls) specified at the end.
The data that I used for this model can be downloaded directly from here.
CSV files are most commonly used for such purposes, though excel sheet can also be used. The only difference would be that instead of read_csv(), we will have to use read_excel() while using an excel sheet as dataset.
After successfully loading the data, our next step is to visualize this data. Data visualization is an important part in the role of a data scientist. It is recommended to visualize the data so as to find any correlation between the different parameters.
#Visualising the data using heatmapplt.figure()sns.heatmap(df.corr(),cmap='coolwarm')plt.show()
Matplotlib and Seashore are excellent libraries that can be used to visualize our data on various different plots.
While visualizing our data, we found that there is a strong correlation between the two parameters: sqft_living and price. Thereby we will be using these parameters for building our model.
#Selecting the required parametersarea = df[‘sqft_living’]price = df['price']x = np.array(area).reshape(-1,1)y = np.array(price)
More parameters can also be added to the model though it might affect its accuracy. The model that uses various features to predict the outcome of the response variables is called Multivariate Regression model.
After selecting the desired parameters the next step is to import the method train_test_split from sklearn library. This is used to split our data into training and testing data. Commonly 70–80% of the data is taken as the training dataset while the remaining data constitutes the testing dataset.
#Import LinearRegression and split the data into training and testing datasetfrom sklearn.model_selection import train_test_splitX_train, X_test, y_train, y_test = train_test_split(x,y,test_size=0.2,random_state = 0)y_train = y_train.reshape(-1,1)y_test = y_test.reshape(-1,1)#Fit the model over the training datasetfrom sklearn.linear_model import LinearRegressionmodel = LinearRegression()model.fit(X_train,y_train)
After this LinearRegression is imported from sklearn.model_selection and the model is fit over the training dataset. The intercept and coefficient of our model can be calculated as shown below:
#Calculate intercept and coefficientprint(model.intercept_)print(model.coef_)pred=model.predict(X_test)predictions = pred.reshape(-1,1)#Calculate root mean squared error to evaluate model performancefrom sklearn.metrics import mean_squared_errorprint('MSE : ', mean_squared_error(y_test,predictions)print('RMSE : ', np.sqrt(mean_squared_error(y_test,predictions)))
The performance of the model can be evaluated by finding the root mean squared error of the model. Lesser the RMSE, better the model.
Gradient descent is an iterative optimization algorithm to find the minimum of a function. To understand this algorithm imagine a person with no sense of direction who wants to get to the bottom of the valley.
He goes down the slope and takes large steps when the slope is steep and small steps when the slope is less steep. He decides his next position based on his current position and stops when he gets to the bottom of the valley which was his goal. Similar is how gradient descent works.
#Initializing the variablesm = 0c = 0L = 0.001epochs = 100n = float(len(x))
The gradient descent approach is applied step by step to our m and c. Initially let m = 0 and c = 0. Let L be our learning rate. This controls how much the value of m changes with each step.
for i in range(epochs):    Y_pred=m*x+c    Dm = (-2/n)*sum(x*(y-Y_pred))    Dc = (-2/n)*sum(y-Y_pred)    m = m-L*Dm    c = c-L*Dcprint(m,c)#Predicting the valuesy_pred = df['sqft_living'].apply(lambda a:c+m*a)y_pred.head()
Preferentially L is assigned a small value in order to improve the accuracy. Our next step is to calculate the partial derivative of the loss function with respect to m and c. Once this is done we update the values of c and m and repeat the process until our loss function is very small.
With that, we have reached the end of this article. I hope this article would have helped you get a feel about the idea behind linear regression algorithms. If you have any question or if you believe I have made any mistake, contact me! You can get in touch with me via: Email or LinkedIn.